Phuong Bich Le and Zung Tien Nguyen Phuong Bich Le: Department of Mathematics Hanoi University of Mining and Geology No. 18 Vien Street Duc Thang Ward Bac Tu Liem District Hanoi City 100000 Vietnam Zung Tien Nguyen: Institut de Mathématiques de Toulouse Université Paul Sabatier 31062 Toulouse France
The main purpose of this work is to study how loss functions in machine learning influence the “binary machines” i.e. probabilistic AI models for predicting binary classification problems. In particular we show the following results: (i) Different measures of accuracy such as area under the curve (AUC) of the ROC curve the maximal balanced accuracy and the maximally weighted accuracy are topologically equivalent with natural inequalities relating them; (ii) the so-called real probability machines with respect to given information spaces are the optimal machines i.e. they have the highest precision among all possible machines and moreover their ROC curves are automatically convex; (iii) the cross-entropy and the square loss are the most natural loss functions in the sense that the real probability machine is their minimizer; (iv) an arbitrary strictly convex loss function will also have as its minimizer an optimal machine which is related to the real probability machine by just a reparametrization of sigmoid values; however if the loss function is not convex then its minimizer is not an optimal machine and strange phenomena may happen.
optimization ; binary classification ; machine learning ; ROC curve ; accuracy metrics ; loss function ; quadratic loss ; quartic loss ; cross-entropy ; convexity ; information space ; optimal machine ; real probability machine ; distorted probabilities (search for similar items in EconPapers)