Hinge Loss For Binary Classifiers

We begin by looking at the disadvantages for using squared error for classification, and hinge loss helps.

The squared Error "loss" term can often be problematic. This is the 2-norm of Aw - d, where A contains the features of the data and D represents the labels. Squared errors is terrible at classifying whenever there are extreme values in the dataset.

Avoid the loss due to the "easy to classify data"

The max margin classifier can have a decision boundary to be the midpoint between the 2 classes. for example if the boundaries is 2.4 and 4.9 we define the radius to be 3.65 in between. 

Now we perform a squared error function and classify if positive or negative, this parabola accurately classifies how this function works. 

Ideally we want to count the number of misclassifications and try to optimize that. There is a 1 if there is a misclassification and 0 if there is a classification. However, this is a nonconvex function, it is really difficult. The solution for this is Hinge Loss. 

This is (1 - dixi^Tw)^2

and the ideal loss is I(diXiTw < 0) which is Nonconvex. 

Hinge loss is a way of approximating an ideal cost function. It does not introduce penalties when the data is easy to classify. 

This causes piecewise function. The loss increases and incurs when going to the left, and this is the price paid for obtaining a convex function.

This narrows the range of misclassifications, and the yellow area indicates the hinge loss for particular labels. 

Hinge loss does a much better loss than squared error for approximating the ideal cost function for optimization.