Gradient Descent for Support Vector Machines and Subgradients

 Today, I'm trying to develop a Gradient Descent Algorithm for Support Vector Machines

There is no closed form solution, but the Hinge loss and the L2 norm is convex, so we can develop a gradient descent Algorithm to find the optimum. The hinge can be a gradient descent algorithm. This reflects the operation of taking the positive part of the quantity inside of the parentheses.

We then we want to introduce sub gradient. A sub derivative generalizes the derivative.

A derivative is limit as x approaches x0 of (f(x) - f(x0))/(x - x0). The derivative at x0 is the slope of the curve. Because the function is convex, f(x) is above its tangent line.

f(x) >= f(x0) + d(x0)(x - x0)

The benefit of the sub derivative is that they produce reasonable downhill directions if we're gonna do some sort of gradient descent. 

Sub gradients generalize the idea of a gradient. They are more difficult to visualize than sub derivatives. We can show that the gradient for l(w) that has to lie above the tangent plane.

We can expand the gradient out and take the sum of the distances and take the derivative of said component. The function l(w) should lie above the corresponding tangent plane associated with v. When gradient descent optimization can be replaced with the sub gradient. There is gradient descent algorithm for support vector machines. 

The sub gradient is differentiated, and the derivative will be in the picture below.  We also take the Gradient of the cost and get the final gradient descent algorithm.