Kernel-Based Support Vector Machines

 The objectives is to reformulate the linear max margin classifier, derive the kernel version of hinge loss with ridge regression, and define support vector machines.

If we have a collection of linearly separable data points, we can decide to define a classifier that maximizes the margin. The distance is (2/||~w||) where ~w is excluding the classifier concept. We can write things as xTw.

b is the distance from the origin, and the decision boundary depends on several data points circled in orange. The decision boundaries depends only on the points called support vectors.

In kernel function we write d(x) = ΦT(x)w where Φ is a high dimensional space.

Applying Kernel representation for regression make the kernel an inner product between x and xk, which gives the equation:

Suppose we want to have  high-dimensional feature space, and take x and replace it by Φ(x). We take our decision boundary and write our inner product and subsequently solve for w, and perform hinge loss with ridge regression. 

With Hinge loss, we return a value when the argument is positive and return 0 when the argument is zero. I'm gonna make the claim that w has to be a linear function, and we can subsequently write a function for w. So we will write an equation to obtain the Support Vector Machine.

We then obtain the final specification of the Support Vector Machine.

Boundary depends only on support vectors and the α is only associated with sample points. A commonly used kernel is a Gaussian Kernel. We need to solve for α using gradient descent.

We finally can easily solve the SVM with gradient descent.