Learning with kernel based regularization schemes

Abstract

Learning theory is the mathematical foundation for machine learning algorithms which have important applications in many areas of science and technology. In this thesis, we mainly consider learning algorithms involving kernel based regularization schemes. While algorithms like support vector machine given by Tikhonov regularization schemes associated with convex loss functions and reproducing kernel Hilbert spaces have been extensively studied in the literature, we introduce some non-standard settings and provide insightful analysis for them. Firstly, we study a regression algorithm with `1 regularizer stated in a hypothesis space trained from data or samples by a nonsymmetric kernel. The data dependent nature of the algorithm leads to an extra error term called hypothesis error, which is essentially different from regularization schemes with data independent hypothesis spaces. By dealing with regularization error, sample error and hypothesis error, we estimate the total error in terms of properties of the kernel, the input space, the marginal distribution, and the regression function of the regression problem. Learning rates are derived by choosing suitable values of the regularization parameter. An improved error decomposition approach is used in our data dependent setting. Secondly, we consider the binary classification problem by learning from samples drawn from a non-identical sequence of probability measures. Our main goal is to provide satisfactory estimates for the excess misclassification error of the produced classifiers. Similar results can be obtained for multi-class classification because we give a comparison theory for error analysis of multi-class classifiers. Finally, we consider the sparsity issue for `1 regularization schemes. This topic has attracted a lot of attention recently and we give some discussion for further study on both theoretical and practical aspects.