Theory of learning algorithms generated by scaling

Abstract

In this thesis, we study learning algorithms generated by means of the scaling operator f ! f( ¢ ¾ ) with a scaling parameter ¾ > 0: The main motivation for introducing scaling to learning theory is to learn function features or information with different frequency components when the scaling parameters ¾ changes, as done for signal processing or image compression in wavelet analysis. Since ¾ varies, learning algorithms provide richer information but the analysis becomes more involved. Such learning algorithms include graph Laplacians, multi-Gaussian classification schemes, moving least-square methods, Parzen windows, diffusion maps and learning the kernel parameters. Firstly, binary classification algorithms generated from Tikhonov regularization schemes associated with general convex loss functions and varying Gaussian kernels are considered. Fast convergence rates are provided for the excess misclassification error. Allowing varying Gaussian kernels in the algorithms improves learning rates measured by regularization error and sample error. Special structures of Gaussian kernels enable us to construct, by a nice approximation scheme with a Fourier analysis technique, uniformly bounded regularizing functions achieving polynomial decays of the regularization error under a Sobolev smoothness condition. The sample error is estimated by using a projection operator and a tight bound for the covering numbers of reproducing kernel Hilbert spaces generated by Gaussian kernels. The convexity of the general loss function plays a very important role in our analysis. Secondly, we consider the multi-class classification problem in learning theory. A learning algorithm by means of Parzen windows is introduced. Under some regularity conditions on the conditional probability for each class and some decay conditions of the marginal distribution near the boundary of the input space, we derive learning rates in terms of the sample size, window width and the decay of the basic window. The choice of the window width follows from bounds for the sample error and approximation error. A novelly defined splitting function for the multi-class classification and a comparison theorem, bounding the excess misclassification error by the norm of the difference of function vectors, is crucial in our analysis. Finally, the moving least-square method is studied for the regression problem in learning theory. We provide a learning algorithm associated with a finite dimensional hypothesis space of real valued functions. Mathematical analysis is conducted and error bounds are given.