Statistical learning algorithms for regression and regularized spectral clustering

Abstract

In this thesis we investigate several algorithms in statistical learning theory, and our contributions consist of the following three parts. First we focus on the least square regularized regression learning algorithm in a setting of unbounded sampling. Our task is to establish learning rates by means of integral operators. By imposing a moment hypothesis on the unbounded sampling outputs and a function space condition associated with the marginal distribution, we derive learning rates which are consistent with those in the bounded sampling setting. Then we consider the spectral clustering algorithms by learning with a regularization scheme in a sample data hypothesis space and l1-regularizer. The data dependent space spanned by means of the kernel function provides great flexibility for learning. The main difficulty in studying spectral clustering in our setting is that the hypothesis space not only depends on a sample, but also depends on some constrained conditions. The technical difficultly is solved by a local polynomial reproduction formula and a construction method. The consistency of spectral clustering algorithms is stated in terms of properties of the data space, the underlying measure, the kernel as well as the regularity of a target function. Finally, we take a learning theory viewpoint to study a family of learning schemes for regression related to positive linear operators in approximation theory. Such a learning scheme is generated from a random sample by a kernel function parameterized by a scaling parameter. The essential difference between this algorithm and the classical approximation schemes is the randomness of the sampling points, which breaks the condition for good distribution of sampling points often required in approximation theory. We investigate efficiency of the learning algorithm in a regression setting and present learning rates stated in terms of the smoothness of the regression function, sizes of variances, and distances of kernel centers from regular grids. The error analysis is conducted by estimating the sample error and approximation error. Two examples with kernel functions related to continuous Bernstein bases and Jackson kernels are studied in detail and concrete learning rates are obtained.