Learning gradients via gradient descent method

Abstract

We discuss the early stopping algorithm for gradient descent schemes on learning the gradient of the regression function. The motivation is to choose \useful" or \relevant" variables by a ranking method for the \large dimension, small sample" problem, where we do the ranking according to the norms of partial derivatives in some function spaces. Satisfactory learning rates are derived. In the algorithm, we used the early stopping technique, instead of the classical Tikhonov regularization method, to avoid over-¯tting. The advantage is that we need no longer consider the choice of the regular- ization coe±cient, for which no e±cient methodology is available. Many practical problems we confront have the character of high- dimension and small-sample, data points are well separated with con¯- dence. We formulate this observation precisely. Then the character is carefully and completely exploited in the analysis of the sample error. As a result, the learning rate has been improved to O(m¡c) (where m denotes the sample size) with c free of the dimension n of the sample space, when n > 23. We also give some analysis of the low-dimensional cases with 2 · n · 23.