Efficient Learning of Generalized Linear and Single Index Models with Isotonic Regression

Transcription

1 Efficient Learning of Generalized Linear and Single Index Models with Isotonic Regression Sha M. Kakade Microsoft Research and Wharton, U Penn skakade@icrosoft.co Varun Kanade SEAS, Harvard University vkanade@fas.harvard.edu Ada Tauan Kalai Microsoft Research adu@icrosoft.co Ohad Shair Microsoft Research ohadsh@icrosoft.co Abstract Generalized Linear Models (GLMs) and Single Index Models (SIMs) provide powerful generalizations of linear regression, where the target variable is assued to be a (possibly unknown) -diensional function of a linear predictor. In general, these probles entail non-convex estiation procedures, and, in practice, iterative local search heuristics are often used. Kalai and Sastry (2009) provided the first provably efficient ethod, the Isotron algorith, for learning SIMs and GLMs, under the assuption that the data is in fact generated under a GLM and under certain onotonicity and Lipschitz (bounded slope) constraints. The Isotron algorith interleaves steps of perceptron-like updates with isotonic regression (fitting a one-diensional non-decreasing function). However, to obtain provable perforance, the ethod requires a fresh saple every iteration. In this paper, we provide algoriths for learning GLMs and SIMs, which are both coputationally and statistically efficient. We odify the isotonic regression step in Isotron to fit a Lipschitz onotonic function, and also provide an efficient O(n log(n)) algorith for this step, iproving upon the previous O(n 2 ) algorith. We provide a brief epirical study, deonstrating the feasibility of our algoriths in practice. Introduction The oft used linear regression paradig odels a dependent variable Y as a linear function of a vector-valued independent variable X. Naely, for soe vector w, we assue that E[Y X] = w X. Generalized linear odels (GLMs) provide a flexible extension of linear regression, by assuing that the dependent variable Y is of the for, E[Y X] = u(w X); u is referred to as the inverse link function or transfer function (see [] for a review). Generalized linear odels include coonly used regression techniques such as logistic regression, where u(z) = /( + e z ) is the logistic function. The class of perceptrons also falls in this category, where u is a siple piecewise linear function of the for /, with the slope of the iddle piece being the inverse of the argin. In the case of linear regression, the least-squares ethod is an highly efficient procedure for paraeter estiation. Unfortunately, in the case of GLMs, even in the setting when u is known, the proble of fitting a odel that iniizes squared error is typically not convex. We are not aware of any classical estiation procedure for GLMs which is both coputationally and statistically efficient, and with provable guarantees. The standard procedure is iteratively reweighted least squares, based on Newton-Raphson (see []). The case when both u and w are unknown (soeties referred to as Single Index Models (SIMs)), involves the ore challenging (and practically relevant) question of jointly estiating u and w,

2 where u ay coe fro a large non-paraetric faily such as all onotonic functions. There are two questions here: ) What statistical rate is achievable for siultaneous estiation of u and w? 2) Is there a coputationally efficient algorith for this joint estiation? With regards to the forer, under ild Lipschitz-continuity restrictions on u, it is possible to characterize the effectiveness of an (appropriately constrained) joint epirical risk iniization procedure. This suggests that, fro a purely statistical viewpoint, it ay be worthwhile to attept jointly optiizing u and w on epirical data. However, the issue of coputationally efficiently estiating both u and w (and still achieving a good statistical rate) is ore delicate, and is the focus of this work. We note that this is not a trivial proble: in general, the joint estiation proble is highly non-convex, and despite a significant body of literature on the proble, existing ethods are usually based on heuristics, which are not guaranteed to converge to a global optiu (see for instance [2, 3, 4, 5, 6]). The Isotron algorith of Kalai and Sastry [7] provides the first provably efficient ethod for learning GLMs and SIMs, under the coon assuption that u is onotonic and Lipschitz, and assuing that the data corresponds to the odel. The saple and coputational coplexity of this algorith is polynoial, and the saple coplexity does not explicitly depend on the diension. The algorith is a variant of the gradient-like perceptron algorith, where apart fro the perceptronlike updates, an isotonic regression procedure is perfored on the linear predictions using the Pool Adjacent Violators (PAV) algorith, on every iteration. While the Isotron algorith is appealing due to its ease of ipleentation (it has no paraeters other than the nuber of iterations to run) and theoretical guarantees (it works for any u, w), there is one principal drawback. It is a batch algorith, but the analysis given requires the algorith to be run on fresh saples each batch. In fact, as we show in experients, this is not just an artifact of the analysis if the algorith loops over the sae data in each update step, it really does overfit in very high diensions (such as when the nuber of diensions exceeds the nuber of exaples). Our Contributions: We show that the overfitting proble in Isotron stes fro the fact that although it uses a slope (Lipschitz) condition as an assuption in the analysis, it does not constrain the output hypothesis to be of this for. To address this issue, we introduce the SLISOTRON algorith (pronounced slice-o-tron, cobining slope and Isotron). The algorith replaces the isotonic regression step of the Isotron by finding the best non-decreasing function with a bounded Lipschitz paraeter - this constraint plays here a siilar role as the argin in classification algoriths. We also note SLISOTRON (like Isotron) has a significant advantage over standard regression techniques, since it does not require knowing the transfer function. Our two ain contributions are:. We show that the new algorith, like Isotron, has theoretical guarantees, and significant new analysis is required for this step. 2. We provide an efficient O(n log(n)) tie algorith for finding the best non-decreasing function with a bounded Lipschitz paraeter, iproving on the previous O(n 2 ) algorith [0]. This akes SLISOTRON practical even on large datasets. We begin with a siple perceptron-like algorith for fitting GLMs, with a known transfer function u which is onotone and Lipschitz. Soewhat surprisingly, prior to this work (and Isotron [7]) a coputationally efficient procedure that guarantees to learn GLMs was not known. Section 4 contains the ore challenging SLISOTRON algorith and also the efficient O(n log(n)) algorith for Lipschitz isotonic regression. We conclude with a brief epirical analysis. 2 Setting We assue the data (x, y) are sapled i.i.d. fro a distribution supported on B d [0, ], where B d = {x R d : x } is the unit ball in d-diensional Euclidean space. Our algoriths and In the ore challenging agnostic setting, the data is not required to be distributed according to a true u and w, but it is required to find the best u, w which iniize the epirical squared error. Siilar to observations of Kalai et al. [8], it is straightforward to show that this proble is likely to be coputationally intractable in the agnostic setting. In particular, it is at least as hard as the proble of learning parity with noise, whose hardness has been used as the basis for designing ultiple cryptographic systes. Shalev-Shwartz et al. [9] present a kernel-based algorith for learning certain types of GLMs and SIMs in the agnostic setting. However, their worst-case guarantees are exponential in the nor of w (or equivalently the Lipschitz paraeter). 2

3 Algorith GLM-TRON Input: data (x i, y i ) Rd [0, ], u : R [0, ], held-out data (x +j, y +j ) s j= w := 0; for t =, 2,... do h t (x) := u(w t x); w t+ := w t + (y i u(w t x i ))x i ; end for Output: arg in h t s j= (ht (x +j ) y +j ) 2 analysis also apply to the case where B d is the unit ball in soe high (or infinite)-diensional kernel feature space. We assue there is a fixed vector w, such that w W, and a non-decreasing -Lipschitz function u : R [0, ], such that E[y x] = u(w x) for all x. The restriction that u is -Lipschitz is without loss of generality, since the nor of w is arbitrary (an equivalent restriction is that w = and that u is W -Lipschitz for an arbitrary W ). Our focus is on approxiating the regression function well, as easured by the squared loss. For a real valued function h : B d [0, ], define err(h) = E (x,y) [ (h(x) y) 2 ] ε(h) = err(h) err(e[y x]) = E (x,y) [ (h(x) u(w x)) 2 ] err(h) easures the error of h, and ε(h) easures the excess error of h copared to the Bayesoptial predictor x u(w x). Our goal is to find h such that ε(h) (equivalently, err(h)) is as sall as possible. In addition, we define the epirical counterparts êrr(h), ˆε(h), based on a saple (x, y ),..., (x, y ), to be êrr(h) = (h(x i ) y i ) 2 ; ˆε(h) = (h(x i ) u(w x i )) 2. Note that ˆε is the standard fixed design error (as this error conditions on the observed x s). Our algoriths work by iteratively constructing hypotheses h t of the for h t (x) = u t (w t x), where u t is a non-decreasing, -Lipschitz function, and w t is a linear predictor. The algorithic analysis provides conditions under which ˆε(h t ) is sall, and using statistical arguents, one can guarantee that ε(h t ) would be sall as well. 3 The GLM-TRON algorith We begin with the sipler case, where the transfer function u is assued to be known (e.g. a sigoid), and the proble is estiating w properly. We present a siple, paraeter-free, perceptronlike algorith, GLM-TRON (Alg. ), which efficiently finds a close-to-optial predictor. We note that the algorith works for arbitrary non-decreasing, Lipschitz functions u, and thus covers ost generalized linear odels. We refer the reader to the pseudo-code in Algorith for soe of the notation used in this section. To analyze the perforance of the algorith, we show that if we run the algorith for sufficiently any iterations, one of the predictors h t obtained ust be nearly-optial, copared to the Bayesoptial predictor. Theore. Suppose (x, y ),..., (x, y ) are drawn independently fro a distribution supported on B d [0, ], such that E[y x] = u(w x), where w W, and u : R [0, ] is a known nondecreasing -Lipschitz function. Then for any δ (0, ), the following holds with probability at least δ: there exists soe iteration t < O(W / log(/δ)) of GLM-TRON such that the hypothesis h t (x) = u(w t x) satisfies ( ) W 2 ax{ˆε(h t ), ε(h t log(/δ) )} O. 3

4 Algorith 2 SLISOTRON Input: data (x i, y i ) Rd [0, ], held-out data (x +j, y +j ) s j= w := 0; for t =, 2,... do u t := LIR ((w t x, y ),..., (w t x, y )) // Fit -d function along w t w t+ := w t + (y i u t (w t x i ))x i end for Output: arg in h t s j= (ht (x +j ) y +j ) 2 In particular, the theore iplies that soe h t has sall enough ε(h t ). Since ε(h t ) equals err(h t ) up to a constant, we can easily find an appropriate h t by picking the one that has least êrr(h t ) on a held-out set. The ain idea of the proof is showing that at each iteration, if ˆε(h t ) is not sall, then the squared distance w t+ w 2 is substantially saller than w t w 2. Since the squared distance is bounded below by 0, and w 0 w 2 W 2, there is an iteration (arrived at within reasonable tie) such that the hypothesis h t at that iteration is highly accurate. Although the algorith iniizes epirical squared error, we can bound the true error using a unifor convergence arguent. The coplete proofs are provided in the full version of the paper ([] Appendix A). 4 The SLISOTRON algorith In this section, we present SLISOTRON (Alg. 2), which is applicable to the harder setting where the transfer function u is unknown, except for it being non-decreasing and -Lipschitz. SLISOTRON does have one paraeter, the Lipschitz constant; however, in theory we show that this can siply be set to. The ain difference between SLISOTRON and GLM-TRON is that now the transfer function ust also be learned, and the algorith keeps track of a transfer function u t which changes fro iteration to iteration. The algorith is inspired by the Isotron algorith [7], with the ain difference being that at each iteration, instead of applying the PAV procedure to fit an arbitrary onotonic function along the direction w t, we use a different procedure, (Lipschitz Isotonic Regression) LIR, to fit a Lipschitz onotonic function, u t, along w t. This key difference allows for an analysis that does not require a fresh saple each iteration. We also provide an efficient O( log()) tie algorith for LIR (see Section 4.), aking SLISOTRON an extreely efficient algorith. We now turn to the foral theore about our algorith. The foral guarantees parallel those of the GLM-TRON algorith. However, the rates achieved are soewhat worse, due to the additional difficulty of siultaneously estiating both u and w. Theore 2. Suppose (x, y ),..., (x, y ) are drawn independently fro a distribution supported on B d [0, ], such that E[y x] = u(w x), where w W, and u : R [0, ] is an unknown non-decreasing -Lipschitz function. Then the following two bounds hold:. (Diension-dependent) ( With probability at least δ, there exists soe iteration t < ( ) ) /3 W O d log(w /δ) of SLISOTRON such that ( (dw ax{ˆε(h t ), ε(h t 2 ) /3 ) log(w /δ) )} O. 2. (Diension-independent) ( With probability at least δ, there exists soe iteration t < ( ) ) /4 W O log(/δ) of SLISOTRON such that ( (W ax{ˆε(h t ), ε(h t 2 ) /4 ) log(/δ) )} O 4

5 As in the case of Th., one can easily find h t which satisfies the theore s conditions, by running the SLISOTRON algorith for sufficiently any iterations, and choosing the hypothesis h t which iniizes êrr(h t ) on a held-out set. The algorith iniizes epirical error and generalization bounds are obtained using a unifor convergence arguent. The proofs are soewhat involved and appear in the full paper ([] Appendix B). 4. Lipschitz isotonic regression The SLISOTRON algorith (Alg. 2) perfors Lipschitz Isotonic Regression (LIR) at each iteration. The goal is to find the best fit (least squared error) non-decreasing -Lipschitz function that fits the data in one diension. Let (z, y ),... (z, y ) be such that z i R, y i [0, ] and z z 2 z. The Lipschitz Isotonic Regression (LIR) proble is defined as the following quadratic progra: Miniize w.r.t ŷ i : 2 (y i ŷ i ) 2 () subject to: ŷ i ŷ i+ i (Monotonicity) (2) ŷ i+ ŷ i (z i+ z i ) i (Lipschitz) (3) Once the values ŷ i are obtained at the data points, the actual function can be constructed by interpolating linearly between the data points. Prior to this work, the best known algorith for this proble wass due to Yeganova and Wilbur [0] and required O( 2 ) tie for points. In this work, we present an algorith that perfors the task in O( log()) tie. The actual algorith is fairly coplex and relies on designing a clever data structure. We provide a high-level view here; the details are provided in the full version ([] Appendix D). Algorith Sketch: We define functions G i ( ), where G i (s) is the iniu squared loss that can be attained if ŷ i is fixed to be s, and ŷ i+,... ŷ are then chosen to be the best fit -Lipschitz non-decreasing function to the points (z i, y i ),..., (z, y ). Forally, for i =,...,, define the functions, G i (s) = in ŷ i+,...,ŷ 2 (s y i) subject to the constraints (where s = ŷ i ), ŷ j ŷ j+ ŷ j+ ŷ j z j+ z j (ŷ j y j ) 2 (4) j=i+ i j (Monotonic) i j (Lipschitz) Furtherore, define: s i = in s G i (s). The functions G i are piecewise quadratic, differentiable everywhere and strictly convex, a fact we prove in full paper []. Thus, G i is iniized at s i and it is strictly increasing on both sides of s i. Note that G (s) = (/2)(s y ) 2 and hence is piecewise quadratic, differentiable everywhere and strictly convex. Let δ i = z i+ z i. The reaining G i obey the following recursive relation. { G i (s) = Gi (s + δ i ) If s s 2 (s y i ) 2 i δ i + G i (s i ) If s i δ i < s s i (5) G i (s) If s i < s As intuition for the above relation, note that G i (s) is obtained fixing ŷ i = s and then by choosing ŷ i as close to s i (since G i is strictly increasing on both sides of s i ) as possible without violating either the onotonicity or Lipschitz constraints. The above arguent can be iediately translated into an algorith, if the values s i are known. Since s iniizes G (s), which is the sae as the objective of (), start with ŷ = s, and then successively chose values for ŷ i to be as close to s i as possible without violating the Lipschitz or onotonicity constraints. This will produce an assignent for ŷ i which achieves loss equal to G (s ) and hence is optial. 5

6 (a) (b) Figure : (a) Finding the zero of G i. (b) Update step to transfor representation of G i to G i The harder part of the algorith is finding the values s i. Notice that G i are all piecewise linear, continuous and strictly increasing, and obey a siilar recursive relation (G (s) = s y ): { G i (s + δ i ) If s s G i δ i i (s) = (s y i ) + 0 If s i δ i < s s i (6) G i (s) If s i < s The algorith then finds s i by finding zeros of G i. Starting fro, G = s y, and s = y. We design a special data structure, called notable red-black trees, for representing piecewise linear, continuous, strictly increasing functions. We initialize such a tree T to represent G (s) = s y. Assuing that at soe tie it represents G i, we need to support two operations:. Find the zero of G i to get s i. Such an operation can be done efficiently O(log()) tie using a tree-like structure (Fig. (a)). 2. Update T to represent G i. This operation is ore coplicated, but using the relation (6), we do the following: Split the interval containing s i. Move the left half of the piecewise linear function G i by δ i (Fig. (b)), adding the constant zero function in between. Finally, we add the linear function s y i to every interval, to get G i, which is again piecewise linear, continuous and strictly increasing. To perfor the operations in step (2) above, we cannot naïvely apply the transforations, shift-by(δ i ) and add(s y i ) to every node in the tree, as it ay take O() operations. Instead, we siply leave a note (hence the nae notable red-black trees) that such a transforation should be applied before the function is evaluated at that node or at any of its descendants. To prevent a large nuber of such notes accuulating at any given node we show that these notes satisfy certain coutative and additive relations, thus requiring us to keep track of no ore than 2 notes at any given node. This lazy evaluation of notes allows us to perfor all of the above operations in O(log()) tie. The details of the construction are provided in the full paper ([] Appendix D). 5 Experients In this section, we present an epirical study of the SLISOTRON and GLM-TRON algoriths. We perfor two evaluations using synthetic data. The first one copares SLISOTRON and Isotron [7] and illustrates the iportance of iposing a Lipschitz constraint. The second one deonstrates the advantage of using SLISOTRON over standard regression techniques, in the sense that SLISOTRON can learn any onotonic Lipschitz function. We also report results of an evaluation of SLISOTRON, GLM-TRON and several copeting approaches on 5 UCI[2] datasets. All errors are reported in ters of average root ean squared error (RMSE) using 0 fold cross validation along with the standard deviation. 5. Synthetic Experients Although, the theoretical guarantees for Isotron are under the assuption that we get a fresh saple each round, one ay still attept to run Isotron on the sae saple each iteration and evaluate the 6

7 Slisotron Isotron SLISOTRON Isotron ± ± ± 0.08 (a) Synthetic Experient Slisotron SLISOTRON Logistic ± ± ± (b) Synthetic Experient 2 Figure 2: (a) The figure shows the transfer functions as predicted by SLISOTRON and Isotron. The table shows the average RMSE using 0 fold cross validation. The colun shows the average difference between the RMSE values of the two algoriths across the folds. (b) The figure shows the transfer function as predicted by SLISOTRON. Table shows the average RMSE using 0 fold cross validation for SLISOTRON and Logistic Regression. The colun shows the average difference between the RMSE values of the two algoriths across folds. epirical perforance. Then, the ain difference between SLISOTRON and Isotron is that while SLISOTRON fits the best Lipschitz onotonic function using LIR each iteration, Isotron erely finds the best onotonic fit using PAV. This difference is analogous to finding a large argin classifier vs. just a consistent one. We believe this difference will be particularly relevant when the data is sparse and lies in a high diensional space. Our first synthetic dataset is the following: The dataset is of size = 500 in d = 500 diensions. The first co-ordinate of each point is chosen uniforly at rando fro {, 0, }. The reaining co-ordinates are all 0, except that for each data point one of the reaining co-ordinates is randoly set to. The true direction is w = (, 0,..., 0) and the transfer function is u(z) = ( + z)/2. Both SLISOTRON and Isotron put weight on the first co-ordinate (the true direction). However, Isotron overfits the data using the reaining (irrelevant) co-ordinates, which SLISOTRON is prevented fro doing because of the Lipschitz constraint. Figure 2(a) shows the transfer functions as predicted by the two algoriths, and the table below the plot shows the average RMSE using 0 fold cross validation. The colun shows the average difference between the RMSE values of the two algoriths across the folds. A principle advantage of SLISOTRON over standard regression techniques is that it is not necessary to know the transfer function in advance. The second synthetic experient is designed as a sanity check to verify this clai. The dataset is of size = 000 in d = 4 diensions. We chose a rando direction as the true w and used a piecewise linear function as the true u. We then added rando noise (σ = 0.) to the y values. We copared SLISOTRON to Logistic Regression on this dataset. SLISOTRON correctly recovers the true function (up to soe scaling). Fig. 2(b) shows the actual transfer function as predicted by SLISOTRON, which is essentially the function we used. The table below the figure shows the perforance coparison between SLISOTRON and logistic regression. 5.2 Real World Datasets We now turn to describe the results of experients perfored on the following 5 UCI datasets: counities, concrete, housing, parkinsons, and wine-quality. We copared the perforance of SLISOTRON (Sl-Iso) and GLM-TRON with logistic transfer function (GLM-t) against Isotron (Iso), as well as standard logistic regression (Log-R), linear regression (Lin-R) and a siple heuristic algorith (SIM) for single index odels, along the lines of standard iterative axiu-likelihood procedures for these types of probles (e.g., [3]). The SIM algorith works by iteratively fixing the direction w and finding the best transfer function u, and then fixing u and 7

8 optiizing w via gradient descent. For each of the algoriths we perfored 0-fold cross validation, using fold each tie as the test set, and we report averaged results across the folds. Table shows average RMSE values of all the algoriths across 0 folds. The first colun shows the ean Y value (with standard deviation) of the dataset for coparison. Table 2 shows the average difference between RMSE values of SLISOTRON and the other algoriths across the folds. Negative values indicate that the algorith perfored better than SLISOTRON. The results suggest that the perforance of SLISOTRON (and even Isotron) is coparable to other regression techniques and in any cases also slightly better. The perforance of GLM-TRON is siilar to standard ipleentations of logistic regression on these datasets. This suggests that these algoriths should work well in practice, while providing non-trivial theoretical guarantees. It is also illustrative to see how the transfer functions found by SLISOTRON and Isotron copare. In Figure 3, we plot the transfer functions for concrete and counities. We see that the fits found by SLISOTRON tend to be soother because of the Lipschitz constraint. We also observe that concrete is the only dataset where SLISOTRON perfors noticeably better than logistic regression, and the transfer function is indeed soewhat far fro the logistic function. Table : Average RMSE values using 0 fold cross validation. The Ȳ colun shows the ean Y value and standard deviation. dataset Ȳ Sl-Iso GLM-t Iso Lin-R Log-R SIM counities 0.24 ± ± ± ± ± ± ± 0.0 concrete 35.8 ± ± ± ± ±. 0.4 ± ± 0.9 housing 22.5 ± ± ± ± ± ± ± 0.78 parkinsons 29 ± ± ± ± ± ± ± 0.2 winequality 5.9 ± ± ± ± ± ± ± 0.03 Table 2: Perforance coparison of SLISOTRON with the other algoriths. The values reported are the average difference between RMSE values of the algorith and SLISOTRON across the folds. Negative values indicate better perforance than SLISOTRON. dataset GLM-t Iso Lin-R Log-R SIM counities 0.00 ± ± ± ± ± 0.00 concrete 0.56 ± ± ± ± ± 0.26 housing 0.20 ± ± ± ± ± 0.53 parkinsons 0.9 ± ± ± ± ± 0.20 winequality 0.0 ± ± ± ± ± Slisotron Isotron Slisotron Isotron (a) concrete (b) counities Figure 3: The transfer function u as predicted by SLISOTRON (blue) and Isotron (red) for the concrete and counities datasets. The doain of both functions was noralized to [, ]. 8

9 References [] P. McCullagh and J. A. Nelder. Generalized Linear Models (2nd ed.). Chapan and Hall, 989. [2] P. Hall W. Härdle and H. Ichiura. Optial soothing in single-index odels. Annals of Statistics, 2():57 78, 993. [3] J. Horowitz and W. Härdle. Direct seiparaetric estiation of single-index odels with discrete covariates, 994. [4] A. Juditsky M. Hristache and V. Spokoiny. Direct estiation of the index coefficients in a single-index odel. Technical Report 3433, INRIA, May 998. [5] P. Naik and C. Tsai. Isotonic single-index odel for high-diensional database arketing. Coputational Statistics and Data Analysis, 47: , [6] P. Ravikuar, M. Wainwright, and B. Yu. Single index convex experts: Efficient estiation via adapted bregan losses. Snowbird Workshop, [7] A. T. Kalai and R. Sastry. The isotron algorith: High-diensional isotonic regression. In COLT 09, [8] A. T. Kalai, A. R. Klivans, Y. Mansour, and R. A. Servedio. Agnostically learning halfspaces. In Proceedings of the 46th Annual IEEE Syposiu on Foundations of Coputer Science, FOCS 05, pages 20, Washington, DC, USA, IEEE Coputer Society. [9] S. Shalev-Shwartz, O. Shair, and K. Sridharan. Learning kernel-based halfspaces with the zero-one loss. In COLT, 200. [0] L. Yeganova and W. J. Wilbur. Isotonic regression under lipschitz constraint. Journal of Optiization Theory and Applications, 4(2): , [] S. M. Kakade, A. T. Kalai, V. Kanade, and O. Shair. Efficient learning of generalized linear and single index odels with isotonic regression. arxiv.org/abs/ [2] UCI. University of california, irvine: [3] S. Cosslett. Distribution-free axiu-likelihood estiator of the binary choice odel. Econoetrica, 5(3), May