SMLR-AMP is a collection of AMP-based algorithms that perform multinomial logistic regression for multiclass linear classification and feature selection.

In linear classification, one is given training data of the form \( \{y_m , \boldsymbol{a}_m \}_{m=1}^M \) where \(y_m \in \{1,...,D\}\) are labels and \(\boldsymbol{a}_m \in \mathbb{R}^N\) are feature vectors, and the goal is to design a weight matrix \(\boldsymbol{X} \in \mathbb{R}^{N \times D} \) that best predicts the unknown label \(y_0\) corresponding to a test feature vector \(\boldsymbol{a}_0\) using a linear prediction of the form \( \widehat{y}_0 = \text{argmax}_d [\boldsymbol{z}_0 ]_d \), where \( \boldsymbol{z}_0 = \boldsymbol{X}^{\text{T}}\boldsymbol{a}_0 \in \mathbb{R}^D \) is known as the "score" vector. In feature selection, the goal is to identify the indices of the most discriminatory features in the training data. Linear feature selection can be performed by extracting the indices of the largest rows of the learned weight matrix \(\boldsymbol{X}\).

Many modern "big-data" applications (e.g. text-mining, micro-array gene expression analysis, multi-voxel pattern analysis) are complicated by the fact that the number of training samples \(M\) is much smaller than the dimensionality of the feature vectors \(N\). However, accurate classification may still be possible if relatively few of the \(N\) features are discriminatory. In the latter case, the optimal weight matrix \(\boldsymbol{X}\) should to be row-sparse (or approximately row-sparse)---a property that can be exploited in weight-matrix design.

Multinomial logistic regression (MLR) is a well known approach to multiclass linear classification and feature selection. In MLR, the probability of the label \(y_m\) given the score vector \(\boldsymbol{z}_m\) is modeled using the multinomial logistic activation function, i.e., \[ p(y_m=y~|~ \boldsymbol{z}_m) = \frac{\exp([\boldsymbol{z}_m]_y)}{\sum_{d=1}^D \exp ([ \boldsymbol{z}_m]_d)}. \] We propose two new algorithms for sparse MLR.


Please contact the authors with questions or comments about SMLR-AMP.


Support for this project was provided by NSF grants CCF-1018368 and CCF-1218754

© 2015 Evan Byrne
Template design by Andreas Viklund