Fit a Generalized Linear Model with Elastic Net Regularization

sgdnet(x, ...)

# S3 method for default
sgdnet(x, y, family = c("gaussian", "binomial",
  "multinomial", "mgaussian"), alpha = 1, nlambda = 100,
  lambda.min.ratio = if (NROW(x) < NCOL(x)) 0.01 else 1e-04,
  lambda = NULL, maxit = 1000, standardize = TRUE,
  intercept = TRUE, thresh = 0.001, standardize.response = FALSE,



input matrix




response variable


reponse type, one of 'gaussian', 'binomial', 'multinomial', or 'mgaussian'. See Supported families for details.


elastic net mixing parameter


number of penalties in the regualrization path


the ratio between lambda_max (the smallest penalty at which the solution is completely sparse) and the smallest lambda value on the path. See Regularization Path for details.


regularization strength


maximum number of effective passes (epochs)


whether to standardize x or not


whether to fit an intercept or not


tolerance level for termination of the algorithm. The algorithm terminates when $$ \frac{|\beta^{(t)} - \beta^{(t-1)}|{\infty}}{|\beta^{(t)}|{\infty}} < \mathrm{thresh} $$


whether y should be standardized for family = "mgaussian"


An object of class 'sgdnet' with the following items:


the intercept


the coefficients stored in sparse matrix format "dgCMatrix". For the multivariate families, this is a list with one matrix of coefficients for each response or class.


the deviance of the null (intercept-only model)


the fraction of deviance explained, where the deviance is two times the difference in loglikelihood between the saturated model and the null model


the number of nozero coefficients along the regularization path. For family = "multinomial", this is the number of variables with a nonzero coefficient for any class.


a matrix of the number of nonzero coefficients for any class (only available for multivariate models)


elastic net mixing parameter. See the description of the arguments.


the sequence of lambda values scaled to the original scale of the input data.


number of observations


accumulated number of outer iterations (epochs) for the entire regularization path


a logical indicating whether an offset was used


a logical indicating if a group lasso penalty was used


the call that generated this fit

Model families

Three model families are currently supported: gaussian univariate regression, binomial logistic regression, and multinomial logistic regression. The choice of which is made using the family argument. Next follows the objectives of the various model families:

Gaussian univariate regression:

$$ \frac{1}{2n} \sum_{i=1}^n (y_i - \beta_0 - x_i^\mathsf{T} \beta)^2 + \lambda \left( \frac{1 - \alpha}{2} ||\beta||_2^2 + \alpha||\beta||_1 \right). $$

Binomial logistic regression:

$$ -\frac1n \sum_{i=1}^n \bigg[y_i (\beta_0 + \beta^\mathsf{T} x_i) - \log\Big(1 + e^{\beta_0 + \beta^\mathsf{T} x_i}\Big)\bigg] + \lambda \left( \frac{1 - \alpha}{2} ||\beta||_2^2 + \alpha||\beta||_1 \right), $$ where \(y_i \in \{0, 1\}\).

Multinomial logistic regression: $$ -\bigg\{\frac1n \sum_{i=1}^n \Big[\sum_{k=1}^m y_{i_k} (\beta_{0_k} + x_i^\mathsf{T} \beta_k) - \log \sum_{k=1}^m e^{\beta_{0_k}+x_i^\mathsf{T} \beta_k}\Big]\bigg\} + \lambda \left( \frac{1 - \alpha}{2}||\beta||_F^2 + \alpha \sum_{j=1}^p ||\beta_j||_q \right), $$ where \(q \in {1, 2}\) invokes the standard lasso and 2 the group lasso penalty respectively, \(F\) indicates the Frobenius norm, and \(p\) is the number of classes.

Multivariate gaussian regression: $$ \frac{1}{2n} ||\mathbf{Y} -\mathbf{B}_0\mathbf{1} - \mathbf{B} \mathbf{X}||^2_F + \lambda \left((1 - \alpha)/2||\mathbf{B}||_F^2 + \alpha ||\mathbf{B}||_{12}\right), $$ where \(\mathbf{1}\) is a vector of all zeros, \(\mathbf{B}\) is a matrix of coefficients, and \(||\dot||_{12}\) is the mixed \(\ell_{1/2}\) norm. Note, also, that Y is a matrix of responses in this form.

Regularization Path

The default regularization path is a sequence of nlambda log-spaced elements from \(\lambda_{\mathrm{max}}\) to \(\lambda_{\mathrm{max}} \times \mathtt{lambda.min.ratio}\), For the gaussian family, for instance, \(\lambda_{\mathrm{max}}\) is the largest absolute inner product of the feature vectors and the response vector, $$\max_i \frac{1}{n}|\langle\mathbf{x}_i, y\rangle|.$$

Relationship with glmnet

sgdnet is modeled to resemble glmnet closely so that users can expect to receive more or less equivalent output regardless of whether sgdnet() or glmnet::glmnet() is called. Nevertheless, there are a few instances where we have decided to diverge from the behavior of glmnet:

  • When the ridge penalty is used (alpha = 0), and a regularization path (\(\lambda~s\)) is automatically generated, glmnet::glmnet() fits the null model as the start of the path (as if \(\lambda = \infty\)) even though the first \(\lambda\) reported actually doesn't yield this fit. In sgdnet, we have opted to fit the model so that it is true to the path that is returned.

See also


# Gaussian regression with sparse features with ridge penalty fit <- sgdnet(abalone$x, abalone$y, alpha = 0) # Binomial logistic regression with elastic net penalty, no intercept binom_fit <- sgdnet(heart$x, heart$y, family = "binomial", alpha = 0.5, intercept = FALSE) # Multinomial logistic regression with lasso multinom_fit <- sgdnet(wine$x, wine$y, family = "multinomial") # Multivariate gaussian regression mgaussian_fit <- sgdnet(student$x, student$y, family = "mgaussian")