Constrained Linear Regression

Regression is the method of adjusting parameters in a model to minimize the difference between the predicted output and the measured output. The predicted output is calculated from a measured input (univariate), multiple inputs and a single output (multiple linear regression), or multiple inputs and outputs (multivariate linear regression).

- linear regression: x and y are scalars
- multiple linear regression: x is a vector, y is a scalar response
- multivariate linear regression: x is a vector, y is a vector response

In machine learning terminology, the data inputs (x) are features and the measured outputs (y) are labels. For a single input and single output, m is the slope and c is the intercept: y=mx+c.

An alternate way to write this is in matrix form. Capital letters are often used to indicate when there are multiple inputs (X) or multiple outputs (Y). The difference between the predicted X β and measured Y output is the error ε: Y=X β+ϵ.

Linear regression analysis determines if the error ε has certain statistical properties. A common requirement is that the errors (residuals) are normally distributed (N(μ,Σ)) with zero mean μ=0 and covariance Σ=I (the identity matrix). This implies that the residuals are i.i.d. (independent and identically distributed) random variables. Statistical tests determine if the data fits a linear regression model or if there are unmodeled features of the data that may require a different type of regression model.

Source Code: https://apmonitor.com/me575/index.php/Main/LinearMultivariateRegression