#3: Linear Regression, Regularization, Bias-Variance content
Linear Regression
Model Composition
p-dimensional Input Vector x=[x1,x2,...,xp] Learnable Parameter Vector βT=[β0,β1,…,βp] Target Function f: assumed linear
Hypothesis Function g: any linear function of parameters
β g(x)=β0+i=1∑pβixi Loss Function (Error Metric)
Measures the discrepancy between the model prediction and the actual value on the training set
Residual Sum of Squares
Loss(β)=RSS(β)=i=1∑N(yi−gβ(xi))2=i=1∑N(yi−(β0+j=1∑pβjxji))2=i=1∑N(yi−β0−j=1∑pβjxji)2Training
🎯
Adjust
β to minimize loss
- Matrix Representation of the Hypothesis
Subsume intercept/bias
β0 into the parameter vector
β:
βT=[β0,β1,…,βp]Augment input vector
X by 1:
X=[1,x1,x2,…,xp]⇒y=gβ(X)=X⋅β
- Compute the derivatives of RSS(β) and set it to zeroLoss(β)=RSS(β)=(y−Xβ)T(y−Xβ)δβδRSS=−2XT(y−Xβ)=0
- Solve β for the closed form solutionβ^=(XTX)−1XTy
Making Prediction
For a new observation
x0, its prediction can be computed as
y^0=[1x0]⋅β^=[1x0]⋅(XTX)−1XTyModel Analysis
Significance of a Single Parameter
σ2 = variance of the observations assumed uncorrelated with constant variance
σ^2=N−P−11i=1∑N(yi−y^i)2For any variable
βj define the Z-score as
zj=σ^vjβ^j- vj is the jth diagonal element of (XTX)−1
Result: smaller
zj → less important
βj Significance of a Group of Coefficiants
F-statistics: measures the change in residual
F=RSS1/(N−p1−1)(RSS0−RSS1)/(p1−p0)- RSS1 = residual sum of squares of the bigger model with p1+1 parameters
- RSS0 = residual sum of squares for the nested smaller model p0+1
Regularization (Shrinkage Methods)
🎯
机器学习 = 训练集
D上的
Loss最小化问题 ⇒
F∗容易过拟合
为对抗过拟合,向损失函数中加入描述模型复杂程度的正则项
Ω(F)Regularized Loss Function = Loss Function + Penalty Term
Lp Penalty =λw
- λ = Regularization parameter
- w = weight associated with the variables
- generally considered to be the Lp-norms
Usage
- Help adjust complexity of hypothesis space
- Balance fitness and generalizability
Ridge Regression
Implementation of
L2 Regularization
- Penalty = squared magnitude (L2-norm) of coefficients
Usage
- Shrink the coefficients but greater than 0
- Confine hypothesis space ⇒ make it smaller than the space of all linear functions
Model
β^ridge =argminβi=1∑N(yi−β0−j=1∑pxji⋅βj)2 subject to j=1∑pβj2≤s- Minimize ∑i=1N(yi−β0−∑j=1pxji⋅βj)2
- Subject to constraint ∑j=1pβj2≤s
Rewritten as Lagrangian Multiplier
β^ridge =argminβi=1∑N(yi−β0−j=1∑pxji⋅βj)2+λj=1∑pβj2Loss Function
RSS(λ)=(y−Xβ)T(y−Xβ)+λβTβClosed Form Solution
β^ridge =(XTX+λI)−1XTyLASSO (Least Absolute Shrinkage and Selection Operator) Regression
Implementation of
L1 Regularization
- Penalty = absolute value (L1-norm) of coefficients
Usage
- Prefers sparsity: less terms ⇒ better
- With a lot of parameters, only some of them have predictive power
- Output is sparse: some coefficients are left out
Model
β^lasso =argminβ∑i=1N(yi−β0−j=1∑pxji⋅βj)2subject toj=1∑p∣βj∣≤t- Shrinkage Factor s=∑j=1p∣βj∣t
Bias-Variance Analysis
Purpose
express
Eout in a way that helps choosing the hypothesis space
- How well H can approximate f
- How well we can zoom in on a good hypothesis h∈H
- A theoretical process: Ex is not accessible in practice
Formula
Given:
- gD= Dependence of g on D
- Ex= expectation over x based on the distribution onX
- ED[gD(x)]= Average hypothesis over multiple draws of D
Example
Given:
🚫
H1:
High bias |
Low Variance
✅
H2:
Low bias |
High Variance
Which one is better?
💡
Match the model complexity to the data resources
NOT the target complexity ⇒ better generalizability
Stick to the simplest answer!! (Occam's Razor)
Multi-Class Classification
💡
Use
k Logistic Regression models to classify
k classes
Likelihood for a Single Sample (Softmax Function)
P(yi=k∣xi)=∑j=1Keajeakak=wkT⋅xi,w∈RP×KThe
softmax function is a function that
turns a vector of K real values into
a vector of K real values that sum to 1.
Likelihood of the Data
P(Y∣X)=i=1∏NP(yi∣xi)=i=1∏Nj=1∏KP(yi=j∣xi)yjiLoss Function (Cross-Entropy Loss)
Lw=−log[i=1∏NP(yi∣xi)]=−i=1∑Nj=1∑Kyji⋅log∑j=1Keajeak- yji value of output to the j-th xi