2 7 A Probabilistic View | Machine Learning

*PROBABILISTIC VIEW*
I Last class, we discussed the geometric interpretation of least squares.
I Least squares also has an insightful probabilistic interpretation that
allows us to analyze its properties.
I That is, given that we pick this model as reasonable for our problem,
we can ask: What kinds of assumptions are we making

Recall: Gaussian density in n dimensions
Assume a diagonal covariance matrix Σ = σ2I. The density is
p(yjµ; σ2) = 1
(2πσ2)n 2 exp−2σ12 (y − µ)T(y − µ):
What if we restrict the mean to µ = Xw
and find the maximum likelihood
solution for w?

Maximum likelihood for Gaussian linear regression
Plug µ = Xw into the multivariate Gaussian distribution and solve for w
using maximum likelihood.
wML = arg max
w
ln p(yjµ = Xw; σ2)
= arg max
w
−
1
2σ2 ky − Xwk2 − n 2 ln(2πσ2):
Least squares (LS) and maximum likelihood (ML) share the same solution:
LS: arg min
w
ky − Xwk2 , ML: arg max
w
−
1
2σ2 ky − Xwk2

Therefore, in a sense we are making an independent Gaussian noise
assumption about the error, i = yi − xiTw.
I Other ways of saying this:
1) yi = xiTw + i, i ∼iid N(0; σ2), for i = 1; : : : ; n,
2) yi ind ∼ N(xiTw; σ2), for i = 1; : : : ; n,
3) y ∼ N(Xw; σ2I), as on the previous slides.
I Can we use this probabilistic line of analysis to better understand the
maximum likelihood (i.e., least squares) solution?

Expected solution
Given: The modeling assumption that y ∼ N(Xw; σ2I).
We can calculate the expectation of the ML solution under this distribution,
E[wML] = E[(XTX)−1XTy] = Z (XTX)−1XTy p(yjX; w) dy
= (XTX)−1XTE[y]
= (XTX)−1XTXw
= w
Therefore wML is an unbiased estimate of w, i.e., E[wML] = w.
#linearregression #linear #regression #probability #gaussian #leastsquare

*About Me:-*
https://www.youtube.com/channel/UCjJXLmp-74V2jz8nFTphngg/about

*Find videos about :-*

#Data #data #Analysis #analysis #ArtificialIntelligence #ai #AI #DataScience #machinelearning #deeplearning #neuralnetworks #artificialneuralnetwork #ann #convolutionalneuralnetwork #cnn #recurrentneuralnetwork #rnn #longshorttermmemory #lstm #gatedrecurrentunit #gru #computervision #naturalLanguageprocessing #nlp #nltk #spacy #tensorflow #keras #linearregression #linear #logisticregression #regression #knearestneighbour #knn #decisiontree #randomforest #supportvectormachine #svm #clustering #cluster #pca #principlecomponentanalysis #ensemble #sklearn #python