Adagrad Algorithm Explained and Implemented from Scratch in Python

Adagrad is an often used extension of stochastic gradient descent that work well for sparse parameter space like text or images. In this video I'll explain and show you how to implement it!

Credit to : Max Olson for the picture in the thumbnail, sorry I to have cut the watermark in the picture. The faint background music is from Youtube Music!

Github: https://github.com/yacineMahdid/artificial-intelligence-and-machine-learning/blob/master/deep-learning-from-scratch-python/Gradient%20Descent%20Optimization%20Algorithms.ipynb

The implementation is very straighforward once the cumulative sum of gradient is understood as it is an extension of the stochastic gradient descent.
You can check out the jupyter notebook with the code over here: https://github.com/yacineMahdid/artificial-intelligence-and-machine-learning/blob/master/Deep%20Learning%20from%20Scratch%20in%20Python/agagrad.ipynb

Also if you want a text tutorial you can check out this one, it's very good: https://ruder.io/optimizing-gradient-descent/index.html#adagrad

Here is a definition of adagrad from wikipedia:
"AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011. Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse. This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative. Examples of such applications include natural language processing and image recognition. It still has a base learning rate η, but this is multiplied with the elements of a vector {Gj,j} which is the diagonal of the outer product matrix."


----
Join the Discord for general discussion: https://discord.gg/QpkxRbQBpf

----
Follow Me Online Here:

Twitter: https://twitter.com/CodeThisCodeTh1
GitHub: https://github.com/yacineMahdid
LinkedIn: https://www.linkedin.com/in/yacine-mahdid-809425163/
Instagram: https://www.instagram.com/yacine_mahdid/
___

Have a great week! ?