[Studying Notes] RNN: Recurrent Neural Networks

The Unreasonable Effectiveness of Recurrent Neural Networks

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

https://news.ycombinator.com/item?id=9584325




Regression and Classification with Neural Networks
http://www.autonlab.org/tutorials/neural13.pdf

The Unreasonable Effectiveness of Recurrent Neural Networks (karpathy.github.io)
		913 points by benfrederickson 123 days ago | 207 comments

Smerity 123 days ago Karpathy is one of my favourite authors - not only is he deeply involved in technical work (audit the CS231n course for more[1]!), he spends much of his time demystifying the field itself, which is a brilliant way to encourage others to explore it :) If you enjoyed his blog posts, I highly recommend watching his talk on "Automated Image Captioning with ConvNets and Recurrent Nets"[2]. In it he raises many interesting points that he hasn't had a chance to get around to fully in his articles. He humbly says that his captioning work is just stacking image recognition (CNN) on to sentence generation (RNN), with the gradients effectively influencing the two to work together. Given that we've powerful enough machines now, I think we'll be seeing a lot of stacking of previously separate models, either to improve performance or to perform multi-task learning[3]. A very simple concept but one that can still be applied to many other fields of interest. [1]: http://cs231n.stanford.edu/ [2]: https://www.youtube.com/watch?v=xKt21ucdBY0 [3]: One of the earliest - "Parsing Natural Scenes and Natural Language with Recursive Neural Networks"http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf