Deep learning basics

The subject is new so most of the wisdom is scattered in papers, but here are two recent books:

• Deep Learning, Yoshua Bengio, Ian J. Goodfellow, Aaron Courville.
• Deep Learning: Methods and Applications, Li Deng and Dong Yu.

And some practical material.

• ACL 2012 + NAACL 2013 Tutorial: Deep Learning for NLP (without Magic)

Neural Networks and Deep Learning by Michael Nielsen. The book is still in progress, but it looks quite interesting and promising. And it's free! Here's the link.

There are only 5 chapters so far, and the most of them talk about usual neural networks, but it's still worth having a look.

Update: the book has been finished!

There's also Richard Socher's recent PhD dissertation on intersection of NLP and deep learning: Recursive Deep Learning for Natural Language Processing and Computer Vision

This link contains an amazing amount of deep learning literature. Summarizing it here(going in the order a beginner ideally should)- NOTE: All these resources mainly use python.

1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net.

Andrew Ng's Coursera course is pretty good too.

2) For Neural networks, nobody explains it better than Dr.Patrick Winston. The assignments should be tried out for better understanding. They are in python.

3) For a better understanding of Neural Networks, Michael Nielsen's course should be done(as suggested by Alexey). It is pretty basic but it works.

4) For deep neural networks, and implementing them faster on GPUs, there are multiple frameworks available, such as Theano, Caffe, Pybrain, Torch,etc. Out of these Theano provides a better low level functionality that allows its user to create custom NNs. It is a python library, so being able to use numpy,scikit-learn, matplotlib, scipy along with it is a big plus. The deep learning tutorial written by Lisa Lab should be tried out for a better understanding of theano.

5) For Convolutional Neural Networks, follow andrej karpathy's tutorial.

6) For unsupervised learning, follow here and here.

7) For an intersection of deep learning and NLP, follow Richard Socher's class.

8) For LSTMs, read Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780 and Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012.

Here is LSTM's Theano code.

Main references:

Courses on deep learning:

• Andrew Ng's course on machine learning has a nice introductory section on neural networks.
• Geoffrey Hinton's course: Coursera Neural Networks for Machine Learning (fall 2012)
• Michael Nielsen's free book Neural Networks and Deep Learning
• Yoshua Bengio, Ian Goodfellow and Aaron Courville wrote a book on deep learning
• Hugo Larochelle's course (videos + slides) at Université de Sherbrooke
• Stanford's tutorial (Andrew Ng et al.) on Unsupervised Feature Learning and Deep Learning
• Oxford's ML 2014-2015 course
• NVIDIA Deep learning course (summer 2015)
• Google's Deep Learning course on Udacity (January 2016)

NLP-oriented:

• Stanford CS224d: Deep Learning for Natural Language Processing (spring 2015) by Richard Socher
• Tutorial given at NAACL HLT 2013: Deep Learning for Natural Language Processing (without Magic) (videos + slides)

Vision-oriented:

• CS231n Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (a previous version, shorter and less polished: Hacker's guide to Neural Networks).

Toolkit-specific tutorials:

• DL4J (Java)
• Theano (Python, Y. Bengio)
• Machine Learning with Torch7 (Lua, LeCun)
• H2O Deep Learning (Java)
• Caffee (C++, UCB)
• Nervana’s Deep Learning Course

For comprehending the derivation of Back propagation algorithm, I suggest Ryan Harris youtube video which is less daunting. You may find second video as well.