What is a deep neural network?

Deep networks have two main differences with 'normal' networks.

The first is that computational power and training datasets have grown immensely, meaning that it's practical to run larger networks and statistically valid (that is, we have enough training examples that we won't just run into over-fitting problems with larger networks).

The second is that back propagation is limited the more layers you have; each layer represents a gradient of the error, and so by the time one is about six layers deep there isn't much error left to modify the neuron weights. But one might reasonably expect earlier neurons to be more important than later neurons, since they represent 'concepts' that are closer to the raw inputs.

New training techniques sidestep this problem, typically by doing unsupervised learning on the raw inputs, creating higher-level 'concepts' that are then useful as inputs for supervised learning.

(For example, consider the problem of determining whether or not an image contains a cat from the pixels. The early layers of the network should be doing things like detecting edges, which one could expect to be shared among all images and mostly independent of what one is trying to do with the output layers, thus also hard to train through 'cat-not cat' signals many layers up.

General structure of an Artificial Neural Network

Input Layer + Hidden Layers + Output Layer

If there are more hidden layers in the artificial neural network, then the neural network is called as Deep Neural network. How many exactly constitute a deep neural network is a point of debate, but in general, the more the hidden layers, the deep is the neural network.

Coming to why they are so popular or important, many problems like object detection, classification, face recognition, speech recognition got solved with the advent of deep neural networks. It's not a exaggeration to say that, the performance of deep neural networks crossed even the human performance in many of the above mentioned tasks. That means now a computer is the best one to do the above tasks than humans. All the above mentioned problems were lying in research field since almost 5 decades. All of them have been solved to perfection only in the last 4,5 years just because of the success of deep neural networks. That is why they are very popular and important. I mentioned very few problems that i worked on, there are many similar tasks that deep neural networks solved with ease in the last decade.

And, at this point in time, many people across the world are working on solving innumerable applications using deep neural networks.

A deep neural network (DNN) is nothing but a neural network which has multiple layers, where multiple can be subjective.

IMHO, any network which has 6 or 7 or more layers is considered deep. So, the above would form a very basic definition of a deep network.