RECURRENT NEURAL NETWORKS IN DEEP LEARNING/PYTHON/ARTIFICIAL INTELLIGENCE

Recurrent Neural Networks

• Backpropagation Through Time (BPTT) and Gradient Challenges
• Types of RNN and Architecture Variants
• Training and Working of RNN
• Advantages and Disadvantages of RNN

Artifical Recurrent Neural Networks (RNN) is a type of mn neural network that uses the output of the previous step as input to the current step. They are very efficient for sequential data types such as text and time series data. RNNs are designed to detect patterns in data sequences, including spoken word, text, genomes, handwriting, and numerical time series from various sources such as government agencies, stock markets, and sensors. They are integrated with popular apps like Google Translate, Siri, and Voice Search.

Backpropagation in mn

A fundamental aspect of RNNs is their Hidden State, which retains information about a sequence, effectively acting as a memory state by recalling previous inputs to the network. RNNs use the same parameters for every input, performing consistent operations across all inputs and hidden layers to generate output. This parameter sharing reduces complexity in comparison to other neural networks. The output of an RNN is influenced by prior elements within the sequence. Additionally, parameter sharing across layers is a distinctive feature setting RNNs apart from other neural network architectures.

Image source original

The backpropagation through time (BPTT) technique is used by recurrent neural networks (RNNs). It differs slightly from regular backpropagation because it is designed for sequential data. The fundamental ideas of classical backpropagation are shared by BPTT: model training is accomplished by computing errors from the output layer toward the input layer, which allows for model parameter modifications. But a crucial distinction is that, in contrast to feedforward networks, which do not need error summation because there is no layer-to-layer parameter sharing, BPTT sums errors at each time step.

RNNs often encounter issues termed exploding and vanishing gradients, both related to the gradient's magnitude, which denotes the slope of the loss function along the error curve. Vanishing gradients occur when gradients become extremely small, leading to their continuous reduction until they approach insignificance (zero), halting learning. Conversely, exploding gradients manifest when gradients become exceedingly large, causing unstable model behavior and weight parameters reaching NaN representations. To mitigate these challenges, reducing the number of hidden layers within the neural network is a potential solution, alleviating some complexity inherent in RNN models.

Real-World Example for Recurrent Neural Networks

Let’s look at a real-world example for Recurrent Neural Networks, we have a startup that wants to revolutionize language learning. Our new product aims to provide personalized and interactive language education experiences to users worldwide. However, our startup faces a significant hurdle: to understand and predict users’ learning patterns to effectively tailor the learning experience.

To overcome this challenge, our startup looks at Recurrent Neural Networks (RNNs). These powerful AI models were trained on vast datasets of language learners’ interactions, including text inputs, audio recordings, and user engagement metrics. The RNNs can retain the memory of past inputs making them ideal for capturing sequential data, like the progression of a user’s language learning journey over time.

If a user interacts with our learning platform, then their interaction with our system is feed to our RNN model, which then analyzes the user sequence of inputs and detects the patterns or trends in the user-generated data. Our model can detect the learning pattern in the user's behavior by analyzing user-generated data, like their preferred learning topics, study behaviors, and areas where the user faces difficulties, our RNN model can adapt to the learning patterns in real-time which suit every user's learning behavior.

With the assistance of RNNs our learning platform transformed into a dynamic and personalized language platform. In our learning platform users can receive lesson recommendations, it has practice exercises, and our learning platform has its own feedback-based learning trajectory for its users which helps the user to learn a new language more effectively and make it more engaging.

How does RNN differ from a Feedforward Neural Network?

An Artificial Neural Network (ANN) is a type of Neural Network that is also known as a feedforward neural network because it does not have looping nodes. In this neural network-type model the information moves from the input node to hidden nodes and then the output node. It is a unidirectional neural network model. It is also known as a multi-layer neural network.

For applications where the input and output are independent, such as picture classification, feedforward neural networks are appropriate. They are unable to automatically remember data from prior inputs, though. They are less useful for examining sequential data because of this constraint.

Image source original

Recurrent Neuron and RNN Unfolding

A recurrent neural network's basic processing unit isn't referred to as a "Recurrent Neuron." The capacity to remain hidden is one of this unit's unique characteristics. This feature keeps information from prior inputs while processing, enabling the network to grasp sequential dependencies. The ability of the RNN to handle long-term dependencies is improved by variations such as Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) versions.

Image source original

Types Of RNN

There are four types of RNNs based on the number of inputs and outputs in the network they are: 

1. One to One
2. One to Many
3. Many to One
4. Many to Many

One-to-One: This type of recurrent neural network is often called a basic RNN or vanilla neural network because it works just like a simple neural network. This configuration has only one input and one output.

. 

Image source original

One to Many: as the name suggests in this type of RNN model there is one input signal and many output signals associated with it. It is very extensively used in image captioning in which a given image is used to predict a sentence having many words.

Image source original

Many-to-One: It is essentially the one-to-many type paradigm in reverse; in this model, numerous inputs are sent to the network at various network states, and each input produces a single output. We apply this kind of network to sentiment analysis tasks. When solving problems involving mn for sentimental analysis, we provide several words as input and anticipate the sentence's sentiment as the only outcome.

Image source original

Many to Many: Depending on the issue, this kind of neural network has a large number of inputs and outputs. One such issue type that uses many to many RNNs is language translation, where we give it several words in one language as input and it predicts many words in the other language as an output. 

Image source original

Recurrent Neural Network Architecture

The input and output structure of RNNs is similar to that of other mn in deep learning neural networks. However the way information moves from the input layer to the output layer is different. In RNNs, the same weights are maintained throughout the network, unlike deep neural networks where there are separate weight matrices for each dense network. 

Image source original

Variant RNN architectures

Bidirectional Recurrent Neural Networks (BRNN) represent a distinct variation within RNN architecture compared to unidirectional RNNs. While traditional RNNs rely solely on past inputs to make predictions, BRNNs incorporate future data, enhancing predictive accuracy. For example, in the phrase "feeling under the weather," the model can anticipate "under" as the second word more effectively by considering "weather" as the last word in the sequence.

Long-term memory (LSTM) networks, a widely used RNN architecture, effectively solve problems such as the vanishing gradient problem and long-term dependencies. LSTMs introduced by Hochreiter and Schmidhuber contain cells in hidden layers that contain three gates: an input gate, an output gate, and a forget gate. These gates play a vital role in controlling the flow of information needed to make accurate predictions. In situations where the information relevant to the prediction lies below, LSTMs allow the network to retain and use this distant context, which is crucial for tasks such as understanding a person's nut allergy in multiple sentences.

About mn

Similar to LSTMs, Gated Recurrent Units (GRUs) address short-term memory limitations commonly encountered in traditional RNNs. However, unlike LSTMs, GRUs do not rely on special "cell spaces"; Instead, they use hidden modes and integrate two ports: a reset port and an upgrade port. These gates control the amount and nature of the data stored in the network, providing flexibility in learning sequential data while maintaining computational simplicity compared to LSTMs.

How does RNN work?

The recurrent neural network (RNN) is undoubtedly made up of several fixed or constant activation function units, usually one for every time step. Every unit preserves what is known as its "hidden state," which is its internal condition. Usually, the network's stored knowledge or information up to a particular time step is contained in this concealed state. This hidden state changes at every time step as the RNN analyzes sequential data, representing the network's changing comprehension or recollection of the past.

The update of the hidden state in an RNN can be represented using a recurrence relation or formula that defines how this hidden state evolves across time steps. The recurrence relation expresses the change or update in the hidden state based on the current input, previous hidden state, and possibly other parameters or inputs related to the network's architecture and task at hand. This recurrent formula dictates how the network retains and updates its memory or knowledge as it processes sequential data.:

The formula for calculating the current state:

Here,

• h_t→current state (h subscript t)
• h_(t-1)→ previous state (h subscript (t-1))
• x_t→ input state (x subscript t)

The formula for applying activation function (tanh)

Here, 

• W_hh→ weight at the recurrent neuron (W subscript hh)
• W_xh→ weight at input neuron (W subscript xh)

The Formula for calculating output:

• y_t→ output (y subscript t)
• W_hy→ weight at the output layer (W subscript hy)

Backpropagation is used to update each of these parameters. Since RNNs process sequential data, we employ updated backpropagation, sometimes referred to as backpropagation across time.

Backpropagation Through Time (BPTT)

Since the RNN is an ordered neural network, each variable is computed individually in the predetermined sequence, such as h1 coming first, h2 coming next, h3 coming last, and so on. Consequently, we sequentially apply backpropagation to each of these hidden temporal stages.

Image source original

In the above diagram the 

• L(θ) (loss function) depends on h3
• h3 in turn depends on h2 and W
• h2 in turn depends on h1 and W
• h1 in turn depends on h0 and W
• where h0 is a constant starting state.

To better understand the above equation, we will apply backpropagation on only one row.

We also know how to compute this because it is the same as any simple deep neural network backpropagation.

We can also see how to apply the backpropagation to this term 

Because we already know h3 = σ(Wh2+b)

In a network like this which is in ordered form, we can’t compute 

 by simply treating h3 as a constant because it is also dependent on the W. The total derivative  has two parts:

• Explicit:  it treats all the other inputs as a constant.
• Implicit: it sums over all indirect paths from h_3 (h subscript 3) to W

Let's see how to achieve this

For better understanding, we short-circuit some of the paths and get this below equation.

After further modifying the above equation, we get the below equation

Where

Therefore, 

This algorithm is called backpropagation through time (BPTT) as we backpropagate over all previous time steps.

Training through RNN

The network receives a single-time step input and uses both the current input and the previous state to calculate the current state. This current state then becomes the previous state for the next time step, and this process is repeated for several steps, allowing the network to assimilate information from all previous states. After all time steps are completed, the final flow state is used to calculate the output. The output is compared to the target output, resulting in an error. This error then propagates back through the network and updates the weights. The RNN is trained over time using a back-propagation method that adjusts the network parameters based on the calculated error.

Advantages of RNN

• Sequential Modeling: We use RNNs mainly with sequential data, we use RNNs with sequential data because RNNs can store and use previous memory or information processed. This feature is very useful when we are dealing with time series forecasting, language translation or speech recognition, or AI voice assistance. 
• Variable-Length Inputs: Unlike traditional feedforward networks, RNNs can handle variable-length sequences. They process input sequences of varying lengths by sharing parameters across different time steps, allowing flexibility in handling diverse data formats.
• Memory and Context Retention: RNNs possess memory cells that maintain information over time, enabling the network to capture long-term dependencies. This feature helps in learning and retaining context, crucial in tasks where understanding context is essential, like language translation or sentiment analysis.
• Flexibility in inputs and outputs: RNNs can process inputs and produce outputs of various data types (e.g., sequences, vectors, or even structural data). This flexibility allows them to perform diverse tasks, including sequence generation, sentiment analysis, and mn machine learning translation.
• Transfer Learning and Pretrained Models: - Already trained RNN models or embeddings learned model on large text dataset can take benefit of the downstream tasks, that can take advantage of transfer learning and can also reduce the need for extensive labeled data.
• Adjusting to real-time data: RNNs can handle real-time generated data and it can also perform computations tasks on this data, which makes RNNs suitable for works like online prediction, video analysis, and live speech recognition.

Disadvantages of RNN

• Vanishing/Exploding Gradient Problem: Because of vanishing or exploding gradients, RNNs may have trouble training over lengthy sequences. This happens when gradients propagate across time during backpropagation and either become exceedingly small (vanishing) or excessively large (exploding), making it difficult to understand long-range dependencies.
• Difficulty in capturing long-term dependencies: despite their ability to retain information across time steps, standard RNNs can struggle with capturing long-term dependencies effectively. This limitation arises because the network might forget or misinterpret crucial information from distant past inputs when processing lengthy sequences.
• Limited Short-term Memory: Traditional RNNs possess limitations in their short-term memory capacity. They might face challenges in retaining information for an extended duration, which can impact tasks where immediate context plays a significant role.
• Computationally inefficient: RNNs present significant computational challenges, especially when dealing with long sequences. The inherent order of processing limits parallelism, resulting in slower training and inference time compared to feedforward networks.
• Sensitivity to Hyperparameters: RNNs are sensitive to hyperparameters like learning rate, network architecture, and initialization. Selecting appropriate hyperparameters can be challenging, and improper choices might hinder their learning capability.
• Training Instability: training RNNs can be unstable, especially when dealing with non-stationary data or noisy sequences. The network might have difficulties in converging or might be sensitive to data preprocessing.

Summary

Recurrent Neural Networks (RNNs) are an important architecture in the area of sequential data analysis. The RNN networks are developed in such a way that they can adapt during the processing of sequential data they can achieve this because they can store the data or memory from their previous steps, which helps them to do tasks like language processing, time series forecasting, and speech recognition easily. The RNN's ability to model temporal dependencies within sequences, helps it to make predictions based on previous data or inputs. RNN may have data storage as an advantage but it can face challenges like vanishing or exploding gradients during training, which can affect the RNN's ability to capture and leverage long-term dependencies. Additionally, they might face limitations in retaining short-term memory over extended sequences, impacting their understanding of immediate context.

To handle this limitation of RNN, it improved itself by adding variants like long-term short-term memory (LSTM), repetitive units (GRU), and attentional mechanisms. These techniques help the RNN to improve its ability which can help the RNN to handle or understand sequential data more easily and also help it to solve other problems like vanishing gradient problems, and capture long-term dependencies more effectively. Despite ongoing challenges, RNNs remain important for sequential data analysis, providing valuable insights and capabilities for tasks involving the understanding and processing of sequential data.

Python Code

CONVOLUTION NEURAL NETWORK IN DEEP LEARNING/PYTHON/ARTIFICIAL INTELLIGENCE

Convolution Neural Network

• Advantages of Convolutional Neural Network
• Disadvantages of Convolutional Neural Network

The Convolutional Neural Network (CNN) operates on a feed-forward basis, its neuron connectivity pattern inspired by the visual cortex. This CNN deep learning architecture is prominently used in computer vision, an artificial intelligence field focused on interpreting visual data. In this blog the convolutional neural network explanation is done in detail.

A deep learning Convolutional Neural Network is a more advanced version of a conventional artificial neural network that is designed specifically to extract features from datasets that resemble grids. These networks, also referred to as covnets, share parameters between layers. The convolutional layer of a CNN extracts relevant characteristics from input images by applying filters to them. To lessen the computing load, a pooling layer also down samples the image; in the end, the fully connected layer in cnn mainly provides the final predictions. Gradient descent and backpropagation are used to determine how the CNN model determines which filters work best.

It also generally has three layers like Artificial Neural Networks with the same name which are the input layer, hidden layer, and output layer.

Image source original

Input layer: As the name implies, this layer receives input from the real world. Its neuron count aligns with the total number of features within the data (in image scenarios, this equates to the number of pixels).

Hidden layer: After the data passes through the input layer, it moves on to the hidden layers of the neural network. The presence and number of these layers depend on the model and the size of the data set. The number of neurons in each hidden layer can vary, often exceeding the number of inputs. The learning weights and biases are used to calculate the output of each layer by multiplying that output by a matrix derived from the output of the previous layer. An activation function is then applied to that result, which introduces nonlinearity in the network.

Output layer: After processing through the hidden layers, the data is fed into a logistic function such as sigmoid or softmax. These functions convert the results of each class into probability scores and provide a probability estimate for each class. Then data is produced as the output from the output layer. The output layer is the last layer of the neural network that is used to produce the output.

The outcome generated from this process is termed "feedforward." Subsequently, an error calculation takes place using an error function, the error function can be a cross-entropy or square loss error. These functions measure the network's performance or the magnitude of error. Following this, derivatives are computed in a step called backpropagation, primarily aimed at minimizing the loss. 

Convolutional Neural Network Example

Take a look at one such case. Accurately analyzing medical photographs in the face of an increasing number of patient scans presented a significant issue for the healthcare sector. Radiology specialists were overburdened with the volume of cases, which made diagnostic delays more frequent and perhaps dangerous for patients.

Convolutional Neural Networks (CNNs) are a ground-breaking method employed by the healthcare sector to tackle this issue. These advanced artificial intelligence algorithms learned to recognize patterns suggestive of different medical disorders from large datasets of medical pictures, including X-rays, CT scans, and MRI images.

As a new patient underwent imaging procedures, their scans were fed into the CNN machine learning model, which thoroughly analyzed each image pixel by pixel. The CNN’s deep layers of convolutional and pooling operations enabled it to get intricate features and nuances from the images, far surpassing the capabilities of traditional image processing techniques.

By using CNN radiologists from the healthcare industry could expedite the diagnostic process while maintaining a high level of accuracy. The CNN can flag abnormalities like tumors, fractures, and anomalies in organs, providing the healthcare industry and radiologist invaluable insights to guide their further process.

Convolutional Neural Network Working

A convolutional neural network (CNN) is an advanced version of an artificial neural network specifically designed to extract features from grid-like matrix datasets. It finds wide application in image or video problems where data patterns are crucial.

CNN Architecture

The CNN has multiple layers like the input layer, convolutional layer, polling layer, and fully connected layers.

Image source original

The first layer that interacts with the image is the convolutional layer, which is responsible for applying filters to the input image to extract its features. After that, the aggregation layer samples the image to reduce the computational load. Finally, the fully combined layer provides the final forecast. The network uses backpropagation and gradient descent to obtain the most efficient filter.

How Convolutional Layers Works

Convolutional Neural Networks (CNNs) share parameters across layers. Consider an image, which can be visualized as a cuboid with length and width representing the image's dimensions and height depicting the channels (e.g., red, green, and blue channels in typical images). 

Image source original

Consider a scenario where we extract a small segment from an image and apply a neural network, referred to as a filter or kernel, to this segment, generating, for instance, K outputs arranged vertically. Extending this process across the entire image involves sliding this neural network across the image. Consequently, we obtain a modified image, differing in width, height, and depth. Unlike the traditional red, green, and blue channels (RGB), this image now possesses additional channels but reduced width and height. This process is known as Convolution. When the patch size aligns with the image size, it essentially operates as a conventional neural network. This use of smaller patches significantly reduces the number of weights involved.

Note: - the LeNet architecture is a pioneering convolutional neural network developed by Yann LeCun and his colleagues, that laid the foundation for modern deep learning models, that revolutionize various fields including computer vision and pattern recognition.

Let’s look at the mathematics that is involved

Image source original

In this whole convolution process.

• Convolutional layers consist of trainable filters, often called kernels. These filters are characterized by small dimensions in both width and height and correspond to the depth of the input volume, which is typically 3 for image inputs.
• Consider an image of dimensions 34x34x3 for convolution. The size of the filter can be aXaX3, where "a" can vary as 3, 5, or 7, but must be smaller than the size of the image.
• In a forward filter, each filter moves through the entire input volume step by step. This movement is called a step, which for large images can take values like 2, 3, or 4. At each step, the network calculates the dot product between the kernel weights and the patch extracted from the input volume.
• As the filters move over the input, a 2-D output is created for each filter. These outputs are then stacked together to form an output volume with a depth equal to the number of filters used. This process allows the network to learn and adjust its filters accordingly.

Layers used to build Convolutional Neural Network

A convolutional neural network (CNN) comprises different layers, each transforming one volume into another using a differentiable function. Here's an overview using an example image of dimensions 32 x 32 x 3.

Input Layer: The input layer receives input from the external world, commonly images or image sequences in CNNs. It holds the original image data, the image data have a width of 32, height of 32, and depth of 3.

Convolutional Layers: Here, the input dataset undergoes feature extraction. Kernels or learnable filters, typically 2x2, 3x3, or 5x5 matrices, are applied to the input images. These filters slide over the input data, performing dot products with corresponding patches. The resulting output is termed feature maps. For instance, if there are 12 filters in this layer, the output volume becomes 32x32x12.

Activation Layer: By introducing nonlinearity in the network, this layer applies an element-wise activation function to the output of the convolution layer. Popular activation features include e.g. ReLU (max(0, x)), Tanh, and Leaky ReLU. Despite the activation function, this layer retains its original dimensions of 32 x 32 x 12.

Pooling Layer: Periodically integrated into the CNN, this layer aims to reduce data volume size for faster computation, less memory usage, and overfitting prevention. Common types are max pooling and average pooling. For instance, using 2x2 filters and a stride of 2 in max pooling would yield an output volume of 16x16x12.

Image source original

Flattening: After the convolution and pooling of layers, the resulting feature maps are smoothed into a one-dimensional vector. This reformulation allows them to be sent to a fully connected layer suitable for solving categorical or regression problems.

Fully Connected Layers: This layer, located immediately before the output layer, receives input from the previous layer and performs computations for the final classification or regression task.

Output layer: The output of the fully connected layers is fed into a logistic function that effectively transforms the output of each class into an equivalent probability score indicating its classification probability, such as with sigmoid or softmax functions.

Advantages of Convolutional Neural Network

• Feature Learning: CNNs automatically learn hierarchical representations of features from input data. Convolutional layers work to extract low-level features like edges and textures, on the other hand, deeper layers learn high-level representations like shapes and patterns. This hierarchical feature extraction reduces the need for manual feature engineering.
• Spatial Hierarchies: CNNs preserve the spatial relationships between pixels in images due to their convolutional and pooling operations. This spatial awareness enables them to capture local patterns regardless of their location in the image.
• Parameter Sharing: By using shared weights via convolutional kernels, CNNs able to significantly reduce the number of trainable parameters, making them more efficient in terms of memory and computation. This sharing also allows them to generalize better to new, unseen data.
• Translation invariance: In CNN we have a translational invariance, it can detect patterns from images regardless of their location. It helps CNN in image translation, rotation and distortions.
• Pooling Layers: Through pooling layers (e.g., max pooling), CNNs mainly down-sample the feature maps, which reduces computational requirements while retaining essential information. This down-sampling aids in learning more robust and invariant features.
• Versatility and transfer learning: already trained CNN models on the big dataset can used in other tasks for feature extraction, therefore we use transfer learning in CNN for various aspects.
• State-of-the-Art Performance: image classification using CNN is one of the most used aspects of CNN architecture that it has achieved. Object detection, semantic segmentation, and other tasks also take benefits from CNN and shows their effectiveness in handling complex visual recognition challenges. 
• Scalability: CNNs can handle large-scale and high-resolution images which make them very useful in real-world applications in which we need to analyze large or high-dimensional data.

Disadvantages of Convolutional Neural Network

• Computational Complexity: CNN needs more computational power compared to the traditional approach the computation power increases as CNN becomes deeper, making CNN more computationally expensive and also resource-intensive. This complexity can hinder their deployment in resource-constrained environments.
• Large Data Requirements: CNNs generally require large labeled datasets for effective training. Inadequate data might lead to overfitting or less optimal performance of the model.
• updating & Hyperparameters: Choosing and tweaking suitable hyperparameters (such as kernel size and number of layers) is a challenging and time-consuming process in creating the best CNN architecture.
• Limitated interpretability: It is challenging to understand how and why CNNs create particular predictions because of their complex architecture. This interpretability issue might be problematic in an area where responsibility and openness are required.
• Overfitting: Using smaller datasets in particular, deeper CNN architectures with more parameters are more likely to overfit. Many times, regularization methods are required to avoid overfitting.
• Technical Specifications: Instruction Complex CNN models sometimes require specialized hardware (such as GPUs or TPUs), which not all users or applications may have easy access to or capacity for.
• Preprocessing and Augmenting of Data: To guarantee consistency, cleanliness, and sufficiency, data preparation for CNNs frequently requires substantial preprocessing and augmentation. Complex and time-consuming might be this procedure.
• Lack of Rotation and Scale Invariance: while CNNs possess translation invariance, they may not inherently generalize well to rotations or scale variations in images without additional techniques or augmentation.

Summary

At the forefront of modern computer vision, Convolutional Neural Networks (CNN) provide a sophisticated architecture designed to extract complex patterns from visual data. Basic components such as convolution and pooling layers play a key role in hierarchical feature extraction. Convolutional layers apply filters to input images and detect various visual features such as edges and textures. These features are then compressed by joining the layers, preserving important information while reducing computational complexity. These networks use hierarchical learning and progress through layers to identify increasingly complex patterns by combining lower-level features. The inclusion of non-linear activation functions such as Rectified Linear Units (ReLU) adds flexibility and complexity to the learning of the network, allowing it to model complex relationships within images.

The prowess of CNNs extends across a spectrum of image-related tasks, from CNN model for image classification and object detection to semantic segmentation. Despite their remarkable performance, CNNs demand substantial computational resources and extensive labeled datasets for training. Moreover, interpreting, CNNs stand as a pillar in reshaping machine understanding of visual information, propelling advancements in artificial intelligence, and bolstering applications in diverse domains, from healthcare and autonomous vehicles to robotics and beyond. Graph convolutional network have emerged as a powerful extension of convolutional neural networks, that offer advanced capabilities for processing graph-structured data.

Python Code