Essential Tricks In Deep Learning

Master key techniques to enhance your deep learning skills. The repository is a concise guide to essential tricks for optimizing model performance.

Table of Contents

Introduction to Neural Networks and Model Building:

1. Building Your First Neural Network - Introduction to neural networks for beginners.
2. Pixels To Prediction Using ConvNet - Building and training your first Convolutional Neural Network.

Data Loading and Processing

1. Memory Efficient DataLoader And Parallelization - Optimize data loading and processing for memory efficiency.
2. Data Augmentation Using torchvision - Implement the data augmentation techniques using the torchvision library.

Model Initialization and Regularization:

1. Decoding Dropout Technique - Unravel the dropout technique and its impact on model training.
2. Optimizing Convolutions With Batchnorm - Enhance model efficiency using batch normalization with convolutions.
3. Improving Generalization With Label Smoothing - Enhance model generalization using label smoothing.
4. Exploring Model Initialization Strategies - Dive into various strategies for initializing neural network weights.

Training Schedule and Strategies:

1. Dynamic LR Scheduling - Implement dynamic learning rate scheduling techniques.
2. Optimizing Learning Rate With LR Finder - Optimize learning rates using a learning rate finder.
3. Warmup Strategies With Cosine Annealing - Implement warmup strategies with cosine annealing for better convergence.
4. Early Stopping Strategies For Training - Selection of a reasonably optimal model.

Gradient Optimization

1. Adaptive Gradient Clipping - Explore gradient clipping and adaptive gradient clipping techniques.
2. Smoothing Gradients With Gradient Penalty - Implement gradient penalty for smoother training gradients.
3. Accumulating Gradient For Efficient Training - Optimize training efficiency by accumulating gradients.
4. Controlling Overfitting With Weight Decay - Mitigate overfitting using weight decay and the AdamW optimizer.
5. Memory Efficient Models With Checkpointing - Efficient memory usage in training.

Precision and Efficiency

1. Automatic Mixed Precision Training - Accelerate training by using a combination of lower-precision and higher-precision numerical representations.

Experiment Management and Monitoring

1. Dynamic Progress Bar Using tqdm - Explore model initialization strategies and minor bug fixes.
2. Ensuring Experiment Reproducibility - Implement practices to ensure reproducibility in your experiments.
3. Effective Model Checkpointing - Selecting the best-performing model.
4. Experiment Tracking With mlflow - mlflow for Experiment Tracking
5. Logging Model Parameters - Logging model parameters - flops, trainable params, size etc.
6. Understanding CUDA Memory Usage - Manage GPU Usage.

Hyperparameter Optimization

1. Hyperparameter Tuning With optuna - Automate the optimization process of hyperparameters.
2. Hyperparameter Tuning With optuna - Training - Notebook demonstrating training with optimal hyperparams.

Model Evaluation and Analysis

1. Deep Dive Into Error Analysis - How to calculate and evaluate your model.
2. Understanding Confusion Matrix in Deep Learning - Implement confusion matrix.
3. Classwise Metrics for Model Evaluation - How to calculate class wise metrics.
4. Model Interpretibilit With captum - Exploring Model Insights with Interpretability.

Performance Optimization

1. torch.compile - latest method to speed up your PyTorch code! Run PyTorch code faster by JIT-compiling PyTorch code into optimized kernels, all while requiring minimal code changes.
2. Model Serialization and Export - Multiple model export strategies like state dict, ONNX, and TorchScript. How to convert PyTorch models to TensorFlow and TensorFlow Lite, with guidance on inference execution.

Transfer Learning

1. Going Deeper with Transfer Learning - Leverage knowledge gained from solving one problem to improve performance on another problem.
2. Freeze Unfreeze Backbone - Selectively enabling or disabling the training of specific layers
3. Differential Learning Rate - Different learning rates are applied to different parameters.
4. Layerwise LR Decay - Training deep neural networks to adjust the learning rate for each layer individually.

Advanced Training Techniques

1. Stochastic Weight Averaging For Improved Convergence - Ensemble model via weight averaging.
2. Improving Stability With EMA - Stabilize training and improve model performance.
3. Progressive Resizing - Adjust the size of images progressively based on the epoch.
4. Online Hard Negative Mining - Prioritize hard examples during training.

Testing

1. Enhancing Testing with Test Time Augmentation - Enhancing model predictions with augmentation.

YouTube Playlist

Check out the corresponding YouTube playlist for video tutorials on these techniques.

Getting Started

Clone the repository and explore the notebooks to level up your deep learning skills.

git clone https://github.com/your_username/essential-tricks-in-deeplearning.git
cd essential-tricks-in-deeplearning

Contributing

We welcome contributions from the community! If you have suggestions, bug reports, or want to add new tricks to the repository, follow these steps:

1. Fork the repository.
2. Create a new branch: git checkout -b feature/new-trick.
3. Make your changes and commit: git commit -m 'Add new trick: Feature Name'.
4. Push to the branch: git push origin feature/new-trick.
5. Open a pull request.

Resources

Articles

• Deep Learning Tips and Tricks cheatsheet
• Performance Tuning Guide
• Speed Up Model Training
• EFFECTIVE TRAINING TECHNIQUES
• Methods and tools for efficient training on a single GPU
• 5 Must-Have Tricks When Training Neural Networks
• Deep Learning Tips and Tricks
• Neural Network Tips and Tricks
• A bunch of tips and tricks for training deep neural networks

GitHub

• Deep Learning Tuning Playbook
• Deep Learning Tricks
• Deep Learning Training Tricks
• Tricks In Deep Learning
• Tricks used in Deep Learning
• Efficient Deep Learning
• Deep Learning Tips and Tricks
• Torch Memory-adaptive Algorithms (TOMA)
• Trading compute for memory in PyTorch models using Checkpointing

Papers

• Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study
• Tricks From Deep Learning

To-Do

Extras -

• Distributed Training
• TensorBoard Logging
• Adversarial training
• Autoscaling Batch Size
• Out of Fold Prediction
• Self Distillation
• OneCycleLR
• Snapshot ensembles
• Focal loss
• Drop path