Course Outline

Introduction

  • Overview of deep learning scaling challenges
  • Overview of DeepSpeed and its features
  • DeepSpeed vs. other distributed deep learning libraries

Getting Started

  • Setting up the development environment
  • Installing PyTorch and DeepSpeed
  • Configuring DeepSpeed for distributed training

DeepSpeed Optimization Features

  • DeepSpeed training pipeline
  • ZeRO (memory optimization)
  • Activation checkpointing
  • Gradient checkpointing
  • Pipeline parallelism

Scaling Models with DeepSpeed

  • Basic scaling using DeepSpeed
  • Advanced scaling techniques
  • Performance considerations and best practices
  • Debugging and troubleshooting techniques

Advanced DeepSpeed Topics

  • Advanced optimization techniques
  • Using DeepSpeed with mixed precision training
  • DeepSpeed on different hardware (e.g. GPUs, TPUs)
  • DeepSpeed with multiple training nodes

Integrating DeepSpeed with PyTorch

  • Integrating DeepSpeed with PyTorch workflows
  • Using DeepSpeed with PyTorch Lightning

Troubleshooting

  • Debugging common DeepSpeed issues
  • Monitoring and logging

Summary and Next Steps

  • Recap of key concepts and features
  • Best practices for using DeepSpeed in production
  • Further resources for learning more about DeepSpeed

Requirements

  • Intermediate knowledge of deep learning principles
  • Experience with PyTorch or similar deep learning frameworks
  • Familiarity with Python programming

Audience

  • Data scientists
  • Machine learning engineers
  • Developers
 21 Hours

Number of participants



Price per participant

Related Courses

Advanced Stable Diffusion: Deep Learning for Text-to-Image Generation

21 Hours

Introduction to Stable Diffusion for Text-to-Image Generation

21 Hours

AlphaFold

7 Hours

TensorFlow Lite for Embedded Linux

21 Hours

TensorFlow Lite for Android

21 Hours

TensorFlow Lite for iOS

21 Hours

Tensorflow Lite for Microcontrollers

21 Hours

Deep Learning Neural Networks with Chainer

14 Hours

Distributed Deep Learning with Horovod

7 Hours

Accelerating Deep Learning with FPGA and OpenVINO

35 Hours

Building Deep Learning Models with Apache MXNet

21 Hours

Deep Learning with Keras

21 Hours

Advanced Deep Learning with Keras and Python

14 Hours

Deep Learning for Self Driving Cars

21 Hours

Torch for Machine and Deep Learning

21 Hours

Related Categories

1