TensorRT Overview

? Useful Links
NVIDIA TRT: https://nvda.ws/3z9uSEc
TRT on GITHUB: https://github.com/NVIDIA/TensorRT
Jarvis/Riva https://youtu.be/sbYolIax190
w/ love ❤️

? About
NVIDIA® TensorRTTM ( https://github.com/NVIDIA/TensorRT ) is a deep learning inference SDK with outstanding performance. It provides a deep learning inference optimizer and runtime for deep learning inference applications with low latency and high throughput.

During inference, TensorRT-based apps are up to 40 times faster than CPU-only systems. You may use TensorRT to improve neural network models trained in all major frameworks, calibrate for reduced precision while maintaining high accuracy, and deploy to hyperscale data centers, embedded systems, or automotive product platforms.

TensorRT is based on CUDA®, NVIDIA's parallel programming model, and allows you to optimize inference using CUDA-XTM libraries, development tools, and technologies for AI, autonomous machines, high-performance computing, and graphics. TensorRT takes advantage of sparse tensor cores on upcoming NVIDIA Ampere Architecture GPUs, delivering an additional performance increase.

For production deployments of deep learning inference applications such as video streaming, speech recognition, recommendation, fraud detection, text generation, and natural language processing, TensorRT provides INT8 using Quantization Aware Training and Post Training Quantization, as well as FP16 optimizations. Reduced precision inference cuts application latency in half, which is essential for many real-time services, as well as autonomous and embedded applications.

?️ Chapters:
00:00 Intro to TensorRT
02:20 Prerequisites
03:20 TensorRT Docker Images
06:27 Jupyter Lab within Docker Containers
07:25 Compile TRT OSS
08:26 HuggingFace GPT-2
13:42 PyTorch on CPU/GPU vs TensorRT on GPU
16:42 Outro

? Credits:
Music: https://www.bensound.com/royalty-free-music/track/hey-happy-cheerful
NVIDIA TensorRT: https://github.com/NVIDIA/TensorRT
Description: https://bazziahmad.com/2021/11/21/nvidia-tensorrt/

#nvidia #tensorRT #pytorch