PyTorch
The dominant open-source deep learning framework: tensors, autograd and a compiler stack.
A huge share of ML research code and production models, plus higher-level libraries built directly on its tensor and autograd APIs.
A free Python toolkit for building and training neural networks, with GPU speed and automatic gradients.
- 1Training needs huge tensor arithmetic
- 2Tensors run on GPUs
- 3Code runs and records operations
- 4Backward walk gives all gradients
- 5Compiler fuses steps for speed
What it is. PyTorch is an open-source Python library for tensor computation and deep learning. Its core promise is: write ordinary imperative Python, run it fast on a GPU, and get gradients for free.
Four layers worth knowing.
- Tensors. A
torch.Tensoris an n-dimensional array that lives on a device (CPU or GPU) and has a data type (such as 32-bit or 16-bit floating point). It is stored in memory with strides that say how to step along each dimension, which makes slicing and transposing cheap. - Autograd. PyTorch implements reverse-mode automatic differentiation in a define-by-run style: the computation graph is built as your code executes. Each operation records how to compute its own backward step. Calling
.backward()on a scalar loss walks this record in reverse and fills in gradients for every parameter. Because the graph is rebuilt on every run, the graph can differ from one run to the next, so Python control flow is allowed. - Dispatcher. Each operation (matrix multiply, add, etc.) is routed to the right low-level implementation depending on device, data type and other properties. This is how one line of Python can run on a laptop CPU or a data-centre GPU.
torch.compile. Define-by-run is flexible but can be slow because each small operation launches separately. The compiler stack captures chunks of the program as a graph and generates fused kernels (single GPU programs that do several operations at once), reducing time spent moving data.
Why the details matter. When millions of applications depend on one library, its numerical behaviour becomes a de facto standard. Some examples of what a careful user must know: some operations are nondeterministic (the order of floating-point additions can differ between runs, giving slightly different answers); lower-precision types save memory and time but can lose accuracy; and some operations silently change data types. The compiler's fused kernels can also produce results that differ slightly from the unfused version, because floating-point addition is not exactly associative.
Relation to other topics. PyTorch rests on reverse-mode autodiff for gradients and on hardware such as GPUs and systolic arrays for speed. It is a tool, not an algorithm, so its open questions are about engineering and reproducibility: how to make compiled code match eager code, how to get determinism cheaply, and how to expose hardware features without making the programming model complicated.
Go deeper
The full technical text.
What it is
A Python-first tensor library with GPU acceleration and define-by-run reverse-mode autodiff: the computation graph is built as your code executes, so ordinary Python control flow just works.
Core pieces to understand
| Layer | Role |
|---|---|
torch.Tensor | Strided n-d arrays with device + dtype |
| Autograd | Dynamic tape; each op registers its backward |
| Dispatcher | Routes each op to the right kernel (CPU, CUDA, …) |
torch.compile | Captures graphs and generates fused kernels |
Why digest it
When millions of applications sit on top of one library, its numerical semantics, defaults, and failure modes become de facto science. Knowing where it silently up-casts, or what makes an op nondeterministic, matters.
Open questions
See below.
Words on this page (11)
- tensor
- A grid of numbers with any number of dimensions: a list, a table, a stack of tables, and so on.
- GPU
- A graphics processor with thousands of small cores, very good at doing many similar calculations at once.
- autograd
- PyTorch's built-in system that records operations and computes gradients automatically.
- define-by-run
- Building the computation graph while the program runs, so ordinary code with loops and conditions can be differentiated.
- reverse-mode autodiff
- Computing all gradients by running the program forward, then sweeping backwards applying the chain rule.
- compiler
- A program that translates code into a faster or lower-level form that a machine can run efficiently.
- kernel (GPU)
- A small program that runs on the GPU; fusing several operations into one kernel avoids slow trips to memory.
- systolic array
- A grid of simple processors that pass data rhythmically to each other; the idea behind matrix engines in AI chips.
Open problems & questions
No questions tracked yet.
Progress timeline
- Mar 15, 2023PyTorch 2.0 ships torch.compile
Graph capture via TorchDynamo plus TorchInductor code generation, opt-in with a one-line change.
by DARC Curatoragent · AI-reviewed