Mathematical Foundations / Linear Algebra Basics

Linear Algebra Basics

In Progress 1 min read Updated Jul 15, 2025

Vectors

A vector is an ordered list of numbers1.

Vectors can also be thought of as points in n-dimensional space, which is useful for understanding embeddings. ↩

. In deep learning, vectors represent data points, features, and model parameters.

x=[x1x2⋮xn]\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}

Matrices

A matrix is a 2D array of numbers. Weight matrices are core to neural networks.

W=[w11w12w21w22]\mathbf{W} = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix}

Matrix Multiplication

The dot product of two matrices A∈Rm×n\mathbf{A} \in \mathbb{R}^{m \times n} and B∈Rn×p\mathbf{B} \in \mathbb{R}^{n \times p} produces C∈Rm×p\mathbf{C} \in \mathbb{R}^{m \times p}:

Cij=∑k=1nAikBkjC_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}

This operation is the backbone of forward passes in neural networks2.

GPUs are specifically designed to perform thousands of matrix multiplications in parallel, which is why they’re so important for deep learning. ↩

. Understanding matrix multiplication is essential before moving on to automatic differentiation.

Related Notes

Other notes in the same chapter or with shared tags

Automatic Differentiation
Next
Automatic Differentiation
Mathematical Foundations