Conv2D on GPUs: Why the Winning Trick Is Matrix Multiplication

A 1×1 convolution is a matrix multiply in disguise, and a 3×3 convolution is nine of them accumulated together. That is the core claim of a new engineering guide from Semiconductor Engineering, which argues that one of machine learning's foundational operations is best implemented not as a unique primitive but as a structured matrix-multiplication pattern tuned for cache behavior.

The stakes are practical. Conv2D appears throughout inference pipelines, image models, and feature extraction stages. Yet the hard part is rarely the mathematics. It is the execution strategy: how data is laid out, how work is grouped, and how memory traffic is controlled.

The article's central insight is that treating convolution as a close relative of matrix multiplication lowers engineering effort, cuts maintenance cost, and increases the chance that future hardware and software improvements benefit both matmul and convolution paths at once.

Why does 1×1 convolution behave like matmul?

A standard Conv2D applies a learned filter tensor across an input tensor to produce an output. The input has shape N × H × W × C_in, the weights have shape H_filter × W_filter × C_in × C_out, and the output has shape N × H_out × W_out × C_out.

The 1×1 case is the cleanest demonstration of the analogy. At each spatial position, the input contributes a feature vector of length C_in, and the weights map it into an output vector of length C_out. That is exactly a matrix multiply from C_in to C_out.

The layout matters here. If the input is already stored in NHWC order, each spatial feature vector naturally forms rows of a matrix without any explicit rearrangement. The implementation can produce NHWC output with no transposition step. This matters because unnecessary data movement is one of the fastest ways to lose performance.

What changes when the filter grows to K×K?

Once the filter exceeds 1×1, each output element depends on multiple spatial locations. A 3×3 convolution combines nine separate spatial contributions. Conceptually, the problem becomes a set of matrix-like products accumulated together.

That reframing shifts the engineering question. The task is no longer just "do matrix multiplication" but "do repeated matrix-like accumulation while minimizing memory waste." A naive implementation can achieve good arithmetic utilization yet repeatedly flush and reload input data through the cache hierarchy. For a 3×3 filter, there are nine C_in × C_out contributions per output location, which makes execution order critical.

Loop ordering is therefore an optimization decision in its own right. A straightforward ordering that treats each filter position as a separate matrix multiply is simple, but it can force the same input region to be fetched repeatedly into L1 cache, and potentially L2 as well. Reordering loops improves locality, reduces refetch traffic, and increases effective cache reuse. The arithmetic stays the same; the memory system sees a friendlier pattern.

What does a representative kernel look like?

The guide walks through a sample loop nest with a broadly recognizable structure: load weights, compute input coordinates, read vectorized input values, synchronize local memory, and accumulate partial products. The performance, it argues, usually lives in specific places:

• Vectorized reads instead of scalar traffic
• Local-memory staging for reusable weights
• Structured accumulation over filter positions
• Explicit synchronization around shared data
• Thread-level parallelism across multiple outputs

The optimization principle the author draws from this: when a kernel is already compute-valid, restructuring loop nesting to improve data reuse is often a higher-value change than low-level instruction tuning.

How do padding, stride, and dilation shape throughput?

Zero padding is the most common strategy, and the reference implementation reflects that. A typical boundary-safe pattern checks whether each computed input coordinate falls inside valid image bounds before issuing the read. This is correct and easy to reason about, but a branch inside a hot loop can be expensive. An alternative redirects out-of-range reads to a dummy zero-filled line or buffer, trading branch cost for setup logic.

Filter size and stride act in less obvious ways. Larger filters generally increase input reuse, which can improve cache efficiency. Larger strides reduce reuse across neighboring outputs and can make memory behavior less favorable. Neither effect is absolute, but both shift the balance between arithmetic work and memory traffic — the balance where many Conv2D kernels are won or lost.

What are the common pitfalls?

The guide lists mistakes that recur when engineers turn Conv2D into a matrix-like GPU kernel:

• Assuming arithmetic cost is the main bottleneck when memory traffic dominates
• Introducing explicit data transforms before checking whether the source layout already works
• Keeping boundary branches inside the hottest loop without measuring their impact
• Optimizing instruction details before validating cache behavior and access ordering
• Ignoring how stride changes data reuse patterns

The caution attached to this list is blunt: a mathematically elegant implementation is not always a performant one. Equivalent formulations can behave very differently on real hardware.

What is the transferable method?

The most durable value, the author argues, is not any single micro-optimization but the method: start from a clear tensor interpretation, identify when the operation maps naturally onto matmul, preserve data layout to avoid unnecessary movement, focus early effort on loop ordering and cache reuse, and treat padding, stride, and boundary logic as performance-relevant design choices.

The recommended mental model has three layers: a math layer that defines what gets multiplied and accumulated, a layout layer that determines how values sit in memory, and an execution layer that sets the order of work so caches, local memory, and vector units run efficiently. Strong kernels come from aligning all three layers, not from over-optimizing one.

For engineers building GPU inference and tensor kernels, the closing lesson is that good performance does not begin with heroic low-level tuning — it begins with choosing the right interpretation of the problem, which in this case puts locality, loop order, and bandwidth efficiency ahead of instruction-level polish.