CNN vs. Transformer: Global Context and NVIDIA DLSS

CNN vs. Transformer: The Key Difference in AI Image Processing

Artificial intelligence has revolutionized how computers understand images. Two of the most important architectures are Convolutional Neural Networks (CNNs) and Transformers. While CNNs were the standard in computer vision for years, Transformers are now gaining ground – including in NVIDIA's DLSS technology. This article explains the fundamental differences and why Transformers offer superior global context.

CNNs: The Local Perspective

Convolutional Neural Networks (CNNs) use filters (kernels) that slide over the image like a magnifying glass. They analyze only small, neighboring pixel regions. In early layers, they detect simple features like edges or textures. Only through many stacked layers can they assemble more complex objects. The downside: CNNs struggle to capture relationships between distant image regions. An object in the top-left and one in the bottom-right are not directly linked unless the network is very deep. This local view limits contextual understanding.

Transformers: The Global Context

Transformers, originally developed for natural language processing, changed the paradigm. Instead of scanning the image pixel by pixel, they divide it into small patches (e.g., 16×16 pixels) and treat them like words in a sentence. The key mechanism is self-attention. Each patch is compared with every other patch in the entire image. The model computes an attention score for each pair, indicating relevance. Thus, a patch in the top-left can directly receive information from a patch in the bottom-right – regardless of spatial distance. This creates a global understanding of the image.

Why Transformers Have Better Context

The crucial advantage of Transformers lies in their ability to weight every element in the context of all other elements. In CNNs, weighting is determined by local neighborhoods. In Transformers, each word (or patch) is represented as a weighted sum of all others. This means: when the model needs to reconstruct a detail, it knows exactly which other parts of the image are relevant. This contextual weighting leads to a more coherent and detailed representation. Especially in tasks like super-resolution or denoising, this advantage becomes evident.

Comparison CNN vs. Transformer

NVIDIA DLSS: The Shift from CNN to Transformer

NVIDIA developed Deep Learning Super Sampling (DLSS) to render games at a lower resolution and upscale them using AI. The early versions (DLSS 1, 2, and 3) were based on CNNs. They analyzed local pixel clusters to estimate missing details. Over time, CNNs hit their limits: artifacts like ghosting (trails behind moving objects) and flickering appeared because temporal coherence between frames was not well maintained.

With DLSS 4, NVIDIA introduced a Transformer-based model. This considers not only the current frame but also the relationships between consecutive frames globally. Self-attention allows it to maintain motion and textures consistently over time. The result: sharper images, fewer artifacts, and significantly improved stability during fast camera movements. Although Transformers are more computationally demanding, NVIDIA optimized the architecture (e.g., reduced VRAM usage) so that performance impact is minimal. The shift demonstrates how important global context is for image quality.

Conclusion

CNNs and Transformers differ fundamentally in their approach: CNNs focus on local patterns, Transformers on global relationships. The self-attention of Transformers enables context-aware weighting of all image elements, leading to superior image quality. NVIDIA has recognized this advantage and now uses Transformers for DLSS in gaming. This trend will continue as demand for realistic and smooth graphics grows. For developers and users, it means: Transformers are not just for language but also for images – they are the future.