State-of-the-Art in Computer Vision: ViT, CNNs and Beyond

Since I graduated in 2022, the landscape of computer vision has been transformed by the rise of vision transformers (ViTs) and their hierarchical counterparts, which have challenged the dominance of convolutional neural networks (CNNs). CNNs, championed by architectures like ConvNeXt, remain highly competitive due to their inherent regularization properties and strong inductive biases. These biases, such as locality and translation equivariance, enable CNNs to efficiently model image data while requiring fewer training examples to generalize well. In the medical field, CNNs are still ubiquitous. The aforementioned biases match the problem quite well: finding the needle (cancer) in a haystack (an X-ray image).

But the same inductive biases inherently limit their flexibility when compared to vision transformers. ViTs, by treating images as sequences of patches, discard these biases in favor of learning global relationships directly from data. This flexibility has enabled vision transformers to excel in scenarios where vast amounts of labeled or pre-training data are available, leveraging their capacity to model long-range dependencies across the image. However, this does not work as well in medical imaging, where the available data is more limited and often has a very high resolution.

In this post, I’ll discuss my experience working with all those networks. How modern CNNs like ConvNext have evolved to remain relevant, how vision transformers such as Swin-V2 and PvT-V2 integrate hierarchical design principles to balance efficiency and expressivity, and how models like Hiera push the boundaries of self-supervised learning.

Locality assumes meaningful patterns or features can be found within small, localized regions of the input data. This concept is fundamental to how convolutional layers operate and why CNNs are so effective for spatially structured data like images. In practice, this is found in the size of the convolution kernels (33 or 55). As the filter slides through the image, only the pixels in the receptive field are considered.

In breast cancer, tumors often have a spiculated appearance. This star-shaped structure can be detected with this type of operation.

Translation equivariance means that when an input is translated (shifted) spatially, the feature map produced by the convolutional layer shifts by the same amount, preserving the spatial relationships. In other words the two operations commute:

\[\text{For any translation operator } T \text{ and convolution } conv, \\ conv(T(x)) = T(conv(x))\]

This is useful in the medical field. When you want to find a tumor in a large image (like a medical scan), the precise location of the tumor does not matter that much. If it’s translated to the right or left, it should produce the same feature map.

This assumes that complex patterns can be built hierarchically from simpler patterns (e.g., edges → textures → objects). In a way, locality is the fine-grained detection of edges in the image, and spatial hierarchy allows the reconstruction of more complex structures. In ImageNet, you would typically detect edges, then a dog’s ear, and then a dog. This was initially how it was presented with AlexNet.

This assumes that the same feature can occur across different parts of the input, so the same set of weights (filters) can be used everywhere.

Once again, in our cancer detection example, this is adequate. Cancer can appear in different parts of scans; its low-level features (edges, etc.) should be detected the same way.

Transformers were initially proposed as an NLP algorithm, where each word was treated as a token. In computer vision, you need to divide images into patches and treat those patches as tokens. Unlike CNNs, Vision Transformers do not rely on locality as an inductive bias. Instead, ViTs treat the input image as a sequence of patches, much like words in a sentence for natural language processing. This approach allows ViTs to learn global relationships between image regions from the start, without assuming that important features are localized.

This is both a curse and a blessing for Transformers. It’s a curse because the attention operation scales quadratically with the size of the image or as you reduce the size of the patches. An alternative is to use bigger patches for bigger images, but that limits the effective resolution of the patches processed in the transformer. This usually leads to poor performance. But this can also be a blessing because, if you can afford it, you can relax the inductive biases and generally get better performance.

ConvNext reimagines traditional CNNs with a modern twist, incorporating design elements inspired by vision transformers, such as large kernel sizes, depthwise convolutions, and layer normalization. While retaining the strong inductive biases of CNNs, ConvNext reduces these limitations by improving flexibility and scalability. This makes it highly competitive on benchmarks, offering a well-regularized alternative to transformers, especially in data-limited settings.

NFNets (Normalizer-Free Networks) are convolutional neural networks introduced by Google Research in 2021, and they share similarities with modern CNNs like ConvNeXt in their architectural improvements. Most notably, they remove the use of Batch Normalization (which has several problems of its own). They also incorporate several design features found in ConvNeXt, such as depthwise separable convolutions for efficiency. This operation splits the standard convolution into two steps. First, a depthwise convolution applies a single filter per input channel rather than aggregating across all channels. Second, a pointwise convolution aggregates depthwise information across all channels. It also proposes better initialization, optimization methods, etc.

Overall, scaling those models results in performance as good as that of Vision Transformers.

Similarly, Vision Transformers have undergone considerable modernization in the last couple of years.

Most noteworthy, and distilled in many variants, is the addition of a hierarchical structure to the network. This drastically reduces the computational cost of those networks. Swin-V2 introduces shifted windows, which enable hierarchical feature learning while maintaining global context through cross-window connections. This mechanism avoids redundant computations by focusing on non-overlapping regions of the input, significantly reducing complexity compared to traditional attention mechanisms.

On the other hand, PvT-V2 achieves a hierarchical structure by progressively reducing spatial dimensions through patch-merging layers, allowing the model to capture multi-scale features efficiently. These hierarchical designs not only improve scalability but also enhance performance on tasks requiring both fine-grained details and global understanding, making them highly effective for large-scale vision applications. In my work, I found PvT-V2 to be an outstanding model, offering high accuracy with an unusually small number of parameters and low computational cost. For example, at a similar performance level, PvT-V2 would take half the memory of ConvNeXt and train twice as fast.

Let’s mention something that is becoming ubiquitous in Transformers: Flash Attention. This is now a default setting in PyTorch if the hardware can support it. Flash Attention is an optimized attention mechanism designed to reduce memory usage and computational overhead, particularly for large models. Unlike traditional attention mechanisms, which have quadratic memory complexity with respect to the sequence length, Flash Attention leverages efficient memory access patterns and kernel optimizations to compute attention in a memory-efficient way. By implementing a more compact representation of the attention matrix and using hardware acceleration techniques, Flash Attention can handle longer sequences with significantly lower memory consumption and faster processing speeds, making it ideal for tasks with large-scale inputs, such as transformers applied to vision and language models. This makes it a promising technique for scaling up models without compromising performance.

• A lot of discussion revolves around how transformers work. There is an overall impression (which I share) that transformers learn more texture-based than shape-based features when compared with CNNs. This is due to the inductive biases we just mentioned.
• ViTs excel in tasks where global context or relationships are crucial, especially when sufficient data and computational resources are available. CNNs are more efficient and effective when local patterns dominate the task (e.g., detecting edges or textures) or when computational resources are limited.

With ChatGPT, SAM2, and more AI-powered solutions, I’ve noticed a shift toward making these models more readily available to end users. This is a welcome change, but it comes with challenges in making the demo compelling. This takes more engineering effort than research effort. I predict that this trend will consolidate and that we will see more applications that leverage the low latency of modern hardware.

I also suspect that self-supervision will become more prevalent, especially for tackling scenarios where data is not available. In the medical field, this could involve a new scanner type. If you have trained an AI algorithm, you would like it to also work well on newer, more sophisticated versions of the scanner. But of course, since it is newer, you don’t have as much training data available. Pre-training can help level that field.

Finally, as much as I would like to see a change in paradigm, it looks like a moonshot. I think the sequence-to-sequence approach in ViT is ill-suited for vision, but it clearly yields remarkable performance.