dropouts are more specific and aggressive in the paper, thanks for letting me know @hilach70

README.md

@@ -1,13 +1,9 @@
-<img src="./vit.gif" width="500px"></img>
+<img src="./vit.png" width="500px"></img>
 
 ## Vision Transformer - Pytorch
 
 Implementation of <a href="https://openreview.net/pdf?id=YicbFdNTTy">Vision Transformer</a>, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch. Significance is further explained in <a href="https://www.youtube.com/watch?v=TrdevFK_am4">Yannic Kilcher's</a> video. There's really not much to code here, but may as well lay it out for everyone so we expedite the attention revolution.
 
-For a Pytorch implementation with pretrained models, please see Ross Wightman's repository <a href="https://github.com/rwightman/pytorch-image-models">here</a>.
-
-The official Jax repository is <a href="https://github.com/google-research/vision_transformer">here</a>.
-
 ## Install
 
 ```bash
@@ -26,7 +22,7 @@ v = ViT(
     num_classes = 1000,
     dim = 1024,
     depth = 6,
-    heads = 16,
+    heads = 8,
     mlp_dim = 2048,
     dropout = 0.1,
     emb_dropout = 0.1
@@ -38,109 +34,6 @@ mask = torch.ones(1, 8, 8).bool() # optional mask, designating which patch to at
 preds = v(img, mask = mask) # (1, 1000)
 ```
 
-## Parameters
-- `image_size`: int.  
-Image size. If you have rectangular images, make sure your image size is the maximum of the width and height
-- `patch_size`: int.  
-Number of patches. `image_size` must be divisible by `patch_size`.  
-The number of patches is: ` n = (image_size // patch_size) ** 2` and `n` **must be greater than 16**.
-- `num_classes`: int.  
-Number of classes to classify.
-- `dim`: int.  
-Last dimension of output tensor after linear transformation `nn.Linear(..., dim)`.
-- `depth`: int.  
-Number of Transformer blocks.
-- `heads`: int.  
-Number of heads in Multi-head Attention layer. 
-- `mlp_dim`: int.  
-Dimension of the MLP (FeedForward) layer. 
-- `channels`: int, default `3`.  
-Number of image's channels. 
-- `dropout`: float between `[0, 1]`, default `0.`.  
-Dropout rate. 
-- `emb_dropout`: float between `[0, 1]`, default `0`.  
-Embedding dropout rate.
-- `pool`: string, either `cls` token pooling or `mean` pooling
-
-## Distillation
-
-<img src="./distill.png" width="300px"></img>
-
-A recent <a href="https://arxiv.org/abs/2012.12877">paper</a> has shown that use of a distillation token for distilling knowledge from convolutional nets to vision transformer can yield small and efficient vision transformers. This repository offers the means to do distillation easily.
-
-ex. distilling from Resnet50 (or any teacher) to a vision transformer
-
-```python
-import torch
-from torchvision.models import resnet50
-
-from vit_pytorch.distill import DistillableViT, DistillWrapper
-
-teacher = resnet50(pretrained = True)
-
-v = DistillableViT(
-    image_size = 256,
-    patch_size = 32,
-    num_classes = 1000,
-    dim = 1024,
-    depth = 6,
-    heads = 8,
-    mlp_dim = 2048,
-    dropout = 0.1,
-    emb_dropout = 0.1
-)
-
-distiller = DistillWrapper(
-    student = v,
-    teacher = teacher,
-    temperature = 3,           # temperature of distillation
-    alpha = 0.5                # trade between main loss and distillation loss
-)
-
-img = torch.randn(2, 3, 256, 256)
-labels = torch.randint(0, 1000, (2,))
-
-loss = distiller(img, labels)
-loss.backward()
-
-# after lots of training above ...
-
-pred = v(img) # (2, 1000)
-```
-
-The `DistillableViT` class is identical to `ViT` except for how the forward pass is handled, so you should be able to load the parameters back to `ViT` after you have completed distillation training.
-
-You can also use the handy `.to_vit` method on the `DistillableViT` instance to get back a `ViT` instance.
-
-```python
-v = v.to_vit()
-type(v) # <class 'vit_pytorch.vit_pytorch.ViT'>
-```
-
-## Token-to-Token ViT
-
-<img src="./t2t.png" width="400px"></img>
-
-<a href="https://arxiv.org/abs/2101.11986">This paper</a> proposes that the first couple layers should downsample the image sequence by unfolding, leading to overlapping image data in each token as shown in the figure above. You can use this variant of the `ViT` as follows.
-
-```python
-import torch
-from vit_pytorch.t2t import T2TViT
-
-v = T2TViT(
-    dim = 512,
-    image_size = 224,
-    depth = 5,
-    heads = 8,
-    mlp_dim = 512,
-    num_classes = 1000,
-    t2t_layers = ((7, 4), (3, 2), (3, 2)) # tuples of the kernel size and stride of each consecutive layers of the initial token to token module
-)
-
-img = torch.randn(1, 3, 224, 224)
-v(img) # (1, 1000)
-```
-
 ## Research Ideas
 
 ### Self Supervised Training
@@ -171,7 +64,7 @@ model = ViT(
 learner = BYOL(
     model,
     image_size = 256,
-    hidden_layer = 'to_latent'
+    hidden_layer = 'to_cls_token'
 )
 
 opt = torch.optim.Adam(learner.parameters(), lr=3e-4)
@@ -197,22 +90,23 @@ A pytorch-lightning script is ready for you to use at the repository link above.
 
 There may be some coming from computer vision who think attention still suffers from quadratic costs. Fortunately, we have a lot of new techniques that may help. This repository offers a way for you to plugin your own sparse attention transformer.
 
-An example with <a href="https://arxiv.org/abs/2102.03902">Nystromformer</a>
+An example with <a href="https://arxiv.org/abs/2006.04768">Linformer</a>
 
 ```bash
-$ pip install nystrom-attention
+$ pip install linformer
 ```
 
 ```python
 import torch
 from vit_pytorch.efficient import ViT
-from nystrom_attention import Nystromformer
+from linformer import Linformer
 
-efficient_transformer = Nystromformer(
+efficient_transformer = Linformer(
     dim = 512,
+    seq_len = 4096 + 1,  # 64 x 64 patches + 1 cls token
     depth = 12,
     heads = 8,
-    num_landmarks = 256
+    k = 256
 )
 
 v = ViT(
@@ -229,94 +123,16 @@ v(img) # (1, 1000)
 
 Other sparse attention frameworks I would highly recommend is <a href="https://github.com/lucidrains/routing-transformer">Routing Transformer</a> or <a href="https://github.com/lucidrains/sinkhorn-transformer">Sinkhorn Transformer</a>
 
-### Combining with other Transformer improvements
-
-This paper purposely used the most vanilla of attention networks to make a statement. If you would like to use some of the latest improvements for attention nets, please use the `Encoder` from <a href="https://github.com/lucidrains/x-transformers">this repository</a>.
-
-ex.
-
-```bash
-$ pip install x-transformers
-```
-
-```python
-import torch
-from vit_pytorch.efficient import ViT
-from x_transformers import Encoder
-
-v = ViT(
-    dim = 512,
-    image_size = 224,
-    patch_size = 16,
-    num_classes = 1000,
-    transformer = Encoder(
-        dim = 512,                  # set to be the same as the wrapper
-        depth = 12,
-        heads = 8,
-        ff_glu = True,              # ex. feed forward GLU variant https://arxiv.org/abs/2002.05202
-        residual_attn = True        # ex. residual attention https://arxiv.org/abs/2012.11747
-    )
-)
-
-img = torch.randn(1, 3, 224, 224)
-v(img) # (1, 1000)
-```
-
-## Resources
-
-Coming from computer vision and new to transformers? Here are some resources that greatly accelerated my learning.
-
-1. <a href="http://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a> - Jay Alammar
-
-2. <a href="http://peterbloem.nl/blog/transformers">Transformers from Scratch</a>  - Peter Bloem
-
-3. <a href="https://nlp.seas.harvard.edu/2018/04/03/attention.html">The Annotated Transformer</a> - Harvard NLP
-
-
 ## Citations
 
 ```bibtex
-@misc{dosovitskiy2020image,
-    title   = {An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
-    author  = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
-    year    = {2020},
-    eprint  = {2010.11929},
-    archivePrefix = {arXiv},
-    primaryClass = {cs.CV}
+@inproceedings{
+    anonymous2021an,
+    title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
+    author={Anonymous},
+    booktitle={Submitted to International Conference on Learning Representations},
+    year={2021},
+    url={https://openreview.net/forum?id=YicbFdNTTy},
+    note={under review}
 }
 ```
-
-```bibtex
-@misc{touvron2020training,
-    title   = {Training data-efficient image transformers & distillation through attention}, 
-    author  = {Hugo Touvron and Matthieu Cord and Matthijs Douze and Francisco Massa and Alexandre Sablayrolles and Hervé Jégou},
-    year    = {2020},
-    eprint  = {2012.12877},
-    archivePrefix = {arXiv},
-    primaryClass = {cs.CV}
-}
-```
-
-```bibtex
-@misc{yuan2021tokenstotoken,
-      title     = {Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet},
-      author    = {Li Yuan and Yunpeng Chen and Tao Wang and Weihao Yu and Yujun Shi and Francis EH Tay and Jiashi Feng and Shuicheng Yan},
-      year      = {2021},
-      eprint    = {2101.11986},
-      archivePrefix = {arXiv},
-      primaryClass = {cs.CV}
-}
-```
-
-```bibtex
-@misc{vaswani2017attention,
-    title   = {Attention Is All You Need},
-    author  = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},
-    year    = {2017},
-    eprint  = {1706.03762},
-    archivePrefix = {arXiv},
-    primaryClass = {cs.CL}
-}
-```
-
-*I visualise a time when we will be to robots what dogs are to humans, and I’m rooting for the machines.* — Claude Shannon

setup.py

@@ -3,7 +3,7 @@ from setuptools import setup, find_packages
 setup(
   name = 'vit-pytorch',
   packages = find_packages(exclude=['examples']),
-  version = '0.7.2',
+  version = '0.2.2',
   license='MIT',
   description = 'Vision Transformer (ViT) - Pytorch',
   author = 'Phil Wang',

vit_pytorch/distill.py

@@ -1,128 +0,0 @@
-import torch
-import torch.nn.functional as F
-from torch import nn
-from vit_pytorch.vit_pytorch import ViT
-from vit_pytorch.efficient import ViT as EfficientViT
-
-from einops import rearrange, repeat
-
-# helpers
-
-def exists(val):
-    return val is not None
-
-# classes
-
-class DistillMixin:
-    def forward(self, img, distill_token = None, mask = None):
-        distilling = exists(distill_token)
-        x = self.to_patch_embedding(img)
-        b, n, _ = x.shape
-
-        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
-        x = torch.cat((cls_tokens, x), dim = 1)
-        x += self.pos_embedding[:, :(n + 1)]
-
-        if distilling:
-            distill_tokens = repeat(distill_token, '() n d -> b n d', b = b)
-            x = torch.cat((x, distill_tokens), dim = 1)
-
-        x = self._attend(x, mask)
-
-        if distilling:
-            x, distill_tokens = x[:, :-1], x[:, -1]
-
-        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]
-
-        x = self.to_latent(x)
-        out = self.mlp_head(x)
-
-        if distilling:
-            return out, distill_tokens
-
-        return out
-
-class DistillableViT(DistillMixin, ViT):
-    def __init__(self, *args, **kwargs):
-        super(DistillableViT, self).__init__(*args, **kwargs)
-        self.args = args
-        self.kwargs = kwargs
-        self.dim = kwargs['dim']
-        self.num_classes = kwargs['num_classes']
-
-    def to_vit(self):
-        v = ViT(*self.args, **self.kwargs)
-        v.load_state_dict(self.state_dict())
-        return v
-
-    def _attend(self, x, mask):
-        x = self.dropout(x)
-        x = self.transformer(x, mask)
-        return x
-
-class DistillableEfficientViT(DistillMixin, EfficientViT):
-    def __init__(self, *args, **kwargs):
-        super(DistillableEfficientViT, self).__init__(*args, **kwargs)
-        self.args = args
-        self.kwargs = kwargs
-        self.dim = kwargs['dim']
-        self.num_classes = kwargs['num_classes']
-
-    def to_vit(self):
-        v = EfficientViT(*self.args, **self.kwargs)
-        v.load_state_dict(self.state_dict())
-        return v
-
-    def _attend(self, x, mask):
-        return self.transformer(x)
-
-# knowledge distillation wrapper
-
-class DistillWrapper(nn.Module):
-    def __init__(
-        self,
-        *,
-        teacher,
-        student,
-        temperature = 1.,
-        alpha = 0.5
-    ):
-        super().__init__()
-        assert (isinstance(student, (DistillableViT, DistillableEfficientViT))) , 'student must be a vision transformer'
-
-        self.teacher = teacher
-        self.student = student
-
-        dim = student.dim
-        num_classes = student.num_classes
-        self.temperature = temperature
-        self.alpha = alpha
-
-        self.distillation_token = nn.Parameter(torch.randn(1, 1, dim))
-
-        self.distill_mlp = nn.Sequential(
-            nn.LayerNorm(dim),
-            nn.Linear(dim, num_classes)
-        )
-
-    def forward(self, img, labels, temperature = None, alpha = None, **kwargs):
-        b, *_ = img.shape
-        alpha = alpha if exists(alpha) else self.alpha
-        T = temperature if exists(temperature) else self.temperature
-
-        with torch.no_grad():
-            teacher_logits = self.teacher(img)
-
-        student_logits, distill_tokens = self.student(img, distill_token = self.distillation_token, **kwargs)
-        distill_logits = self.distill_mlp(distill_tokens)
-
-        loss = F.cross_entropy(student_logits, labels)
-
-        distill_loss = F.kl_div(
-            F.log_softmax(distill_logits / T, dim = -1),
-            F.softmax(teacher_logits / T, dim = -1).detach(),
-        reduction = 'batchmean')
-
-        distill_loss *= T ** 2
-
-        return loss * alpha + distill_loss * (1 - alpha)

vit_pytorch/efficient.py

@@ -1,43 +1,40 @@
 import torch
+from einops import rearrange
 from torch import nn
-from einops import rearrange, repeat
-from einops.layers.torch import Rearrange
 
 class ViT(nn.Module):
-    def __init__(self, *, image_size, patch_size, num_classes, dim, transformer, pool = 'cls', channels = 3):
+    def __init__(self, *, image_size, patch_size, num_classes, dim, transformer, channels = 3):
         super().__init__()
         assert image_size % patch_size == 0, 'image dimensions must be divisible by the patch size'
-        assert pool in {'cls', 'mean'}, 'pool type must be either cls (cls token) or mean (mean pooling)'
         num_patches = (image_size // patch_size) ** 2
         patch_dim = channels * patch_size ** 2
 
-        self.to_patch_embedding = nn.Sequential(
-            Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),
-            nn.Linear(patch_dim, dim),
-        )
+        self.patch_size = patch_size
 
         self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, dim))
+        self.patch_to_embedding = nn.Linear(patch_dim, dim)
         self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
         self.transformer = transformer
 
-        self.pool = pool
-        self.to_latent = nn.Identity()
+        self.to_cls_token = nn.Identity()
 
         self.mlp_head = nn.Sequential(
             nn.LayerNorm(dim),
-            nn.Linear(dim, num_classes)
+            nn.Linear(dim, dim * 4),
+            nn.GELU(),
+            nn.Linear(dim * 4, num_classes)
         )
 
     def forward(self, img):
-        x = self.to_patch_embedding(img)
-        b, n, _ = x.shape
+        p = self.patch_size
 
-        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
+        x = rearrange(img, 'b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = p, p2 = p)
+        x = self.patch_to_embedding(x)
+
+        cls_tokens = self.cls_token.expand(img.shape[0], -1, -1)
         x = torch.cat((cls_tokens, x), dim=1)
-        x += self.pos_embedding[:, :(n + 1)]
+        x += self.pos_embedding
         x = self.transformer(x)
 
-        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]
-
-        x = self.to_latent(x)
+        x = self.to_cls_token(x[:, 0])
         return self.mlp_head(x)

vit_pytorch/t2t.py

@@ -1,74 +0,0 @@
-import math
-import torch
-from torch import nn
-
-from vit_pytorch.vit_pytorch import Transformer
-
-from einops import rearrange, repeat
-from einops.layers.torch import Rearrange
-
-# classes
-
-def conv_output_size(image_size, kernel_size, stride, padding):
-    return int(((image_size - kernel_size + (2 * padding)) / stride) + 1)
-
-class RearrangeImage(nn.Module):
-    def forward(self, x):
-        return rearrange(x, 'b (h w) c -> b c h w', h = int(math.sqrt(x.shape[1])))
-
-# main class
-
-class T2TViT(nn.Module):
-    def __init__(
-        self, *, image_size, num_classes, dim, depth, heads, mlp_dim, pool = 'cls', channels = 3, dim_head = 64, dropout = 0., emb_dropout = 0., t2t_layers = ((7, 4), (3, 2), (3, 2))):
-        super().__init__()
-        assert pool in {'cls', 'mean'}, 'pool type must be either cls (cls token) or mean (mean pooling)'
-
-        layers = []
-        layer_dim = channels
-        output_image_size = image_size
-
-        for i, (kernel_size, stride) in enumerate(t2t_layers):
-            layer_dim *= kernel_size ** 2
-            is_first = i == 0
-            output_image_size = conv_output_size(output_image_size, kernel_size, stride, stride // 2)
-
-            layers.extend([
-                RearrangeImage() if not is_first else nn.Identity(),
-                nn.Unfold(kernel_size = kernel_size, stride = stride, padding = stride // 2),
-                Rearrange('b c n -> b n c'),
-                Transformer(dim = layer_dim, heads = 1, depth = 1, dim_head = layer_dim, mlp_dim = layer_dim, dropout = dropout),
-            ])
-
-        layers.append(nn.Linear(layer_dim, dim))
-        self.to_patch_embedding = nn.Sequential(*layers)
-
-        self.pos_embedding = nn.Parameter(torch.randn(1, output_image_size ** 2 + 1, dim))
-        self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
-        self.dropout = nn.Dropout(emb_dropout)
-
-        self.transformer = Transformer(dim, depth, heads, dim_head, mlp_dim, dropout)
-
-        self.pool = pool
-        self.to_latent = nn.Identity()
-
-        self.mlp_head = nn.Sequential(
-            nn.LayerNorm(dim),
-            nn.Linear(dim, num_classes)
-        )
-
-    def forward(self, img):
-        x = self.to_patch_embedding(img)
-        b, n, _ = x.shape
-
-        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
-        x = torch.cat((cls_tokens, x), dim=1)
-        x += self.pos_embedding[:, :(n + 1)]
-        x = self.dropout(x)
-
-        x = self.transformer(x)
-
-        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]
-
-        x = self.to_latent(x)
-        return self.mlp_head(x)

vit_pytorch/vit_pytorch.py

@@ -1,10 +1,7 @@
-import math
 import torch
-from torch import nn, einsum
 import torch.nn.functional as F
-
-from einops import rearrange, repeat
-from einops.layers.torch import Rearrange
+from einops import rearrange
+from torch import nn
 
 class Residual(nn.Module):
     def __init__(self, fn):
@@ -35,47 +32,45 @@ class FeedForward(nn.Module):
         return self.net(x)
 
 class Attention(nn.Module):
-    def __init__(self, dim, heads = 8, dim_head = 64, dropout = 0.):
+    def __init__(self, dim, heads = 8, dropout = 0.):
         super().__init__()
-        inner_dim = dim_head *  heads
         self.heads = heads
-        self.scale = dim_head ** -0.5
+        self.scale = dim ** -0.5
 
-        self.to_qkv = nn.Linear(dim, inner_dim * 3, bias = False)
+        self.to_qkv = nn.Linear(dim, dim * 3, bias = False)
         self.to_out = nn.Sequential(
-            nn.Linear(inner_dim, dim),
+            nn.Linear(dim, dim),
             nn.Dropout(dropout)
         )
 
     def forward(self, x, mask = None):
         b, n, _, h = *x.shape, self.heads
-        qkv = self.to_qkv(x).chunk(3, dim = -1)
-        q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = h), qkv)
+        qkv = self.to_qkv(x)
+        q, k, v = rearrange(qkv, 'b n (qkv h d) -> qkv b h n d', qkv = 3, h = h)
 
-        dots = einsum('b h i d, b h j d -> b h i j', q, k) * self.scale
-        mask_value = -torch.finfo(dots.dtype).max
+        dots = torch.einsum('bhid,bhjd->bhij', q, k) * self.scale
 
         if mask is not None:
             mask = F.pad(mask.flatten(1), (1, 0), value = True)
             assert mask.shape[-1] == dots.shape[-1], 'mask has incorrect dimensions'
-            mask = rearrange(mask, 'b i -> b () i ()') * rearrange(mask, 'b j -> b () () j')
-            dots.masked_fill_(~mask, mask_value)
+            mask = mask[:, None, :] * mask[:, :, None]
+            dots.masked_fill_(~mask, float('-inf'))
             del mask
 
         attn = dots.softmax(dim=-1)
 
-        out = einsum('b h i j, b h j d -> b h i d', attn, v)
+        out = torch.einsum('bhij,bhjd->bhid', attn, v)
         out = rearrange(out, 'b h n d -> b n (h d)')
         out =  self.to_out(out)
         return out
 
 class Transformer(nn.Module):
-    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout = 0.):
+    def __init__(self, dim, depth, heads, mlp_dim, dropout):
         super().__init__()
         self.layers = nn.ModuleList([])
         for _ in range(depth):
             self.layers.append(nn.ModuleList([
-                Residual(PreNorm(dim, Attention(dim, heads = heads, dim_head = dim_head, dropout = dropout))),
+                Residual(PreNorm(dim, Attention(dim, heads = heads, dropout = dropout))),
                 Residual(PreNorm(dim, FeedForward(dim, mlp_dim, dropout = dropout)))
             ]))
     def forward(self, x, mask = None):
@@ -85,44 +80,44 @@ class Transformer(nn.Module):
         return x
 
 class ViT(nn.Module):
-    def __init__(self, *, image_size, patch_size, num_classes, dim, depth, heads, mlp_dim, pool = 'cls', channels = 3, dim_head = 64, dropout = 0., emb_dropout = 0.):
+    def __init__(self, *, image_size, patch_size, num_classes, dim, depth, heads, mlp_dim, channels = 3, dropout = 0., emb_dropout = 0.):
         super().__init__()
-        assert image_size % patch_size == 0, 'Image dimensions must be divisible by the patch size.'
+        assert image_size % patch_size == 0, 'image dimensions must be divisible by the patch size'
         num_patches = (image_size // patch_size) ** 2
         patch_dim = channels * patch_size ** 2
-        assert pool in {'cls', 'mean'}, 'pool type must be either cls (cls token) or mean (mean pooling)'
 
-        self.to_patch_embedding = nn.Sequential(
-            Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),
-            nn.Linear(patch_dim, dim),
-        )
+        self.patch_size = patch_size
 
         self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, dim))
+        self.patch_to_embedding = nn.Linear(patch_dim, dim)
         self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
         self.dropout = nn.Dropout(emb_dropout)
 
-        self.transformer = Transformer(dim, depth, heads, dim_head, mlp_dim, dropout)
+        self.transformer = Transformer(dim, depth, heads, mlp_dim, dropout)
 
-        self.pool = pool
-        self.to_latent = nn.Identity()
+        self.to_cls_token = nn.Identity()
 
         self.mlp_head = nn.Sequential(
             nn.LayerNorm(dim),
-            nn.Linear(dim, num_classes)
+            nn.Linear(dim, mlp_dim),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(mlp_dim, num_classes),
+            nn.Dropout(dropout)
         )
 
     def forward(self, img, mask = None):
-        x = self.to_patch_embedding(img)
-        b, n, _ = x.shape
+        p = self.patch_size
 
-        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
+        x = rearrange(img, 'b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = p, p2 = p)
+        x = self.patch_to_embedding(x)
+
+        cls_tokens = self.cls_token.expand(img.shape[0], -1, -1)
         x = torch.cat((cls_tokens, x), dim=1)
-        x += self.pos_embedding[:, :(n + 1)]
+        x += self.pos_embedding
         x = self.dropout(x)
 
         x = self.transformer(x, mask)
 
-        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]
-
-        x = self.to_latent(x)
+        x = self.to_cls_token(x[:, 0])
         return self.mlp_head(x)