JacobLinCool
|
ab63fc9cc8
|
remove duplicated qkv computation in na_vit_nested_tensor_3d.py (#341)
|
2025-01-19 05:52:46 -08:00 |
|
Kale Kundert
|
b7ed6bad28
|
add option to set frame padding for 3D CCT (#339)
|
2025-01-04 07:55:27 -08:00 |
|
lucidrains
|
e7cba9ba6d
|
add a simple vit flavor for a new bytedance paper that proposes to break out of the traditional one residual stream architecture - "hyper-connections"
|
2024-12-20 17:43:50 -08:00 |
|
lucidrains
|
56373c0cbd
|
make value residual learned
|
2024-11-24 08:21:28 -08:00 |
|
lucidrains
|
24196a3e8a
|
allow for qk norm to be turned off for na vit nested tensor
|
2024-11-20 10:59:22 -08:00 |
|
lucidrains
|
d47c57e32f
|
fix tests
|
2024-11-10 09:43:54 -08:00 |
|
lucidrains
|
0449865786
|
update minimum version for nested tensor of NaViT
|
2024-11-10 09:37:48 -08:00 |
|
lucidrains
|
6693d47d0b
|
update comment for navit 3d
|
2024-11-07 20:02:07 -08:00 |
|
Phil Wang
|
141239ca86
|
fix value residual
|
2024-10-31 06:48:24 -07:00 |
|
lucidrains
|
0b5c9b4559
|
add value residual based simple vit
|
2024-10-28 09:19:00 -07:00 |
|
lucidrains
|
e300cdd7dc
|
fix multiheaded qk rmsnorm in nViT
|
2024-10-10 19:15:17 -07:00 |
|
Phil Wang
|
36ddc7a6ba
|
go all the way with the normalized vit, fix some scales
|
2024-10-10 10:42:37 -07:00 |
|
Phil Wang
|
74b62009f8
|
go for multi-headed rmsnorm for the qknorm on hypersphere vit
|
2024-10-10 08:09:58 -07:00 |
|
Phil Wang
|
f50d7d1436
|
add a hypersphere vit, adapted from https://arxiv.org/abs/2410.01131
|
2024-10-09 07:32:25 -07:00 |
|
lucidrains
|
82f2fa751d
|
address https://github.com/lucidrains/vit-pytorch/issues/330
|
2024-10-04 07:01:48 -07:00 |
|
lucidrains
|
fcb9501cdd
|
add register tokens to the nested tensor 3d na vit example for researcher
|
2024-08-28 12:21:31 -07:00 |
|
roydenwa
|
9d43e4d0bb
|
Add ViViT variant with factorized self-attention (#327)
* Add FactorizedTransformer
* Add variant param and check in fwd method
* Check if variant is implemented
* Describe new ViViT variant
|
2024-08-21 19:23:38 -07:00 |
|
Phil Wang
|
5e808f48d1
|
3d version of navit nested tensor
|
2024-08-21 07:23:21 -07:00 |
|
lucidrains
|
73199ab486
|
Nested navit (#325)
add a variant of NaViT using nested tensors
|
2024-08-20 15:12:29 -07:00 |
|
Phil Wang
|
dfc8df6713
|
add the u-vit implementation with simple vit + register tokens
|
2024-08-07 08:45:57 -07:00 |
|
lucidrains
|
9992a615d1
|
attention re-use in lookup vit should use pre-softmax attention matrix
|
2024-07-19 19:23:38 -07:00 |
|
Phil Wang
|
4b2c00cb63
|
when cross attending in look vit, make sure context tokens are normalized
|
2024-07-19 10:23:12 -07:00 |
|
Phil Wang
|
ec6c48b8ff
|
norm not needed when reusing attention in lookvit
|
2024-07-19 10:00:03 -07:00 |
|
Phil Wang
|
bd72b58355
|
add lookup vit, cite, document later
|
2024-07-19 09:48:58 -07:00 |
|
lucidrains
|
e3256d77cd
|
fix t2t vit having two layernorms, and make final layernorm in distillation wrapper configurable, default to False for vit
|
2024-06-11 15:12:53 -07:00 |
|
lucidrains
|
90be7233a3
|
rotary needs to be done with full precision to be safe
|
2024-05-11 08:04:32 -07:00 |
|
Phil Wang
|
bca88e9039
|
address https://github.com/lucidrains/vit-pytorch/issues/300
|
2024-05-02 08:46:39 -07:00 |
|
Phil Wang
|
96f66d2754
|
address https://github.com/lucidrains/vit-pytorch/issues/306
|
2024-04-18 09:44:29 -07:00 |
|
Phil Wang
|
12249dcc5f
|
address https://github.com/lucidrains/vit-pytorch/issues/304
|
2024-04-17 09:40:03 -07:00 |
|
lucidrains
|
5578ac472f
|
address https://github.com/lucidrains/vit-pytorch/issues/292
|
2023-12-23 08:11:39 -08:00 |
|
lucidrains
|
d446a41243
|
share an idea that should be tried if it has not been
|
2023-11-14 16:55:36 -08:00 |
|
lucidrains
|
0ad09c4cbc
|
allow channels to be customizable for cvt
|
2023-10-25 14:47:58 -07:00 |
|
Artem Lukin
|
fb4ac25174
|
Fix typo in LayerNorm (#285)
Co-authored-by: Artem Lukin <artyom.lukin98@gmail.com>
|
2023-10-24 12:47:21 -07:00 |
|
lucidrains
|
53fe345e85
|
no longer needed with einops 0.7
|
2023-10-19 18:16:46 -07:00 |
|
Phil Wang
|
1616288e30
|
add xcit (#284)
* add xcit
* use Rearrange layers
* give cross correlation transformer a final norm at end
* document
|
2023-10-13 09:15:13 -07:00 |
|
lucidrains
|
bbb24e34d4
|
give a learned bias to and from registers for maxvit + register token variant
|
2023-10-06 10:40:26 -07:00 |
|
lucidrains
|
df8733d86e
|
improvise a max vit with register tokens
|
2023-10-06 10:27:36 -07:00 |
|
lucidrains
|
680d446e46
|
document in readme later
|
2023-10-03 09:26:02 -07:00 |
|
lucidrains
|
a36546df23
|
add simple vit with register tokens example, cite
|
2023-10-01 08:11:40 -07:00 |
|
lucidrains
|
d830b05f06
|
address https://github.com/lucidrains/vit-pytorch/issues/279
|
2023-09-10 09:32:57 -07:00 |
|
Phil Wang
|
8208c859a5
|
just remove PreNorm wrapper from all ViTs, as it is unlikely to change at this point
|
2023-08-14 09:48:55 -07:00 |
|
Phil Wang
|
b194359301
|
add a simple vit with qknorm, since authors seem to be promoting the technique on twitter
|
2023-08-14 07:58:45 -07:00 |
|
lucidrains
|
950c901b80
|
fix linear head in simple vit, thanks to @atkos
|
2023-08-10 14:36:21 -07:00 |
|
Phil Wang
|
3e5d1be6f0
|
address https://github.com/lucidrains/vit-pytorch/pull/274
|
2023-08-09 07:53:38 -07:00 |
|
Phil Wang
|
6e2393de95
|
wrap up NaViT
|
2023-07-25 10:38:55 -07:00 |
|
Phil Wang
|
32974c33df
|
one can pass a callback to token_dropout_prob for NaViT that takes in height and width and calculate appropriate dropout rate
|
2023-07-24 14:52:40 -07:00 |
|
Phil Wang
|
17675e0de4
|
add constant token dropout for NaViT
|
2023-07-24 14:14:36 -07:00 |
|
Phil Wang
|
23820bc54a
|
begin work on NaViT (#273)
finish core idea of NaViT
|
2023-07-24 13:54:02 -07:00 |
|
roydenwa
|
d4daf7bd0f
|
Support SimpleViT as encoder in MAE (#272)
support simplevit in mae
|
2023-07-24 06:43:01 -07:00 |
|
Phil Wang
|
9e3fec2398
|
fix mpp
|
2023-06-28 08:02:43 -07:00 |
|