15 Commits

Author	SHA1	Message	Date
Phil Wang	f50d7d1436	add a hypersphere vit, adapted from https://arxiv.org/abs/2410.01131	2024-10-09 07:32:25 -07:00
lucidrains	90be7233a3	rotary needs to be done with full precision to be safe	2024-05-11 08:04:32 -07:00
Phil Wang	8208c859a5	just remove PreNorm wrapper from all ViTs, as it is unlikely to change at this point	2023-08-14 09:48:55 -07:00
Phil Wang	4e6a42a0ca	correct need for post-attention dropout	2022-03-30 10:50:57 -07:00
Phil Wang	f196d1ec5b	move freqs in RvT to linspace	2021-10-05 09:23:44 -07:00
Jonathan Tow	6f3a5fcf0b	Add use_glu flag to RvT	2021-04-30 02:07:41 -04:00
Phil Wang	60b5687a79	cleanup rvt	2021-04-27 11:45:46 -07:00
Phil Wang	bad4b94e7b	fix all issues with rotary vision transformer	2021-04-25 12:09:32 -07:00
Phil Wang	e42e9876bc	offer a way to turn off ds conv in rotary vision transformer for ablation	2021-04-20 10:12:03 -07:00
Phil Wang	566365978d	add ability to turn off rotary, for ablation	2021-04-20 09:00:27 -07:00
Phil Wang	34f78294d3	fix pooling bugs across a few new archs	2021-04-19 22:36:23 -07:00
Phil Wang	4c29328363	fix frequency in rotary vision transformer	2021-04-15 16:06:32 -07:00
Phil Wang	fa216c45ea	tweak	2021-04-14 16:52:53 -07:00
Phil Wang	53b3af05f6	use convolution on query with padding to give the network absolute spatial awareness in addition to relative encoding from rotary embeddings	2021-04-14 15:56:02 -07:00
Phil Wang	4497f1e90f	add rotary vision transformer	2021-04-10 22:59:15 -07:00