70 Commits

Author	SHA1	Message	Date
Phil Wang	04f86dee3c	implement SOTA new self-supervised learning technique from facebook for vision transformers, Dino	2021-05-02 14:00:36 -07:00
Phil Wang	6549522629	be able to accept non-square patches, thanks to @FilipAndersson245	2021-05-01 20:04:41 -07:00
Phil Wang	65bb350e85	0.17.2	2021-04-30 06:44:54 -07:00
Phil Wang	7807f24509	fix small bug	2021-04-29 15:39:41 -07:00
Phil Wang	30a1335d31	release twins svt	2021-04-29 14:55:25 -07:00
Phil Wang	4f3dbd003f	for PiT, project to increased dimensions on first grouped conv for depthwise-conv	2021-04-29 12:41:00 -07:00
Phil Wang	0df1505662	add zeroing of weight parameters of batchnorm in levit just before residual connection, noticed by @EelcoHoogendoorn	2021-04-27 08:41:16 -07:00
Phil Wang	3df6c31c61	fix norm issues in cvt	2021-04-27 08:36:17 -07:00
Phil Wang	54af220930	fix cvt	2021-04-26 20:37:51 -07:00
Phil Wang	bad4b94e7b	fix all issues with rotary vision transformer	2021-04-25 12:09:32 -07:00
Phil Wang	e42e9876bc	offer a way to turn off ds conv in rotary vision transformer for ablation	2021-04-20 10:12:03 -07:00
Phil Wang	566365978d	add ability to turn off rotary, for ablation	2021-04-20 09:00:27 -07:00
Phil Wang	34f78294d3	fix pooling bugs across a few new archs	2021-04-19 22:36:23 -07:00
Phil Wang	4c29328363	fix frequency in rotary vision transformer	2021-04-15 16:06:32 -07:00
Phil Wang	27ac10c1f1	0.16.3	2021-04-14 16:53:05 -07:00
Phil Wang	1d8b7826bf	update personal pet vit	2021-04-14 15:56:39 -07:00
Phil Wang	6289619e3f	0.16.1	2021-04-14 08:05:08 -07:00
Phil Wang	30b37c4028	add LocalViT	2021-04-12 19:17:32 -07:00
Phil Wang	4497f1e90f	add rotary vision transformer	2021-04-10 22:59:15 -07:00
Phil Wang	b50d3e1334	cleanup levit	2021-04-06 13:46:19 -07:00
Phil Wang	e075460937	stray print	2021-04-06 13:38:52 -07:00
Phil Wang	2cb6b35030	complete levit	2021-04-06 13:36:11 -07:00
Phil Wang	3a3038c702	add layer dropout for CaiT	2021-04-01 20:30:37 -07:00
Phil Wang	b1f1044c8e	offer hard distillation as well	2021-04-01 16:56:14 -07:00
Phil Wang	05b47cc070	make sure layerscale epsilon is a function of depth	2021-03-31 22:53:04 -07:00
Phil Wang	9ef8da4759	add CaiT, new vision transformer out of facebook AI, complete with layerscale, talking heads, and cls -> patch cross attention	2021-03-31 22:42:16 -07:00
Phil Wang	da950e6d2c	add working PiT	2021-03-30 22:15:19 -07:00
Phil Wang	4b9a02d89c	use depthwise conv for CvT projections	2021-03-30 18:18:35 -07:00
Phil Wang	518924eac5	add CvT	2021-03-30 14:42:39 -07:00
Phil Wang	e712003dfb	add CrossViT	2021-03-30 00:53:27 -07:00
Phil Wang	d04ce06a30	make recorder work for t2t and deepvit	2021-03-29 18:16:34 -07:00
Phil Wang	8135d70e4e	use hooks to retrieve attention maps for user without modifying ViT	2021-03-29 15:10:12 -07:00
Phil Wang	3067155cea	add recorder class, for recording attention across layers, for researchers	2021-03-29 11:08:19 -07:00
Phil Wang	15294c304e	remove masking, as it complicates with little benefit	2021-03-23 12:18:47 -07:00
Phil Wang	b900850144	add deep vit	2021-03-23 11:57:13 -07:00
Phil Wang	173e07e02e	cleanup and release 0.8.0	2021-03-08 07:28:31 -08:00
Phil Wang	1698b7bef8	make it so one can plug performer into t2tvit	2021-02-25 20:55:34 -08:00
Phil Wang	6760d554aa	no need to do projection to combine attention heads for T2Ts initial one-headed attention layers	2021-02-24 12:23:39 -08:00
Phil Wang	a82894846d	add DistillableT2TViT	2021-02-21 19:54:45 -08:00
Phil Wang	3744ac691a	remove patch size from T2TViT	2021-02-21 19:15:19 -08:00
Phil Wang	6af7bbcd11	make sure distillation still works	2021-02-21 19:08:18 -08:00
Phil Wang	e3205c0a4f	add token to token ViT	2021-02-19 22:28:53 -08:00
Phil Wang	3f2cbc6e23	fix for ambiguity in broadcasting mask	2021-02-17 07:38:11 -08:00
Phil Wang	85314cf0b6	patch for scaling factor, thanks to @urkax	2021-01-21 09:39:42 -08:00
Phil Wang	e8ca6038c9	allow for DistillableVit to still run predictions	2021-01-11 10:49:14 -08:00
Phil Wang	2263b7396f	allow distillable efficient vit to restore efficient vit as well	2020-12-25 19:31:25 -08:00
Phil Wang	74074e2b6c	offer easy way to turn DistillableViT to ViT at the end of training	2020-12-25 11:16:52 -08:00
Phil Wang	0c68688d61	bump for release	2020-12-25 09:30:48 -08:00
Phil Wang	db98ed7a8e	allow for overriding alpha as well on forward in distillation wrapper	2020-12-24 11:18:36 -08:00
Phil Wang	dc4b3327ce	no grad for teacher in distillation	2020-12-24 11:11:58 -08:00