6 Commits

Author	SHA1	Message	Date
Phil Wang	8208c859a5	just remove PreNorm wrapper from all ViTs, as it is unlikely to change at this point	2023-08-14 09:48:55 -07:00
Phil Wang	5699ed7d13	double down on dual patch norm, fix MAE and Simmim to be compatible with dual patchnorm	2023-02-10 10:39:50 -08:00
Phil Wang	4e6a42a0ca	correct need for post-attention dropout	2022-03-30 10:50:57 -07:00
Phil Wang	3a3038c702	add layer dropout for CaiT	2021-04-01 20:30:37 -07:00
Phil Wang	05b47cc070	make sure layerscale epsilon is a function of depth	2021-03-31 22:53:04 -07:00
Phil Wang	9ef8da4759	add CaiT, new vision transformer out of facebook AI, complete with layerscale, talking heads, and cls -> patch cross attention	2021-03-31 22:42:16 -07:00