lucidrains

An implementation of Performer, a linear attention-based transformer, in Pytorch

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

Transformer based on a variant of attention that is linear complexity in respect to sequence length

A concise but complete implementation of CLIP with various experimental improvements from recent papers

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

Implementation of Bottleneck Transformer in Pytorch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

Implementation of Voicebox, new SOTA Text-to-speech network from MetaAI, in Pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

Implementation of the Point Transformer layer, in Pytorch

Implementation of MagViT2 Tokenizer in Pytorch

A simple way to keep track of an Exponential Moving Average (EMA) version of your Pytorch model

Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch

Implementation of Parti, Google's pure attention-based text-to-image neural network, in Pytorch

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

Pytorch implementation of SIREN - Implicit Neural Representations with Periodic Activation Function

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch

Implementation of E2-TTS, "Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS", in Pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch