lucidrains

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

Implementation of Pixel-level Contrastive Learning, proposed in the paper "Propagate Yourself", in Pytorch

Implementation of rectified flow and some of its followup research / improvements in Pytorch

Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch

Explorations into some recent techniques surrounding speculative decoding

Implementation of Key-Locked Rank One Editing, from Nvidia AI

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

Implementation of Magic3D, Text to 3D content synthesis, in Pytorch

Implementation of MedSegDiff in Pytorch - SOTA medical segmentation using DDPM and filtering of features in fourier space

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

Implementation of Block Recurrent Transformer - Pytorch

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2

Implementation of MetNet-3, SOTA neural weather model out of Google Deepmind, in Pytorch

Implementation of fused cosine similarity attention in the same style as Flash Attention

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

A Pytorch implementation of Stylegan2 with UNet Discriminator

Implementation of Flash Attention in Jax

Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena

Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch

Implementation of the neural network proposed in Natural Speech, a text-to-speech generator that is indistinguishable from human recordings for the first time, from Microsoft Research

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Implementation of ResMLP, an all MLP solution to image classification, in Pytorch

An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates concepts from neural fields, top-down-bottom-up processing, and attention (consensus between columns), for emergent part-whole heirarchies from data

Implementation of Fast Transformer in Pytorch

This repository will house a visualization that will attempt to convey instant enlightenment of how Attention works to someone not working in artificial intelligence, with 3Blue1Brown as inspiration

Implementation of Dreamer v3, Deepmind's first neural network that was able to learn to collect diamonds in Minecraft, in Pytorch

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

Pytorch implementation of Compressive Transformers, from Deepmind