Let you know the principle of AI models from excellent works
This post is an introduction to the Transformer, in terms of its origin, application and development.
RNN 2015
⭐ Sequence to sequence model with attention mechanism, to avoid forgeting too long vectors, compute each vector’s weight (correlation) of decoder with all the vectors in encoder. Machine Translation, Natural Language Processing (NLP)
📄 Neural Machine Translation by Jointly Learning to Align and Translate, 2015
RNN 2016
⭐ Self-Attention model, not limited in sequence to sequence model.
📄 Long Short-Term Memory-Networks for Machine Reading, 2016
Transformer 2017
⭐ Attention without RNN, in other words, it consists of Attention and Self-Attention layers only.
📄 Attention is all you need, 2017
Vision Transformer 2021
⭐ It uses Transformer (encoder part) to do the image classification task. It split the image into small patches as input, and record the position infomation of each patch. It performs well if pretrained on large datasets (more than 100 million). Short for ViT.
📄 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021
Vision Transformer in segmentation 2021
⭐ It first uses Transformer to do the image segmentation task. At the same time, it combines the CNN in encoder part to better restore the low level information, in decoder part, it also uses CNN.
📄 TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation, 2021-02-08
📄 Medical Transformer: Gated Axial-Attention for Medical Image Segmentation, 2021-02-21
📄 CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation, 2021-03-04
📄 TransBTS: Multimodal Brain Tumor Segmentation Using Transformer, 2021-03-07
📄 UNETR: Transformers for 3D Medical Image Segmentation, 2021-03-18
📄 Pyramid Medical Transformer for Medical Image Segmentation, 2021-04-29
📄 Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation, 2021-05-12
📄 UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation, 2021-07-02
📄 DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation, 2021-07-12
📄 TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation, 2021-07-12
📄 UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer, 2021-09-09
📄 nnFormer: Interleaved Transformer for Volumetric Segmentation, 2021-09-07
📄 MISSFormer: An Effective Medical Image Segmentation Transformer, 2021-09-15
📄 Transformer-Unet: Raw Image Processing with Unet, 2021-09-17
📄 VT-UNet: A Robust Volumetric Transformer for Accurate 3D Tumor Segmentation, 2021-11-26
📄 Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review, 2023-01-10, 63 pages!
👉 Acknowledgements: Wang Shusen
