Encoder Decoder Terminology

encoder-decoder architecture

A machine that consumes sequences and outputs sequences. Comprised of two stages: Encoding stage and Decoding stage. Each stage can be implemented with a different architecture.

decoder stage

Creates a human readable output (i.e. sentence) from the input vector representation.

BERT

First used in 2018

T5

First used in 2019

GPT-3

First used in 2020

PaLM

First used in 2022

encoder stage

Produces a vector representation of the input sequence.

sequence-to-sequence architecture

Takes a sequence of words as input and outputs a sequence of words.

LLM

abbreviation for Large pre-trained Language Model. First used in 2017.

RNN

abbreviation for Recurrent Neural Network