Two ways to use it

Google built the transformer to translate, one half reading and one half writing.

People quickly noticed each half was useful on its own, and the two halves grew into the two great families of language model. They split on one question: can a word see what comes after it, or only what comes before?

The reader that sees both sides. The first family stayed at Google, and was best known through a model called , published in 2018.

You have already played its training game. Earlier you read The waiter brought us a bottle of pripple, and guessed pripple was a drink purely from the tokens around it, on both sides. That is how BERT learns: hide a token, show it the whole sentence with a blank, and make it guess the missing token.

Because it sees context on both sides at once, it builds a deep understanding of a sentence as a whole. Great for reading: search, answering questions, sorting text. Not built for writing new text, because it always assumed it could peek at both sides, and when you are generating, there is no "after" yet.