One model, several senses

There is a gap between the machine described in this journey and the thing you have actually used, and it is time to close it.

Everything built here eats tokens. Text in, text out. But you have almost certainly photographed something and asked a model what it was, or handed it a screenshot, a chart, a page of handwriting.

The word for this is multimodal, meaning a system that handles more than one kind of input.

The connection begins with the embeddings chapter. Text enters a transformer as vectors. Other input types can also be converted into vectors the network can process.

One common design cuts an image into patches and encodes each patch as a vector. Another uses a separate vision network and connects its output to a language model. Attention can then link visual information with text. The details vary across systems.

Audio and video also need input-specific encoders or tokenizers. They are not simply text with a different label.

The shared idea is that different signals can be transformed into numerical representations and connected. A multimodal system may reuse a transformer core, add specialist encoders and decoders, or train several components together.