Multimodality is the faculty to work with data that comes in different content formats, such as text, audio, images, video, etc.

Different components of an ecosystem, system, or model can process multimodal data, which enables them to handle a mix of content formats seamlessly. For example:

• Chat Models — They could accept and generate multimodal inputs and outputs, handling various content formats like text, images, audio, and video.
• Embedding Models — These models can represent multimodal content. They embed various data assets in their own content format (text, images, audio) in structures called embeddings or vectors, which are kept in vector spaces.
• Vector Stores — They can search through embeddings that represent multimodal data, making it possible to retrieve information of various kinds.

For the rest of this post, we will only discuss text-based data, and we will leave multimodality aside.