What is multimodality for LLMs, actually?

So, to keep it simple, multimodality is when a LLM (or any other AI model) manages to understand something other than pure text. It can analyze and process different modalities of information, like text and images or even sound. In short, it becomes a real Swiss army knife.

An example: when you dictate a message to your phone, the model just transcribes what you say. But, if someone else talks at the same time as you, it will transcribe what that other person says too…

With a multimodal model, that changes everything: it can theoretically understand that someone else is talking, and ignore what it “hears”. We go from a blind transcription to a finer understanding of the sound context.

Another example: before, to analyze an image or a chart with text, you could extract the text easily, but understanding the structure of a chart? Good luck. Now, with the “vision” capability (like the ones you find in some recent models), a model can literally understand images. For instance, you could detect every image containing violence or break-in attempts and trigger an alert when that happens.

And beyond text and images: these new possibilities go much further. Multimodal LLMs can now extract structured data from unstructured sources: PDFs, PowerPoint presentations, even video. That’s huge potential for anything touching knowledge bases with docs full of unstructured data (hello PPTX files packed with charts). 💼

The latest one: Mistral and its Pixtral model on September 11th. The French startup Mistral shipped Pixtral as open source multimodal (text + vision). It’s not the first of its kind, but at the scale of open source models, it’s clearly near the top of the ranking today. And it’s nice to see high-performing models available to everyone, rather than letting big closed proprietary models blow up the price per token.

So there you go, multimodality isn’t just a buzzword. It’s literally what extends the capabilities of LLMs to many more use cases.