AI models are specialists — here's the field guide to 8 types
Table of Contents
Today I learned that “AI model” isn’t one thing — it’s a family of specialists. Like a company has a writer, an accountant and a salesperson, each model type is built for a different job. Knowing which one you’re talking about changes everything.
The 8 types at a glance
| Type | The job | Analogy |
|---|---|---|
| Chat / LLM | Text in → text out | General practitioner |
| Embedding | Text in → numbers out | Librarian cataloguing meaning |
| Vision (VLM) | Image in → text out | A photographer’s eye |
| Reasoning | Question in → step-by-step working + answer | Exam sitter who shows working |
| Agentic / tool-use | Task in → tool calls + actions | Assistant with a phone |
| TTS | Text in → audio out | Voice actor |
| STT | Audio in → text out | Court stenographer |
| Image gen (diffusion) | Text in → image out | Painter — a whole different architecture |
Four more ways to slice the family
The 8 above are jobs — what a model does. But models also differ along other axes:
- By architecture: dense models fire every parameter on every input; MoE (mixture of experts) routes each token to just a few specialist sub-networks — big-model knowledge at small-model cost (DeepSeek V4 Pro is 1.6T total, only 49B active per token).
- By specialty: code models are just LLMs fine-tuned to be fluent in code — same family, extra homework (CodeLlama, Qwen-Coder).
- By tuning: a raw base model can’t hold a conversation — it just completes text. “Refined for talking” is a real thing: fine-tune a base model on conversational data (+ RLHF) and it becomes a chat model.
- By alignment: most models ship with guardrails baked into the weights; “uncensored” variants have them weakened or removed — via fine-tuning, weight surgery, or merging. Same model, different alignment.
The bigger win: “type” isn’t one axis. Job, architecture, specialty, tuning stage, and alignment are five separate ways to slice the same model — mixing them up is how people end up confused about what a model actually is.
The one that bit me: embeddings
An embedding model turns text into a list of numbers (a vector) that captures meaning — “the cat sat on the mat” and “a feline rested on the rug” land close together, despite sharing almost no words. That’s the magic behind semantic search and RAG.
But here’s the rule I learned the hard way: an embedding model is a lock, not a key. You must use the same model — and the same number of dimensions — when building a search index and when querying it. Swap models and the stored vectors live in a different “meaning space” — garbage results. And providers retire models without warning; when mine got retired mid-project, the 404s started everywhere and the index was stranded until a replacement with matching dimensions arrived.
The transferable win: pick your embedding model once, record its name + dimensions, and treat it as a fixed contract. It saves you a full index rebuild.
Bonus confusion: reasoning vs agentic
Reasoning models think before answering — great at math, but they burn output budget on hidden chain-of-thought (one chat endpoint of mine silently ate its entire token budget and returned empty answers). Agentic models instead emit tool calls — they’re the ones driving “agents” that actually do things.
Model types combine freely (GPT-5 = chat + vision + reasoning). But knowing the job you need is the first step of picking the right one — the hype words don’t tell you which specialist you’re actually hiring.