What "AI PM" actually means: evals, agents, and observability

In 2026, calling yourself a "technical PM" increasingly means AI fluency — three skills in particular: reading and defining evals (the tests that score an AI's output), understanding agents (an AI that takes actions, not just chats), and reading observability (the record of what the model actually did). You don't need to train a model; you need to know what each of these is and what each one costs.

"AI PM" has become a title people put on their profile without a shared definition. The hiring manager does have one, and it isn't "has used ChatGPT." It's closer to this: can you reason about an AI feature the way a good PM reasons about any system — by naming what each part does and what it costs.

Here are the three words that show up most, in plain terms, with the tradeoff each one carries.

What is an eval, and why do PMs keep getting asked about them?

An eval is a test that scores an AI's output against examples you've decided are good. You give the model a set of inputs, you have an answer or a rubric for each, and you get back a number: how often it got it right.

That number is the point. Without it, "we improved the prompt" is a feeling. With it, you can say the model went from 71% to 84% on the cases you care about — or that your change made it worse, which happens just as often.

The tradeoff: an eval is only as good as the examples in it. A model can score 95% and still fail real users, because your eval didn't include the cases they actually hit. Building the eval set is a product judgment — which cases matter, which failures are unacceptable, which are fine. So when someone asks whether you can "define an eval," they're asking whether you can turn a fuzzy quality goal into a concrete, scorable set of cases. That's a PM skill wearing a technical name.

What is an agent, really — and how is it different from a chatbot?

An agent is an AI that takes actions by calling tools in a loop, instead of just answering in text. A chatbot replies. An agent does: it reads your request, looks something up, sends an email, checks whether it worked, and keeps going until the job is done or it gets stuck.

That loop is the difference, and it's where the product decisions live. Every action an agent can take is a door you opened, and every door is something that can go wrong on its own.

The tradeoff: an agent that can do more can do more damage. A chatbot's wrong answer wastes a minute; an agent with the wrong permissions can send the wrong message, delete the wrong record, or spend real money. So the product questions become which actions you let it take, what it confirms with a human first, and what happens when a step fails halfway through — the same questions you'd ask about any system that acts on a user's behalf.

What is observability, and what does a PM do with it?

Observability is the record of what the model actually did, step by step — often called traces or logs. For a plain model call, it's the input, the output, and how long it took. For an agent, it's the whole loop: it decided this, called that tool, got this back, decided the next thing.

Why a PM cares: "it gave a bad answer" is not a bug report anyone can act on. The trace is. It shows the model searched for the wrong thing, called the tool with a garbled input, or looped four times when it should have stopped.

The tradeoff: traces are detailed, and detail is noise until you know which part to read. A single agent run can be hundreds of steps. The skill isn't reading all of it — it's knowing where the decision happened that produced the bad outcome, the same way you'd follow one request through an architecture diagram instead of reading every box.

Where does PMLabs fit — and where doesn't it?

Straight answer first: PMLabs does not teach evals, agents, or observability directly. Its curriculum is classic system design — you get a problem and a blank canvas, drag the boxes on, wire them together, and write one line on why each one is there. A rubric scores it in about 30 seconds, so you find out what you missed while it's cheap to miss it. Sessions run about 15 minutes. No code — you're designing systems, not shipping production code.

What it trains is the reflex those three words sit on top of: thinking in parts and costs, and naming the tradeoff out loud. The AI-specific layer is the adjacent frontier you add once the habit is there.

One honest note, the same one we give everywhere: the scoring is AI against a rubric. Treat it as a sparring partner, not an architecture review or professional engineering advice. It won't replace your engineers and it isn't infallible — it gives you the reps so the reasoning becomes automatic.

If you want to build that habit, Module 1 is free, no card: pmlabs.app. If you'd rather keep the one-page version on your desk, the cheat sheet covers the core components. For the kind of questions this habit turns into, see the system design questions PMs actually get asked, and if diagrams are where you freeze, start with how to read an architecture diagram without knowing how to code.