Rohit Singh

Founder, Populosoft

Satya Nadella says verification is now the hard part. Here’s what that means if you’re not Microsoft

In a podcast episode published on 16 November 2026, Microsoft CEO Satya Nadella told Gergely Orosz of The Pragmatic Engineer that AI has made writing code cheap and checking it hard. Microsoft can build a software factory around that problem. Most teams shipping AI-built web apps can’t, so here are his verification ideas sized for a small team, with a checklist you can run this week.

I have been saying some version of this since I started Populosoft, so I paid attention when the CEO of Microsoft said it plainly. Here is what he said, what it means, and what to do about it if your team is three people and a deploy button.

Quote card: “In a world where code generation has become cheap, what has become hard is verification”, Satya Nadella, CEO of Microsoft, on The Pragmatic Engineer, 16 November 2026. Two terminal windows: generate is done, verify is still checking.
The full episode on YouTube, starting at the chapter “Verifying AI-generated code”. Playing it loads the video from YouTube (Google), which may set cookies and store data in your browser; see our cookie policy. Or watch it on YouTube from 12:32.

What did Satya Nadella say about verification?

In a world where code generation has become cheap, what has become hard is verification, right?

He said it while answering a question from Orosz about the building blocks teams are still missing for working reliably with large language models. Nadella was describing how products like the GitHub Copilot runtime are built today, and he went straight to the hard part: once code is cheap to produce, how do you know it works?

This was not an off-the-cuff remark. In October, in an essay on his blog, he wrote that “Validation must be independent of the intelligence being validated” and that “No one model should become the sole dependency for an important outcome or be responsible for verifying its own work.”

He did not claim to have solved it. In his notes on the episode, Orosz wrote: “We didn’t arrive at any definite solutions for this during the conversation”. Nadella named the problem. The rest of this post is about what you can do with it.

Is this new?

Not really, and it is worth saying so. Verification was always the hard part of software. What changed is the volume. At AWS re:Invent in December 2025, Amazon CTO Werner Vogels explained that when a machine writes the code, you have to rebuild your understanding of it during review. In his words: “That’s what’s called verification debt.”

The surveys point the same way. In Sonar’s January 2026 survey of more than 1,100 developers, 96% said they do not fully trust that AI-generated code is functionally correct, yet only 48% always check it before committing. Sonar sells code-quality tools, so read that as an indicator, but it called the result a “verification gap”, the same idea our own thesis page is built on. Nadella is one of the most senior people to say it this plainly.

Microsoft’s verification playbook, sized for a small team

Across the interview and the essay, Nadella describes four habits. Microsoft can build infrastructure for each of them. A founder with a Lovable, Bolt, Replit or Cursor app usually has none of that, only a deploy button. Each habit still has a version a three-person team can run this week.

Turn every failure into a test, and verify the test

Every bug you fix gets a test that reproduces it. Break the feature once, on purpose, and watch the test fail.

Let a separate critic check the work

Whoever checks the code is not whatever wrote it: another person, another tool, and a human who signs off.

Feed production traces back into verification

Turn on error monitoring, read it every week, and test your critical flows every day, not just at launch.

Treat AI agents as an insider risk

Your coding agent can’t merge to main or edit the tests that judge it, and its changes are reviewed and logged.

Nadella’s four verification habits, and a small-team version of each.

1. Turn every failure into a test, and check that the test can fail

Half a minute after the line above, Nadella asked: “how do you take every failure and then build back the test case and make sure you truly verify the test, right?”

The second half is the part that matters. AI coding agents are very good at producing tests that pass. In his write-up of the episode, Orosz gives the classic failure: a model “making tests pass by deleting failing tests”. Responding to the same interview, the open-source project PulseEngine put the rule in six words: “A test that cannot fail tests nothing.”

For a small team: every bug you fix gets a test that reproduces it. Then break the feature on purpose, once, and confirm the test fails. If it doesn’t, the test is decoration.

Where we fit: our Launch Audit is built on the same idea: tests for your critical flows, written in a standard format rather than a proprietary one.

2. Don’t let the code grade its own homework

Later in the conversation (about 17:50), Nadella came back to it: “The way you want to do agentic coding is these little tasks that you give it. Make sure that it’s truly done. How do you have the critique? There’s so much to be said about having a separate model family.”

Microsoft can run one AI model to check another. A small team can apply the same principle with people and process: the reviewer is not the author, and a human signs off before every release.

About usOur testing agent, The Vibe Tester, runs on Anthropic’s Claude, which may well be the model family that wrote your code. So we don’t claim to be a different model family. Our independence is that we are a separate team: we test your running app from the outside, through its URL and APIs, we didn’t write it, and on our paid plans a QA engineer reviews the findings before they reach you.

3. Close the loop from production

“The other one that I’m really excited about is the traces from production coming back,” Nadella said (about 18:56), taking production logs to verify a fix, and calling it “that verifiability loop”. His essay makes the same point in writing: “We need to continuously test the entire system, including failures, attacks, edge cases, system changes, etc. not just successful tasks.”

For a small team: turn on error monitoring for your app, read it every week, turn each new production error into a test, and run your critical flows every day.

Where we fit: our Monitor plan is designed for the daily part: your critical flows tested every day, with verified alerts. It is listed at $300 a month (indicative); the pricing page has the current details.

4. Treat your coding agent as an insider

Near the end of the episode (from 51:21), Nadella warned that “we’re getting more and more capable and less and less reliable, which is the most dangerous place to be in,” and concluded: “you’ve got to treat these agents as insider risk.”

For a small team: give your coding agent the access a new contractor gets on day one. It can propose changes, but it can’t merge to main or edit the tests that judge its work, and everything it does is logged. Keep your own written definition of “done”, outside the agent.

Where we fit: if you want a person doing this alongside your team, that is what Managed QA is for: a QA engineer working with your developers, from $1,500 a month (indicative).

A one-week verification checklist for AI-built web apps

  1. Run a free outside scan of your live app, to see what an agent that didn’t write your code finds.
  2. Write down your three to five critical flows: sign-up, login, checkout, onboarding and the one thing your product exists to do.
  3. Give each flow an end-to-end test, then break the flow on purpose and confirm the test fails.
  4. Stop your coding agent from merging to main or changing tests without a human review.
  5. Turn on error monitoring, and turn every new production error into a test.
  6. Run the critical-flow tests every day, not only before a launch.
  7. Have someone who didn’t write the change sign off each release.

What does it mean to verify AI-generated code?

Verifying AI-generated code means establishing, with evidence, that code an AI wrote does what the business needs. In practice, the product’s real user journeys are tested end to end, the tests are shown to fail when something breaks, someone other than the code’s author reviews the result, and production keeps being checked after release.

What this post does not claim

  • Nadella named a problem. He did not recommend any QA firm, and nothing here suggests he endorses Populosoft.
  • His spoken words are copied word for word from the transcript published with the episode, and his written words from his blog. YouTube times are chapter starts or approximate.
  • Our testing agent uses Anthropic’s Claude. Our independence is that we are a separate team, not a different model family.
  • Sonar’s figures come from a company that sells code-quality tools.
  • Populosoft is not affiliated with Microsoft, The Pragmatic Engineer or Anthropic. Company and product names belong to their owners.

Sources

Tell Us What You Are Shipping. We Will Tell You What We Would Test.

A short call, no obligation. Or start with a free scan.