Meta engineering team proposed introducing LLMs for Catching Just-in-time Testing back in 2025.
They proved it’s doable and scalable as of 2026 February – see the paper.
This is a great example of agentic engineering.
“Catching” and “Just-in-Time” – What do these mean?
If you aren’t a software engineer, think of code like a massive, million-piece LEGO castle that thousands of people are building at the same time.
The Old Way (Hardening): You add a new tower to the castle. Once you’re done, you check if the front door still opens. This is “Hardening”—making sure you didn’t break what was already there.
The New Meta Way (Catching): The moment you start sketching a new tower, an AI “inspector” looks at your drawing and says: “Wait! If you build it this way, the weight will make the floor underneath it collapse.” It creates a “Catching Test”—a simulation specifically designed to prove your new design has a flaw before you even pick up a LEGO brick.
In the past, automated tools were like “The Boy Who Cried Wolf.” They would find thousands of “potential” bugs, but 99% of them were fake alarms. Engineers eventually just turned the notifications off.
So Meta engineering team didn’t just make the AI smarter at finding bugs; they made it smarter at filtering them. They built an “AI Assessor” that reads the code like a human senior engineer would. It asks: “Is this a real fire, or just smoke?”
By filtering out the junk, they reduced the “noise” by 70%. This means when an engineer gets a ping from this system, they actually listen.
Long story short,
Meta proved that you can use AI to hunt for bugs in real-time across a codebase of hundreds of millions of lines. It caught 8 confirmed bugs – 4 of which were “critical” failures that would have crashed the apps for millions of users.
