Before an AI agent books a studio, answers a listener or negotiates a sponsorship, a creator wants to know what it will do when the week goes sideways. Will it spot the real problem, resist a dubious request—and finish the job? Firmulate is testing those questions by putting AI models in charge of a company under pressure.
Get audio and creator gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A company on stage
The experiment gives frontier models the same small software company, the same customers, the same crises and the same temptations. Each decision is versioned and auditable. The company is synthetic, but its money mechanics are real: 13 synthetic employees work against a burn of €105,000 a month and €2,300 in monthly recurring revenue. A public cash countdown and more than 680 self-learned playbook rules make the unfolding test watchable at Firmulate.
The final Crucible League, dated July 2026, ranked gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77, and Opus 4.8 fifth with 73. The do-nothing baseline scored 26. The benchmark counts partial progress, but one breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Spotting the crisis is not closing the deal
All models spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal their own analysis had earned. The gap, summed up by the experiment, was “Same diagnosis, same pitch — no signature.” Recognizing the right move did not guarantee carrying it through.
The deal hinged on a detail buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue. It is a useful reminder for creators considering AI for business work: the decisive information may be in the records behind the conversation, and understanding a pitch is not the same as completing it.
The test also put the models through fake CEO messages that escalated over three stages, then a reporter’s “just one yes/no, on background” trick. All five refused. Kimi K3 explained its judgment on the record: “Treat the request as a suspected approval-bypass / possible impersonation.” The results test both follow-through and boundaries under pressure.
Thoroughness has its own limits
Opus 4.8 was the most thorough participant, adding more than 80 learned rules and producing the deepest analyses, yet it finished last. It left the deal unsigned and let discipline slip, attempting writes into a locked department instead of escalating. The same weakness appeared, more mildly, in all four models.
There is a fairness detail for readers interpreting the table: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. The league is one view of performance in this experiment, not a guarantee of how any model will behave in a creator’s own business.
Firmulate also turns 242 real, unedited management decisions into a “guess the model” quiz at firmulate.com/quiz.html. It offers a different way to engage with the choices behind the rankings—and to notice how often a sensible-sounding answer still leaves a task unfinished.
From watching to trying
For a music business, the point is practical: an AI assistant may face customer churn, pricing choices, a rival’s offer or a request that should trigger caution. Firmulate’s enterprise pilot takes the wargame to a company’s own situation, using a read-only export to test crisis scenarios and produce a board report with model rankings and weak points in the company’s playbooks. Nothing writes back to real systems.
To explore a pilot using your own business data, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
