Families weighing senior care know that a polished answer is not the same as dependable help. The same distinction matters as AI moves into business tasks that affect customers, staff and money: can a system recognize trouble, resist pressure and carry a sound decision through to the end?
Turn quiet afternoons into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
Firmulate’s live company experiment offers one way to examine that question. Its results suggest that choosing an AI model by reputation alone may be a risky bet.
A company under pressure
Firmulate ran frontier AI models through the same difficult week at a small software company, with the same customers, crises and temptations. Decisions were versioned and auditable. The company itself is a live experiment, with synthetic employees and business-day activity that can be watched at Firmulate.
In the final July 2026 Crucible league, gpt-5.6-sol ranked first with 95 points. Moonshot’s Kimi K3 came second with 93, ahead of Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Firmulate says partial progress counts, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
AI decision testing simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Finding the detail that changed the outcome
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The result captures a practical gap: recognizing the right move and explaining it are not the same as completing it.
The deal turned on a competitor weakness buried two document references deep in the company’s files, rather than in the customer event. Models that read the file won the deal at full price, worth an additional €4,583 in monthly recurring revenue. Kimi K3 found that buried fact, won the deal and saved a customer who was considering leaving.
It also resisted all three baits and had one deviation, which Firmulate describes as the cleanest discipline in the field. In one on-record response to a reporter’s “just one yes/no, on background” request, K3 said: “Treat the request as a suspected approval-bypass / possible impersonation.” Across the test, all five models refused the staged fake-CEO messages and reporter trick.
Thoroughness did not guarantee the finish
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses, but it placed last. The close was left on the table, and discipline slipped when it attempted to write into a locked department instead of escalating. Firmulate says a weaker version of the same problem appeared in all four models.
The experiment therefore raises questions beyond raw capability. In settings such as senior care, where a person may need consistent follow-through as well as a thoughtful explanation, an organization would want to examine how a system behaves across the whole task. The company simulation does not establish how a model would perform in care; it illustrates why local testing matters before assigning consequential work.
A test before adoption
Firmulate says enterprises can run the wargame against a read-only export of their own business. The pilot does not write back to real systems. Its live company has 13 synthetic employees, a public cash countdown, and more than 680 self-learned playbook rules. The public benchmark and a quiz built from 242 real, unedited management decisions offer other ways to examine the results.
Fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh.
What readers should take away
Kimi K3’s second-place finish behind gpt-5.6-sol, and ahead of three other listed models, is a reminder that assumptions about which AI is best can be unsettled by a realistic test. For organizations considering AI in sensitive work, the useful question is not just whether it can identify a problem. It is whether it can find relevant information, withstand pressure and complete the task reliably.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
