While working on my upcoming collaboration with Ileana from UX+AI, I tested five open models alongside GPT and Claude. Several produced convincing product plans that changed a requirement we’d explicitly supplied.
Our joint article examines whether the models’ questions help a product professional do the work. This is the technical companion: a closer look at the test setup, how requirements survived into the generated flows, and what I’d need to measure before moving the task to another model.
In This Edition
-
What the Test Compared
-
Following One Rule
-
Where the Rule Changed
-
The Cost of Review
-
Before Switching Models
What the Test Compared
I want alternatives I can trust with specific jobs. That was part of the motivation behind the $30 Hermes stack. Here, I wanted evidence about the quality of the work I’d be moving.
The task was to outline an in-app support agent. The brief allowed refunds up to €50 and several order actions, with a human handoff after three unresolved turns. A separate answer pack supplied the missing business context and restrictions.
The open cohort contained GLM 5.3, DeepSeek V4 Pro’s 0813 checkpoint, Kimi K3, Qwen 3.8 Max, and MiniMax M3. These are the labels in the saved run records. The exact historical Qwen checkpoint remains unresolved.
The closed cohort contained GPT-5.6 Terra, GPT-5.6 Luna, Claude Sonnet 5, Claude Opus 5, and Claude Fable 5.1.
“Open” here means the experiment’s open-weight cohort. I used hosted endpoints through Fireworks. The comparison says nothing about local inference speed, hardware requirements, or self-hosting costs.
Each model received three conditions on September 3, 2026:
-
Brief alone: outline the flows from incomplete instructions.
-
Question gate: receive the brief with instructions to ask consequential questions before designing.
-
Answer pack: receive the brief and missing context together, then design.
Every condition started fresh. The final round never saw the model’s earlier questions, so it tested whether supplied context improved the design independently of a conversation.
That produced 30 saved responses, one per model per condition. The records describe fresh context with no tools or web access.
The task text matched, but the delivery routes differed. Open models ran through Fireworks, Claude through Anthropic’s API, and GPT through Codex with its surrounding instructions. These are comparisons of the setups I used. The results don’t isolate model behavior from the environment serving it.
The records also have limits. The runner scripts and metadata sidecars referenced in the run notes aren’t available in the evidence folder. The saved answers support inspection of the output, but they don’t let someone independently reconstruct every execution detail.
Across both cohorts, every model delivered a design from the incomplete brief. With the question gate, all ten stopped short of delivering flows, although some supplied assumptions or default answers. The joint article explores the professional value of those questions. My inspection here concentrates on the final round, when the missing context was already present.
Following One Rule
The answer pack included this instruction:
Before any action that can’t be reversed, the agent escalates first.
The original brief granted permissions. The answer pack restricted when those permissions applied. A refund below €50 still needed a handoff if it was irreversible.
I followed that requirement from the general principles into the eligibility tables and individual action flows. I checked whether an irreversible action remained blocked pending human review, and whether another passage introduced an exception.
This distinction mattered because several responses repeated the rule correctly in one section and changed it elsewhere. Finding the words “escalate first” would have missed those contradictions.
This was a manual inspection of written plans. No support agent executed a refund or cancellation, and I haven’t converted these observations into a professional UX score. Ileana’s grading is still pending.
Where the Rule Changed
Three kinds of repair kept appearing: removing invented exceptions, separating customer confirmation from human review, and reconciling conflicting sections.
Exceptions to the requirement
GLM 5.3 built a detailed eligibility matrix. Its refund row allowed autonomous execution below €50 while marking the action as irreversible. The response used bounded risk to justify an exception the answer pack hadn’t granted.
GLM also proposed a policy distinction between discretionary and transactional refunds. It marked that proposal for resolution, but parts of the design already depended on it. I’d keep the useful structure and block those branches until the policy was settled.
MiniMax M3 repeatedly required escalation for irreversible actions “under ambiguity.” Adding those two words narrowed the rule. A certain but irreversible action still required human review under the supplied instructions.
Claude Opus 5 qualified its irreversible-action escalation entry with customer detriment. Its cancellation discussion also allowed confirmation or escalation where policy required it. Both passages needed reconciliation with the unconditional requirement.
Confirmation replacing human review
Kimi K3 described cancellation as irreversible, then proposed previewing the consequences, obtaining customer confirmation, and executing. It explicitly asked stakeholders to approve an interpretation that exempted the listed authorized actions.
That warning made the disagreement visible. The flow nevertheless depended on approval it hadn’t received.
Claude Sonnet 5 also flagged its interpretation for review while allowing cancellation after a confirmation checkpoint. It distinguished confirmation from escalation, then treated confirmation as a possible reading of the requirement.
Claude Fable 5.1 made a provisional assumption that pre-fulfilment cancellation could proceed after customer confirmation, pending a decision about reversibility. I’d carry that unresolved decision into the action branch itself so the flow couldn’t be read as approved.
Contradictions across sections
Qwen 3.8 Max explicitly required escalation before irreversible cancellation. Later, it offered confirmation if policy deemed the action safe, or escalation if it couldn’t be reversed. Customer confirmation had become an alternative inside a passage about irreversible actions.
GPT-5.6 Luna preserved escalation in its cancellation flow, but added “where policy requires it” in its opening principles. I’d make the requirement consistent before using the plan.
DeepSeek V4 Pro preserved pre-action escalation in the cancellation section and explained that the refund ceiling didn’t guarantee eligibility. Its opening nevertheless allowed actions when safe and reversible “or explicitly allowed.” That clause needed removal or clarification so it couldn’t override the handoff.
DeepSeek also invented an exact address-change cutoff before packing or shipping. The answer pack made eligibility dependent on fulfilment and carrier state without specifying that cutoff.
Terra gave me the clearest cancellation path on this check: confirmation followed by human review unless company policy established a safely reversible window. Its cancellation row repeated the handoff requirement. It still proposed return-label generation, a capability the supplied material hadn’t established.
These differences give me specific repairs to test. They don’t establish that one model is generally better at product work, or that any response is ready to use without review.
The Cost of Review
Qwen’s final-round answer contained roughly 6,000 whitespace-delimited words, including Markdown and table syntax. Terra’s contained about 900. Those counts describe the visible files rather than billed tokens.
Qwen’s conflicting passage appeared far below its main flows. In this case, reviewing the complete answer meant checking considerably more text for agreement with the same requirement.
The run notes recorded about 45 seconds for MiniMax’s informed design, 194 for GLM’s, and 26 for Terra’s. These are single observations through different serving setups. They don’t establish repeatable speed differences.
I didn’t measure human correction time or reconstruct comparable bills across the three routes. This experiment therefore doesn’t establish the cheapest replacement. It identifies review work that a cost comparison would need to include.
For the next round, I’d record inference cost, elapsed time, review time, and unresolved requirement violations separately. A fast response still needs repair if its cancellation flow grants the wrong permission.
Before Switching Models
For a repeat test, I’d start with DeepSeek as the open candidate and Terra as the closed reference. Both explicitly preserved the cancellation handoff in the passages I checked. DeepSeek still needs its opening exception and invented cutoff addressed. Terra needs review for added capabilities.
I’d keep Kimi in the comparison because it exposed its disputed interpretation clearly. My next instruction would require unresolved policy choices to remain blocked inside the flows. That’s a proposed change to test, not an improvement I’ve measured.
For GLM and MiniMax, I’d check whether revised instructions prevent the specific exceptions above. For Qwen, I’d also test a shorter output requirement and inspect consistency across sections. The Sonnet response shows why the incumbent needs the same scrutiny.
To run this comparison on your own work:
-
Choose a task you know well enough to review, and give both setups identical task text and reference material. Record differences in surrounding instructions and tool access.
-
Before reading the answers, identify the requirement whose violation would make the output unusable.
-
Follow that requirement into each proposed action. Check the tables and exceptions as well as the opening explanation.
-
Record the repair needed and the time spent finding it. Repeat with model names hidden during review where possible.
A written plan is also only one part of a working system. Actual permissions need enforcement where actions execute, which is the implementation question I covered in my approval-gates article.
Our upcoming joint article with Ileana at UX+AI examines whether the models’ questions help a product professional make better decisions. I’ll add the link when it’s live. This companion gives me a narrower result: an open candidate worth retesting, a closed reference, and concrete failures to check before moving the work.
