An upgrade from 85% to 86%.
A founder posed a useful hypothetical while describing Nibuno: one model scores 85%, another scores 86%, and the second costs twice as much. Is the upgrade worth it?
Those numbers are an illustrative scenario, not a benchmark. Their value is the question they force: what exactly improved, and what did each acceptable result cost?
Hold the prompt and test cases fixed.
Keep the prompt, representative inputs, output requirements, and scoring criteria fixed when you compare models. Record the model versions and relevant settings. Where configuration cannot match, make the difference explicit.
A separate prompt makes this comparison easier to organize. You can ask whether a model change improves a defined behavior without also changing the instructions and losing track of the cause.
Read the cases that improved or regressed.
In an illustrative 100-case comparison, 85 and 86 acceptable results differ by one net success. That does not mean only one case changed. The new model might fix several cases and break several others.
Read the individual wins and losses. A correct answer on a critical case can justify more expense. A small stylistic improvement may not. Check failures that an overall score would conceal.
Calculate the cost of acceptable work.
Suppose the first run costs $1 and the second costs $2. At 85 and 86 acceptable results, that is about 1.18 cents and 2.33 cents per acceptable result. The second model costs about 1.98 times as much per success in this hypothetical.
Use actual usage charges when you run the comparison. Include retries, fallback calls, latency, and any human review that changes between options. Repeat cases when output variability could alter the decision. A small sample cannot settle every tradeoff.
Include the current model in the comparison.
One of the real questions behind Nibuno concerned a review-writing feature built only months earlier. New models had appeared. The team needed to decide whether an upgrade was justified.
The right outcome might be an upgrade, a cheaper replacement, or keeping the existing model. Nibuno’s planned model comparison would show the prompt, individual outputs, costs, and reviewer decision for each run.