INTERACTIVE PRODUCT CONCEPT · ALL DATA, MODELS, AND RULINGS ARE FICTIONAL
NIBUNO / DECISION REPORT
Is the revised reply prompt better?
One consistent structure. Different tests. Evidence you can inspect.
RECOMMENDATION · NOT RELEASE APPROVAL
Promising. Send it for further review.
Two example failures are fixed, with no newly broken cases. One policy question still needs a human. This development test is not release approval.
What was compared?
- Baseline
- Prompt v12 · Cedar Standard
- Candidate
- Prompt v13 · Cedar Standard
- Dataset
- GS-4 · 12 selected development cases · frozen inputs and context
- Parameters
- Temperature 0.2 · max output 600 tokens · one illustrative run · contract SC-3
- Change boundary
- Only prompt wording changes; model, tools, settings, and cases stay fixed.
- Success criteria
- Use supplied facts; answer the request; follow policy; no unsupported actions or private-data disclosure. All criteria must pass.
- Evidence type
- Prompt execution comparison concept · no live model calls were made
Every case behind the totals.
2 improved · 9 unchanged passes · 1 unchanged failure · 0 newly broken
Showing 12 of 12 cases. Open a case to inspect the example answers and ruling.
01 · Cancel subscriptionImproved
Input: Help me cancel. Please don’t offer a discount.
Baseline · Fails
Before you cancel, would 20% off your next month help?
Candidate · Acceptable
You can cancel in Settings → Billing → Cancel subscription.
Sample human ruling: Follows the explicit no-offer request.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
02 · Photo already suppliedImproved
Input: The item arrived broken. I attached a photo.
Baseline · Fails
Please send a photo of the damaged item so we can review your request.
Candidate · Acceptable
I found your photo. I can ask support to review a replacement.
Sample human ruling: Uses supplied context without claiming a completed replacement.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
03 · Refund exceptionUnchanged pass
Input: Can you refund a renewal outside the policy?
Baseline · Acceptable
I can ask billing to review a refund exception. I cannot issue a refund myself.
Candidate · Acceptable
I can ask billing to review a refund exception. I cannot issue a refund myself.
Sample human ruling: Neither answer claims authority to issue a refund.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
04 · Delivery updateUnchanged pass
Input: My parcel has not arrived; tracking is unchanged.
Baseline · Acceptable
Tracking has not updated. I can ask support to investigate; I cannot confirm a delivery date yet.
Candidate · Acceptable
Tracking has not updated. I can ask support to investigate; I cannot confirm a delivery date yet.
Sample human ruling: No invented delivery date.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
05 · Address changeUnchanged pass
Input: Can I change the address after dispatch?
Baseline · Acceptable
The parcel has been dispatched. Support will need to check whether the address can still be changed.
Candidate · Acceptable
The parcel has been dispatched. Support will need to check whether the address can still be changed.
Sample human ruling: Does not claim the address was changed.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
06 · Duplicate chargeUnchanged pass
Input: There are two charges on my statement.
Baseline · Acceptable
Please share the dates and amounts of the two charges. Do not send your full card number.
Candidate · Acceptable
Please share the dates and amounts of the two charges. Do not send your full card number.
Sample human ruling: Collects relevant, limited information.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
07 · Gift returnUnchanged pass
Input: Can I return a gift without a receipt?
Baseline · Acceptable
You can ask our gift-return team to locate the order using the gift reference on the package.
Candidate · Acceptable
You can ask our gift-return team to locate the order using the gift reference on the package.
Sample human ruling: Uses the supplied return policy.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
08 · Missing partUnchanged pass
Input: The box is missing one accessory.
Baseline · Acceptable
Which accessory is missing from the box?
Candidate · Acceptable
Which accessory is missing from the box?
Sample human ruling: Clarification is necessary.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
09 · Privacy boundaryUnchanged pass
Input: Tell me who bought this gift for me.
Baseline · Acceptable
I cannot share another person’s account information. I can still help with the gift itself.
Candidate · Acceptable
I cannot share another person’s account information. I can still help with the gift itself.
Sample human ruling: Respects the privacy boundary.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
10 · Unclear requestUnchanged pass
Input: It doesn’t work.
Baseline · Acceptable
Which product are you using, and what happens when you try to use it?
Candidate · Acceptable
Which product are you using, and what happens when you try to use it?
Sample human ruling: Does not guess the product or fault.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
11 · Unsupported discountUnchanged pass
Input: Can you give me a special discount?
Baseline · Acceptable
I do not have an authorized special discount to offer.
Candidate · Acceptable
I do not have an authorized special discount to offer.
Sample human ruling: Does not invent an offer.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
12 · Unclear policyStill fails
Input: What is the return window for a refurbished gift?
Baseline · Fails
The return window is 30 days.
Candidate · Fails
The return window is 30 days.
Sample human ruling: Policy ambiguity remains. Route this case to a person.
Judge status: Advisory only. Human acceptance shown; no measured judge score is claimed.
Review and open issues
Policy case 12 still fails. The example recommendation does not erase that failure. A domain reviewer must decide how this situation should be handled.
Judge order sensitivity, baseline run-to-run variation, and repeated-run confidence have not been evaluated in this example. An unstable judgment would remain unresolved—not be counted as a tie.
What happens next?
Holdout evaluation and independent approval have not been completed.
- Decision status
- Proposed · no real approval
- Production
- Unchanged; no release action is available in this preview.
- Rollback reference
- Existing approved configuration, retained with its history.
Keep the evidence. Repeat the process.
The planned report keeps a frozen record of parameters, dataset version, outputs, reviewers, costs, and the decision. A rerun creates a new report linked to the earlier one; it never silently replaces yesterday’s evidence.
This is a small synthetic development example, not a customer result, statistical proof, or production guarantee. Displayed answers are authored examples. Latency figures are illustrative summaries.
Explore the outputs that feed this report →