A benchmark tells you whether a model can do a task. It doesn’t tell you whether a person gets what they asked for. Brello is held to the second, so we test the whole product the way people use it, and count what it changed rather than what it said.
Four layers
| Layer | What | How | Result |
|---|---|---|---|
| First use | 310 requests a new person types in their first ten minutes | Played end to end on seeded sandboxes, on the command line and on the app window’s own path; scored by code | 310 of 310 on both paths |
| Real use | 353 scenarios, 1,301 phrasings | On a same-shape copy of a lived-in Mac’s folders; scored by code, then read by a strict reader | 1,289 good, 0 wrong; every check passes |
| Acceptance | Every page and every control of the app | In the real app, driven by its development autopilot, every capture looked at | Pages 11 of 11, the popup 15 of 15 |
| Unit tests | The package and the app | swift test, the app’s tests, and a privacy test that reads every source file | 2,475 and 93 pass |
What a new person types
The first-use suite is what a new person types in their first ten minutes: vague requests, chatty ones, facts, apps, files, follow-ups and nonsense. Each case says what a good agent does (answer, ask one short question, act, act after asking, or refuse) and code checks the result by reading the sandbox afterward. What Brello says it did is never the evidence. The story of its first run is in The first ten minutes.
Every case plays twice: through the command line, and through the app window’s own path, headless, with each message sent as the Ask bar sends it. The second path exists because the first one missed something: on October 20, follow-up questions were found broken in the window while they worked on the command line. The bar is 95% on both paths, with every follow-up case passing. The latest run handled 310 of 310 on both.
Every scenario, every phrasing
The real-use suite holds 353 scenarios a person would try, each written several ways: casual, terse, polite, with a typo, in British English, and rambling. That makes 1,301 phrasings. Requests that change files run on a same-shape copy of a lived-in Mac’s Desktop, Documents and Downloads: the real names, sizes and dates, with empty sparse files in place of their contents, cloned fresh for each run and checked back to its first state afterward. Questions about the Mac itself are read, and only read, on a real one.
Each reply is scored twice. Code checks what changed. Then a strict reader marks it good, weak or wrong, and weak counts as failing. On October 25, 1,289 of the 1,301 phrasings were good (99.1%), none was wrong, and code’s checks passed on all of them. One scenario in four is played again on the window’s path and must agree; 93 of 93 did.
The real app, by hand and by script
Suites play the product’s code. Acceptance plays the app. A development build drives every page and every control of the real window and the popup, captures each moment, and every capture is looked at. The second run, on October 25, found real bugs no suite had: Health’s question button answered with a video file whose name matched its words, and the keep-awake time was cut short in the menu bar. Each was fixed with a test, and the run’s pages and popup controls then passed 11 of 11 and 15 of 15.
Some things only a person can check: a system-wide shortcut, dragging a file out to another app, a first launch with nothing installed. Those are listed as not checked, never as passed.
The rules we keep
- What is tested is what a person uses. The window’s path, not just the command line; a copy of a real Mac’s folders, not just a tidy sandbox.
- A test is never loosened to pass. Changes are audited for weakened checks before they merge.
- The model’s account is never the record. Code reads the disk, the setting or the document afterward.
- Our own Mac’s settings are never changed by a test. Every system effect runs on a made-up Mac in tests, and real changes are checked only in a separate user account set aside for it.