How we test Brello

Four layers, each played on the real product: 310 first requests on both paths, 353 scenarios in 1,301 phrasings on a same-shape copy of a lived-in Mac, every page and control in the app itself, and 2,475 package tests.

A benchmark tells you whether a model can do a task. It doesn’t tell you whether a person gets what they asked for. Brello is held to the second, so we test the whole product the way people use it, and count what it changed rather than what it said.

Four layers

Table 1. The four layers and their latest results, as of October 25 and 26, 2026, on the merged product.
LayerWhatHowResult
First use310 requests a new person types in their first ten minutesPlayed end to end on seeded sandboxes, on the command line and on the app window’s own path; scored by code310 of 310 on both paths
Real use353 scenarios, 1,301 phrasingsOn a same-shape copy of a lived-in Mac’s folders; scored by code, then read by a strict reader1,289 good, 0 wrong; every check passes
AcceptanceEvery page and every control of the appIn the real app, driven by its development autopilot, every capture looked atPages 11 of 11, the popup 15 of 15
Unit testsThe package and the appswift test, the app’s tests, and a privacy test that reads every source file2,475 and 93 pass

What a new person types

The first-use suite is what a new person types in their first ten minutes: vague requests, chatty ones, facts, apps, files, follow-ups and nonsense. Each case says what a good agent does (answer, ask one short question, act, act after asking, or refuse) and code checks the result by reading the sandbox afterward. What Brello says it did is never the evidence. The story of its first run is in The first ten minutes.

Every case plays twice: through the command line, and through the app window’s own path, headless, with each message sent as the Ask bar sends it. The second path exists because the first one missed something: on October 20, follow-up questions were found broken in the window while they worked on the command line. The bar is 95% on both paths, with every follow-up case passing. The latest run handled 310 of 310 on both.

Every scenario, every phrasing

The real-use suite holds 353 scenarios a person would try, each written several ways: casual, terse, polite, with a typo, in British English, and rambling. That makes 1,301 phrasings. Requests that change files run on a same-shape copy of a lived-in Mac’s Desktop, Documents and Downloads: the real names, sizes and dates, with empty sparse files in place of their contents, cloned fresh for each run and checked back to its first state afterward. Questions about the Mac itself are read, and only read, on a real one.

Each reply is scored twice. Code checks what changed. Then a strict reader marks it good, weak or wrong, and weak counts as failing. On October 25, 1,289 of the 1,301 phrasings were good (99.1%), none was wrong, and code’s checks passed on all of them. One scenario in four is played again on the window’s path and must agree; 93 of 93 did.

The real app, by hand and by script

Suites play the product’s code. Acceptance plays the app. A development build drives every page and every control of the real window and the popup, captures each moment, and every capture is looked at. The second run, on October 25, found real bugs no suite had: Health’s question button answered with a video file whose name matched its words, and the keep-awake time was cut short in the menu bar. Each was fixed with a test, and the run’s pages and popup controls then passed 11 of 11 and 15 of 15.

Some things only a person can check: a system-wide shortcut, dragging a file out to another app, a first launch with nothing installed. Those are listed as not checked, never as passed.

The rules we keep

  • What is tested is what a person uses. The window’s path, not just the command line; a copy of a real Mac’s folders, not just a tidy sandbox.
  • A test is never loosened to pass. Changes are audited for weakened checks before they merge.
  • The model’s account is never the record. Code reads the disk, the setting or the document afterward.
  • Our own Mac’s settings are never changed by a test. Every system effect runs on a made-up Mac in tests, and real changes are checked only in a separate user account set aside for it.