Methods and sources

On 5 September, a limited review checked six headline figures per file against the source calculations. Four values were incorrect across two outputs: three in the Haiku Run 1 deck and one in its Run 2 email. Related wording observations were recorded separately. These results do not validate every calculation or statement in a file.

The Haiku Run 1 email and both Fable 5.1 emails missed the word limit. Sonnet’s Run 2 email stopped at its spending limit after saving its unchanged file. Its reported cost and time end at that stop; later activity is excluded.

On 6 September, we inspected the Run 1 One-pagers and decks as rendered pages and read the original emails. The design comments describe visible layout and formatting. Reviewers could see model names. No blind review assessed overall quality, and no numerical design score was assigned.

Four token charges are calculated from saved usage and prices: other input, input saved for reuse, reused input, and generated output. Their sum, plus active session time at $0.08 per hour, reproduces every recorded task cost when rounded to cents. The token prices were saved on 3 September; the active-time rate was verified from official documentation on 5 September and was not in the original price file.

Completion time measures the elapsed wait from the first task message to the recorded stop. Billed active time is a separate service measure. Tools have no separate recorded dollar charge or exclusive measured execution time. A request’s charge therefore cannot be divided among its tool calls from these records.

The preview calculation divides request token charges at the first recorded image-opening call. The later share includes requests with a higher request number and excludes session time. It includes all later activity, so it is not a measured price for revision alone.

The activity exhibit groups all requests from the five Run 1 One-pagers using reviewed tool instructions. Document work includes writing or changing PDF programs, rendering pages, reading document code, opening previews, and checking file properties or text. A command can combine several document actions; those charges remain together. Mixed document and non-document actions remain separate, as do responses without tools. Each amount includes the entire request’s input and output. It does not measure the tool’s own cost, time, or potential savings.

Model names identify configured sessions. Individual request records do not identify the serving model, so calculations apply each session’s saved model price to its recorded usage. Complete request inputs were not preserved. Exact reuse of particular instructions or tool results is unknown. The main page’s request-loop and caching diagrams explain the process; they do not reconstruct a recorded request.

Two runs in a fixed order cannot establish typical variation, causes for every action, or future savings. A grouped command can contain several operations, and a response can request several tools. Neither grouping nor preview counts establishes simultaneous execution, effectiveness, or complete review.

The test software did not explicitly set the effort control, which influences token use. Records show high effort for four settings and no value for Haiku. Three early Run 2 sessions had spending limits that differed from the plan. Effects beyond the recorded Sonnet email stop were not measured. The downloaded procedure documents these setup differences.

Future research will test common assumptions about applied AI and quantify the effects of recommended practices. We will compare the quality, usage cost, and completion time of the same work under different setups. Skills, templates, independent checks, MCP and API access, tool-call efficiency, model choice, conversation history, and caching are planned comparisons. Their effects on quality, cost, time, and consistency remain to be measured.

Uber’s software-factory research breaks spending into model requests, token use, and prices. Its examples compare tool loading, repeated status checks, and returning smaller results. Ramp’s Inspect reports describe giving agents a working environment for tests and visual checks, then directing them at specific failures.

These company reports inform future comparisons. Their reported savings and performance do not measure the results of this study.