Build log

Entry 011 quoted 2.8, 2.9 and 3.6 seconds for three questions in a row on one sheet. Three in a row on one sheet is the least representative case there is: the second question rides on the first one’s rendered crop and cached prefix. Re-measured on the production box the same afternoon, the first question on a sheet took 8.5 and 6.8 seconds; the same sheet again, 3.6 and 2.6.

On a set nobody has opened, every sheet is the first one. That is the demo case and it is the customer case. The demo cue had been written around three seconds. It now leads with seven to nine, and a test refuses to let the warm figure appear anywhere without “same sheet again” next to it. Quoting three and then standing there for eight in front of two people is the moment a demo turns.

Then the contention claim. Entry 011 said two idle programs had been evicting the reader’s cached context. Four days later, on a quiet box, medians of three, a 37-sheet set: with nothing else on the model, a cold ask took 8.75 s and a warm one 2.17 s. With a second consumer sending one message every thirty seconds, 9.63 and 2.45. With it hammering continuously, harder than any person can type, 10.05 and 3.93. Zero cache evictions across the whole run, in 43 slot launches.

So the mere presence of a second consumer costs about ten percent on a cold ask, and it does not evict anything. What broke things in 011 was that particular pair of programs: one of them re-reading ten to sixteen thousand tokens into the model every turn, against a prompt cache a third the size it is now. The fix in 011 was right. The explanation was too broad, and this is the correction.

What was not shown: a second consumer shaped like an agent, with a hundred times the prompt, which is exactly the regime where eviction starts. That was not tested. And the model server’s resident memory grew from 19 GB to 26 GB across about fifty requests, which is not settled either.

Why these numbers can be trusted at all. Every long prompt in the run carried a unique tag, and the server reported the same number of tokens processed as sent, 3,485 of 3,485, on all nine of them. Without that check the cache serves the previous answer back and the benchmark measures nothing while reporting a very good time.

Entry 005 was twelve quotes out of twelve on a real bid package, and no conflicts found because none were there. That tests whether a citation is real. It says nothing about whether the tool catches anything. For that you need a document with a known answer, so we wrote one: a 38-page invented Division 23 specification with four contradictions planted in it and three near-misses that look like conflicts and are not. One of the four is the classic: a cross-reference that repeats an insulation thickness instead of pointing at it, and the value went stale when the other section was revised.

The scoring rules were written down before the first run. A planted pair returned as “not a conflict” is a miss, and the tool’s reasoning is recorded rather than argued with. A near-miss returned as a real conflict is a false alarm. A single pass is capped at six candidates, so no run could enumerate all seven items even if the model were perfect; that is a property of the cap and it is disclosed as one.

Five runs. Over the whole document it found two, three, two and two of the four. Never four. The two it never missed both have a number on each side: an insulation thickness and a supply voltage. The two it kept losing have none: a damper driven closed in one section and open in another, twenty-four pages apart, and who has to be present at start-up. A pass is capped at six candidates, and the model kept spending three of them on correct negatives it thought worth reporting. Narrow the range to three sections and the damper conflict comes back first. No near-miss was called real in any run, and neither was any of the eleven pairs the model raised on its own. Each pass took 34 to 64 seconds.

The quote locator earned its own line. It is the check that decides whether a passage the model quoted actually exists in the document, and more than once a naive version of it has accused a correct model of making things up. Once it reported eleven of twelve when all twelve were there. In a two-pass run it dropped three of twenty-four quotes as unlocatable, and all three were the checker’s fault. And in the first run above, the model found the start-up conflict outright and the checker threw it away. Under the rules that is still a miss, and it is ours rather than the model’s. Every time, the fix was in how we read the PDF.

So the honest pair of numbers is this. Citation: twelve of twelve, on somebody else’s real document. Catching: two or three of four, on a specimen we wrote ourselves. Spec against a drawing schedule, and one document against another, are not built, and the front page does not claim otherwise.

A question across a whole set works like this: pick the sheets most likely to carry the answer, read each one, answer with the sheet attached, and open with what was and was not read. That opening line is the point of the feature. A bare “not found” hides whether it looked at all.

First real run, on the 333-page set, every page indexed: the answer opened with “Page coverage of this set is not known.” The indexer had written the page count under one field name and the answerer read it under another. Each module’s tests passed, because each was tested against a fake of the other that encoded its own author’s assumption. It was one of three such mismatches found that morning; in the worst of them, twenty-one tests were green against a double that did not match the thing it doubled. Fixed the same morning; the next run opened with “Searched all 333 pages.”

Search then failed in the other direction. “MCC” returned 40 pages of that manual. The manual has no motor control center in it. The hits were pages containing the word “motor”. Fixing it took the count to zero, which is the right answer and looks like a regression. Two more from the same review: a code that appeared on eight scanned sheets and two readable ones returned nothing, because a scoring term had gone negative and was thrown away; and “3 in the room” against “5 in the corridor” was reported as a conflict of in values, a disagreement that appears in neither input. Units only now, and they have to be unambiguous.

Timing, from the job records rather than the plan. One machine answers one thing at a time, and a whole-set question reads five sheets at 8 to 26 seconds each. The four recorded five-sheet questions took 43 to 100 seconds wall clock. The demo cue said forty to fifty. The record says otherwise, so the front page now says otherwise.

The rule that was dead the whole time. A named mark is supposed to outrank everything else when choosing sheets; that is the entire reason to match codes exactly. It never ran in production. The index never supplied the one field the router checks for, and the router degrades quietly without it. Every mark test passed, against a fake index that met the contract. The contract was right; the index was the half that never met it. A test suite that passes against a stand-in proves the stand-in.

The box runs one model. Since roughly the week it came up it also ran an agent front-end and a memory service that spoke to that same model, both left over from an earlier experiment, both still holding their place in the queue. A question about a drawing had to wait its turn behind whatever they were doing, and worse, their cached context and ours kept evicting each other.

We stopped both and measured immediately, on the machine that serves the app, timing the whole round trip rather than the model. Three consecutive single-sheet questions: 2.8 s, 2.9 s, 3.6 s — mean 3.1. The same question used to take 6.5 to 20 seconds, and about 40 seconds at its worst.

Memory barely moved: 32 GB to 31. So this was never a capacity problem. It was contention, which is a much less interesting explanation and the correct one.

What that buys is not a benchmark, it is a different demo. Under twenty seconds you need a progress bar, a line to say while it thinks, and a reason for the wait. At three seconds you hand somebody a phone and they ask the second question before they have finished reading the first answer.

The code is still written as though the constraint exists — one job at a time, one hard ceiling on prompt size — and it stays that way. One worker against one model is the right shape whether or not anything else is competing for it, and the ceiling is there to catch a runaway prompt, not a co-tenant. Removing defensive code because today's measurement no longer needs it is how you get to rediscover the measurement.

The whole change is reversible in one command and nothing was deleted. That was deliberate: we wanted the option to put the contention back and re-measure if the number ever looked too good, rather than having to rebuild the conditions from memory.

Up to this point the tool graded itself on whether it found a schedule and how many rows it got out of it. Both of those can only go up. A parser that reads a general note as a table looks, by that measure, like a parser that got better.

So we started hand-labelling. Take a real sheet, work out what the table actually says, check every mark and every dimension against the printed page, and score the tool against that instead of against itself. One school-district set has a door schedule and a window schedule, one page each, fifteen rows in total, hand-checked mark by mark.

The tool reported that document as ok. It had extracted 138 rows. Three of them were right. That is a precision of 0.02 — ninety-eight percent of what it handed back was not a schedule row at all — on a document that had been sitting in the healthy column the entire time. ok is not the same as correct, and no row count would ever have told us that.

The score history is the part we want to keep, because it looks like failure and is not. The first labelled document gave a macro-averaged F1 of 1.00. Honest, and nearly meaningless: one document. Six documents took it to 0.59. Eight took it to 0.44. The number fell every time the measurement got better and the software stayed still.

Then the gate did its job on us. A change we liked scored 0.52 — the best the eight-document measurement has ever returned — and it broke 32 documents to get there, three of which had hand-labelled answers proving the new behaviour was wrong. The gate refused it. We put the parser back to the last build that had actually passed, which scores 0.44 with nothing regressed, and that is what stands today.

We would rather publish 0.44 that survived a rule than 0.52 that we chose. The whole argument for this product is that its answers are checkable; a score we can overrule is not a check.

Why the rule has teeth. A regression can be waived — sometimes the new behaviour is right and the old number was flattering. But a waiver is only accepted if that exact document has a hand-labelled answer attached, in the same run. So excusing a regression costs the same work as proving it. Twenty broken documents can no longer be waved through on the strength of seven improved ones, because we would have to hand-read twenty sheets first, and by row nineteen we would have talked ourselves out of it.

An earlier round of accuracy figures all came from one survey run over the whole document collection. That survey had been generated against the experimental parser — the one that the same round of work measured, rejected and deleted. The comparison was scored with the version that had already been thrown away.

The tell was a timing. The survey recorded 0.0 seconds for a one-page document that demonstrably takes real work to parse and yields fifteen rows. It had not run slowly. It had not run at all. Everything downstream of that file is now quarantined rather than deleted, with a note saying why, because a wrong number you can still read beats a wrong number you only half-remember correcting.

So the "roughly one set in four" figure is withdrawn, along with every count that came out of that survey. The write-up it appeared in stays up as written, marked rather than quietly edited. This log is written as it happened, and a page that silently repairs itself is not that.

The corpus was also not what we said it was. Counted properly, by hashing every file rather than trusting its name: 218 files, 184 genuinely distinct documents. Thirty-four were byte-identical copies — one file had been downloaded four separate times and counted four times. Seven were HTML error pages saved with a .pdf extension, which the collector had faithfully filed as documents. Three are encrypted and the pipeline refuses them. What remains is 169 real drawing sets, and that is the population everything is now scored against.

Naming a document by its hash rather than its filename sounds like housekeeping. It is the difference between "this got better" and "this got counted twice."

What survives is the part that mattered: we taught the parser four new formats, each one fixed the document in front of us, and across the collection more sets broke than improved — the direction is corroborated by a second record kept at the time. So the change was reverted, and that judgement holds. The figures quoted alongside it needed re-running, and the counts above are that re-run.

What came out of it. There is now a file recording every decision this project has reversed, reconstructed by reading roughly a hundred and thirty-five megabytes of our own working transcripts rather than our own documentation — because the documentation records the current state and nothing recorded what had already been tried. It caught a live instruction to ship a build that had failed its own quality gate, sitting in a handoff file for hours after the decision had been reversed in conversation. A reversal that only exists in somebody's memory is not a reversal.

Everything up to here had been tested against a handful of documents. So we went and got more: 182 public drawing sets, 3.4 GB, pulled from school district bid portals, city permit portals, university facilities archives, and the free pre-approved accessory-dwelling plans that several California counties publish. Real sets, drawn by real people who had never heard of us.

The first honest number: it could find and read a schedule in about one set in four. Not because the reading is hard — because every drafting office formats a schedule differently, and ours had learned exactly one dialect.

The best result came from a state office building getting its windows replaced. Its schedule lists 278 windows by mark; its elevations carry those same marks as text. It matched 208 of 208 scheduled marks against the drawings. That is the check we actually want this thing to do, done on somebody else's real document.

Then we tried to generalise. Four changes, each one aimed at a format we had watched fail: headers that stack across two lines, width and height in separate columns, schedules with no orientation column at all, and one office's habit of setting the inches of 2'‑0" on a slightly different baseline from the feet, which split every dimension in half.

Every one of those fixes worked on the set in front of us. Then we ran the whole corpus again. Sets that could find a schedule fell from 54 to 42. Sets that could read rows fell from 45 to 32. Seven sets improved and twenty broke.

So we restored the previous version and kept the measurement instead. That is the entry: not the parser, the harness. Before this we tuned against whatever document was open and shipped on the strength of it looking better. Now one command scores 182 real sets against a pinned baseline, and a change that helps the document in front of us but costs twenty others shows up as a number instead of a good feeling.

The way through is one dialect at a time, each proved against the whole corpus before it stays. Slower. But we have now watched the fast way fail with a number attached to it.

A PDF plotted out of CAD still contains the real geometry — every wall, every opening, as actual line work with actual coordinates. So the tool stopped guessing at sizes from pixels and started measuring the drawing itself.

On a set where we had independent ground truth, a window measured 8.880 ft by 5.000 ft against a true 8.88 by 5.0. Zero error. The door measured 5.714 by 7.001 against a true 5.71 by 7.0.

That precision buys something better than a number. The schedule for that window said 8.9' × 5.0'. The label printed next to it on the elevation said 9'×5'. Three documents, three answers. Because the geometry could be measured, the tool can say which one is wrong: the drawn opening agrees with the schedule to a fifth of a percent, so the label is the rounded one. Not "these disagree" — "the schedule is right and the label is loose."

Then the part we did not expect. To measure anything you need to know how many points make a foot, and a drawing tells you twice. The sheet prints SCALE: 1/4" = 1'-0", which is 18 points per foot. Its own geometry measures 11.76. The sheet was plotted at about 65% of the scale it claims.

Trusting the printed note would have overstated every dimension by 53%, and every number would have looked entirely reasonable. So the scale note is now the last source it will use, it caps confidence when it is the only one available, and when two sources disagree by more than a couple of percent the tool says so instead of quietly picking one.

The bug that taught us to check: our first attempt took the largest single shape on the sheet as the building outline. An elevation is drawn as several stacked rectangles — two storeys and a grade band — so it grabbed one storey and came out exactly twice wrong vertically while being perfectly right horizontally. It was caught only because the tool compares the horizontal and vertical scales against each other and refuses to publish when they disagree. A measurement you cannot cross-check is a guess with a decimal point.

Somebody asked a specific question about a specific set: a car wash, 37 sheets, plotted out of AutoCAD. Is the motor control center big enough?

The first thing the tool found is that the set never says how big it is. The one-line diagram reads MOTOR CONTROL CENTER — 480V, 3~ 4W — TOTAL CONNECTED LOAD: 431.7A — BREAKER SIZE TBD MLO (VERIFY T.C.L.), and the equipment list gives the MCC line item an ampere value of literally TBD. The drawing is asking the same question back at you.

None of that was reachable as text. All three sheets that mattered — the equipment lists and the one-line — are drawn as outlined vector, so a text search returns the disclaimer and nothing else. The tables had to be read as pictures, then every horsepower checked back against the code tables rather than the amps printed on the sheet, because Article 430 sizes to the table value and not to the nameplate.

Recomputed from the branches actually on the one-line, the connected load came to 430.8 A against the sheet's stated 431.7. A fifth of a percent apart. That agreement is the only reason we trust the rest of it.

And then the mistake. We had a total from the equipment list — 544 A — and a total from the one-line — 431.7 A — and we nearly reported the larger one. They are both correct. They describe different sets of loads: the equipment list covers the whole package, including a vacuum panel on its own drive, a 120-volt air dryer and the water treatment skid, none of which are fed from the MCC. Sizing off the equipment list oversizes the gear by about 110 amps.

So the tool no longer publishes a rating when the two sources disagree by more than a few percent. It reports the mismatch, names both numbers, and says which loads need reconciling. It refuses to answer rather than answer plausibly, which is the only version of this we would put in front of somebody who does it for a living.

It also found a motor carrying three different currents in three places — 1.65 A on the equipment list, 3.4 A on the one-line, 2.1 A in the code table for its horsepower. The 3.4 belongs to the two-horsepower motor drawn beside it. Small, and exactly the sort of thing that gets an entire submittal a second look.

Everything before this had been run against documents we made up. That is a comfortable way to test software and it proves almost nothing. So we pulled a real one: a public project manual for a school-district HVAC replacement, 333 pages, issued for bid.

It read the full mechanical division — all eight Division 23 technical sections, 49 pages of specification — in 75 seconds, entirely on the box. Then we took every passage it had quoted and went back to the source to see whether those words were actually there.

Twelve of twelve, verbatim. Every paragraph number it cited was real. That is the load-bearing claim of this whole product and it held on first contact with a document we did not write.

It surfaced six candidate conflicts and then argued itself out of all six. Every one is exactly the false positive a keyword matcher would have dropped on a coordinator's desk: dielectric unions rated 250 psig against flexible connectors rated 125 — different components, not a contradiction. Duct insulation at 1-1/2" concealed against 1" exposed — separated by scope. Duct liner against external wrap — different assemblies entirely.

The sixth is the one we keep thinking about. Two sections both call for 5,000 psi grout, and one of them writes it psig. It declined to call that a conflict and flagged it as the spec writer's typo instead, on the grounds that compressive strength is not a gauge pressure.

Nothing was found because nothing was there. But six times it declined to waste our afternoon, and once it caught something the spec writer missed. Precision is the thing every checking tool we have been sold got wrong — an unread clash report is just a different way of missing the conflict.

What we can say is that it reads a real bid package and cites it honestly. Whether it reliably catches a conflict is a different claim, and it needs a set where one actually exists before we will make it.

The schema that couldn't say "fine." The first attempt failed, and the reason was ours, not the model's. We had asked for a list of findings with no way to record a negative. So when it correctly concluded that a pair of clauses did not conflict, it had nowhere to put that conclusion except the findings list — and filed it as a finding anyway. Then it looped, re-titling the same entry "(Final Check)", "(Final Final Check)", until it hit the token ceiling. Its judgment was right the entire time; our output contract was wrong. Any schema like this needs a legitimate place to say we checked this and it's fine.

A false alarm worth recording. Our verification script initially reported one quote as a hallucination. It wasn't. The PDF text layer had split 10-feet across a line break, and our whitespace handling turned that into a mismatch. Strip everything before you accuse a model of making something up.

The assistant running on the box had developed a stall. Long pauses, no output, then eventually a normal answer. We diagnosed it three times and were wrong three times — we blamed the time it takes to read a long prompt, then the memory store, then the hardware.

The actual cause: one particular model simply never stopped generating when handed a certain class of prompt. Individual calls were running twenty-nine to thirty-nine minutes before anything gave up. It was not slow. It was not finishing.

We are writing this one down because the first three explanations were all plausible, all consistent with the symptom, and all wrong. The thing that broke the tie was timing the individual calls instead of theorizing about the pipeline.

There is a version of this product that ships with the second explanation baked in as a permanent workaround, and nobody ever finds out the real cause. Measuring is cheaper than being clever.

We ran the thing end to end for the first time: hand it a PDF of building inputs, have it pull out the envelope details, calculate a heating load. Ninety-two seconds, start to finish. Every figure it produced was arithmetically correct.

And the method was wrong. It used gross wall area instead of net — it never subtracted the window openings, so it counted that glass twice, once as wall and once as window. On the room we tested it was a small error. On an elevation that is mostly glazing, it would not be.

This is the reason the calculation engine is built the way it is: the model reads the document, and vetted code does the math. A language model is superb at finding "R-21" buried in a general note and useless as an authority on which areas belong in which term. Those are different jobs and they stay separate.

The same shape of mistake shows up more than once in this project — a model reasoning correctly and then being let down by the structure it was asked to answer in. Each time the fix was in our design, not in the model.

We had been treating the box as full. It wasn't. Every limit we ran into turned out to be an assumption in software rather than anything physical about the machine.

There were three walls and none of them were made of silicon. The first was a setting — the split between processor and graphics memory was configured in a way that fought the software instead of feeding it. The second was a bug in the runtime, not the machine: a different engine on identical hardware immediately saw 95 GB where the first saw 59, and two kernel parameters took that to 120. The third wasn't a memory limit at all — garbled output we had been blaming on the model was a formatting bug in how requests were being packaged. The model had been fine the whole time.

Not all of the speed belongs to those three walls, and it would be dishonest to imply it does. The memory headroom let us switch to a mixture-of-experts model rather than a dense one, and that architecture is simply faster on this hardware. The walls are what made the switch possible; the switch is what you see in the second row.

The practical result is that one model now handles text, tools and images together. Point it at a scanned drawing and it reads the drawing. That used to take two models competing for the same memory.

The ceiling moved because we could get underneath it. That is an argument for measuring rather than believing the first number you are given — not an argument about who owns the metal.

The appliance is a single Ryzen AI MAX+ 395 with 128 GB of unified memory — one pool, shared between the processor and the graphics engine, with the division between them set in firmware. Get that division wrong and you have bought memory you cannot use.

Left on its default setting, the firmware carved out 64 GB for graphics and left 62 GB as system memory, of which the runtime could address roughly 58.8 GB. Models loaded, the accelerator stack came up, and answers came back locally. It worked.

It also looked, at that point, like the hard limit of the hardware. It was not. Every ceiling we ran into on this machine turned out to be software.