How we benchmark QTO accuracy every night

Accuracy claims mean nothing without a method behind them. Here is ours.

A fixed ground-truth set. We maintain a benchmark suite of real BIM and PDF files, each with a known-correct BOQ — extracted manually or directly from the BIM model. The set spans architectural, structural, MEP, and landscape disciplines, plus anonymized consultant drawings.

Nightly runs against production. Every night a job submits these files through the live pipeline — the same one customers use — and compares the output BOQ to ground truth. Each case is scored 0–100 and classified Excellent / Good / Acceptable / Below. Any case that drops 10 or more points versus the committed baseline triggers a regression alert before it can reach a customer.

Where we stand today. The current baseline: 9 of 9 BIM benchmark cases score Excellent, with an average of 87.8/100. Our v9.4 architecture regression suite passes 18 of 18 checks. Format accuracy ranges from 85–92% on PDF up to 98–99% on Revit.

Transparency over hype. We publish what CAIS does well and where it still needs a human — waterproofing, complex MEP, and country-specific code compliance always warrant engineer review. If you find a case CAIS gets wrong, send us the file and the expected output; it goes straight into the regression set.

This is why the numbers on our homepage are specific, not round marketing figures.