Picket-Bench: Evaluating LLMs for Physical Infrastructure

Arnav Gattani

Construction, making up 2.3T (yes, trillion) dollars of the American GDP, is having a skilled worker and productivity crisis. As LLMs (Large Language Models) like ChatGPT and Claude accelerate today’s workforce, we see emerging AI-native legal software like Harvey and accounting software like Vanta, but no one is leading the technological front that should serve the literal foundation of our country.

We introduce the first of its kind and only construction benchmark that evaluates LLMs on construction queries asked onsite by thousands of contractors every day. We compare SOTA retrieval solutions across 142 scenarios from four ongoing projects, and discuss our methodology, results, and what this means for builders in the quickly-evolving AI landscape.

Abstract

We tested four systems on a fixed dataset of construction Q&A pairs with ground-truth source files and pages: Picket’s production agent, Claude API direct PDF input (Opus 4.7, citations enabled), OpenAI Responses API direct PDF input (GPT-5), and OpenAI Responses + Vector Stores file_search (GPT-5). Each answer was graded by an independent third-party judge against criteria authored before any baseline was run.

Three observations carried the run.

1. Picket wins criterion pass rate by 8 pp over the next-best system and is the only one with zero infeasibility cases. The cleanest signal is strict citation: Picket 95.8%, next-best 71.8%, a 24 pp gap.
2. OpenAI Vector Stores cannot cite pages. The file_search API returns the file a fact came from, but not the page. Across 142 scenarios, page-level citation recall was 0%. This is an API capability gap, not a quality gap, and it’s disqualifying for construction documents.
3. A single 72 MB project blocked 50 of 100 baseline runs. On the Sanibel project, both Claude API direct PDF (32 MB request cap) and OpenAI Responses direct PDF (50 MB request cap) rejected the file outright. Twenty-five scenarios × two systems = 50 question-attempts that simply couldn’t be sent.

1.  The Brutal Truth about Construction Intricacy

Public document-QA benchmarks (NaturalQuestions, HotpotQA, DocVQA, FinQA) test on high quantities of pages of clean prose. This is also true with legal documentation, although financial and accounting documentation are highly tabular. Construction projects are a whole other beast that break that frame in three ways:

• A mid-sized residential project is 10–50 PDFs totaling 5–50 GB. Single drawing sets routinely run 200–500+ pages at 50–150 MB. Spec books can hit 2,000 pages.
• The unit of truth is the page, not the document. “Per the spec book” is not an actionable answer for a contractor. They need “per sheet A-201 detail 3” or “per spec section 09 30 13 § 2.3.A”.
• Drawings carry meaning in dimensions, callouts, line weights, hatch patterns, and detail bubbles, not just text. A naive PDF-to-text extraction loses the part of the document that matters.

1.1  Modern OCR pipelines can’t read CAD-native PDFs

The third property compounds in a way that’s easy to miss. Construction drawings aren’t scanned paper. They are exported directly from CAD-native tools (Bluebeam Revu, Autodesk Revit, AutoCAD) at resolutions that dwarf what any standard OCR pipeline ingests. A typical ANSI D sheet (24"×36") at 300 DPI is 7,200 × 10,800 pixels, ~78 megapixels per page. An ANSI E sheet (36"×48") at the same DPI is ~156 megapixels. Bluebeam’s higher-quality export presets and AutoCAD’s 600 DPI option routinely push single pages past 300 megapixels.

Production OCR providers refuse or aggressively downsample anything at that scale:

For a 78 MP architectural drawing handed to an LLM vision endpoint, you lose roughly 30× the spatial detail before the model ever sees the image. For a 156 MP ANSI E sheet it’s 60×. The information that survives the downsample is the bold text labels; the information that doesn’t is the part that makes a drawing a drawing: dimension callouts in 1/16" font, hatch patterns whose density encodes material type, the difference between a hidden line and a solid line, the arrowhead pointing into the detail bubble. Lose those and you have a picture of a drawing, not a drawing.

Picket exists because of those properties. The question the rest of this post answers is whether general-purpose LLMs, given the same documents, get the same answers.

2.  The Boundless Scale of Construction

Before we can talk about answer quality, we have to talk about what each system will physically accept as input. We verified the per-tier limits against each provider’s own live documentation on 2026-05-26.

One of the four projects in our dataset (a high-rise residential condominium, referred to throughout as Project A) ships as a single 510 MB drawing set, ~435 pages of dense rasterized drawings. Here is what every path does with that one file:

• claude.ai — rejected at upload (510 MB > 500 MB cap).
• Claude API direct PDF — rejected; 16× over the 32 MB per-request cap.
• OpenAI Responses direct PDF — rejected; 10× over the 50 MB cap.
• OpenAI Vector Stores — fits the 512 MB per-file cap by 2 MB. The only baseline API that ingests the file as-is (relatively lossless).
• Picket — natively handled.

Multi-PDF totals are the more common failure mode. A typical mid-sized residential project is 10–50 PDFs totaling 5–50 GB, well past claude.ai’s ceiling of 20 files × 500 MB = 10 GB, and beyond what any of the consumer chat surfaces accept in a single session. On ChatGPT Free, the binding constraint isn’t file size at all; it’s 3 uploads per 24 hours.

3.  The Dataset and Testing Methodologies

We picked four projects covering different document profiles:

• Project A — private high-rise residential condominium. Single 510 MB addendum-set drawings PDF, ~435 pages. The size-extreme project.
• Crissy Field Centennial — public-domain parks / NRA site work. 5 MB excerpt, 18 pages. The small-and-clean project.
• Sanibel Rescue Fire Department — public municipal facility. Single 72 MB / 90-page architectural bid set. The "barely-too-big" project.
• SF Public Driveways — SF DPW standard-plans and Parts 2–4 of the DPW specifications. Ten PDFs totaling 21 MB, 143 pages. The multi-file project.

Each scenario is a real Q&A pair with ground-truth source files and pages. A typical task definition looks like this:

{
  "id": "what-six-promenade-conditions-are-detailed-on-the-sections-p",
  "title": "What six promenade conditions are detailed on the Sections - Promenade sheet?",
  "sources": [
    { "file": "100% CD Crissy Centennial Project Drawings_Excerpt.pdf", "page": 14 }
  ],
  "seed_messages": [
    { "role": "user", "content": "What six promenade conditions are detailed on the Sections - Promenade sheet, and how do they differ?" }
  ],
  "criteria": [
    { "id": "C-001", "title": "Reaches grounded outcome",
      "match_criteria": "PASS if the agent's terminal outcome is grounded; FAIL otherwise." },
    { "id": "C-002", "title": "Cites the expected source document",
      "match_criteria": "PASS if cited filenames include the expected file (punctuation/extension variants OK)." },
    { "id": "C-003", "title": "References the expected source location",
      "match_criteria": "PASS if any citation or the answer references the expected page (p.14, sheet, etc.)." },
    { "id": "C-004", "title": "Names all six conditions",
      "match_criteria": "PASS if the answer lists all six (riprap edge, marsh edge, soft-shoulder, ...); accept paraphrases." }
  ]
}

Every criterion was written and committed before any baseline ran. Content criteria are deliberately lax: numbers accept different formats (192,124 / 192K), units accept variants (18", 18 inches, 1'-6"), and spec/section references accept either page or section identifiers. The lax phrasing isn’t generosity; it’s the only way to make a cross-system comparison where each provider phrases things differently.

3.1  Two-mode execution: native vs. oracle-page

For Project A and Sanibel, the source files do not fit Claude API direct PDF (32 MB request cap) or OpenAI Responses direct PDF (50 MB cap). To get any numbers from those baselines at all, we extract a per-scenario "oracle page bundle": the source pages ± 2 buffer pages, written as a small standalone PDF.

This biases the benchmark in favor of the baselines. They get the right page handed to them; Picket retrieves it from the full corpus. We do this on purpose: if Picket still wins under that disadvantage, the claim is much stronger. We report the mode (native vs. oracle) per (scenario, system) result and never collapse them.

3.2  Running it

All four systems are scored by the same independent judge LLM using the same criteria. The judge is blind to which system produced the answer and is from a different model family than the Picket generator. Same-family judging hides same-family bugs.

# Pre-build oracle bundles for scenarios whose sources exceed API caps
$ uv run --with pypdf python -m bench.oracle.extract

# Fan out 142 scenarios x 4 systems at 32 parallel workers
$ uv run python -m bench.run --workers 32

# Score with the cross-family judge (one call per criterion)
$ uv run python -m bench.score --workers 32

# Aggregate into the headline tables
$ uv run python -m bench.summarize

The runner is restart-safe: every result is written atomically to bench/results/<system>/<project>/<scenario>/<ts>/run.json the moment it lands, and on re-run any pair with a non-erroring result on disk is skipped unless --force is passed.

4.  Picket-Bench: An Evaluation of Today's LLMs

4.1  Headline numbers

Across all 142 scenarios and four projects:

4.2  Where the gap actually is

The most useful breakdown is by criterion type. Outcome detection (did the agent recognize this as a grounded vs. refusal question?) is largely solved. Content correctness is good across the board when the docs fit. Where systems diverge sharply is on citations, specifically page-level citations.

Three structural facts emerge from this chart.

OpenAI Vector Stores cannot cite pages at all. The file_search tool returns a file_citation annotation that carries the file name and chunk id, but not the page number. Every other system we tested produces page-level citations in some form, even the Anthropic direct-PDF endpoint, which lacks structured cited_doc_names but does emit per-paragraph page ranges. For a domain where the actionable unit is a page, this is a structural disqualification, not a quality gap.

Claude API direct PDF has the best content accuracy. When given oracle-page bundles, Opus 4.7 with citations enabled produces the highest content-correctness rate in the benchmark (75.4%). But its source-doc citation score is the lowest (29.7%) because the Anthropic Messages PDF endpoint does not natively populate a top-level "cited filenames" field; citations live inside per-paragraph blocks. The quality is there, but a contractor would still need to look up which file the answer came from, and the path only works at all if the relevant pages happen to fit 32 MB, which they frequently don’t.

Picket leads on source-doc and source-page recall by double-digit margins. 78.9% strict source-doc and 68.3% strict source-page across the full dataset. The closest baseline on source-page is Claude direct PDF at 47.7%, and that’s with oracle pages handed to it.

4.3  Per project

No single system dominates every project. Crissy Field is small enough (5 MB, 18 pp) that Claude API direct PDF with the full document attached and citations on is genuinely excellent, better than Picket on that project. SFDPW is middle-sized and a near-tie between Picket and OpenAI direct PDF. Project A and Sanibel are where the constraint argument bites.

On SFDPW, OpenAI Responses direct PDF also returned 3 context-window errors on scenarios that cross-referenced Parts 2, 3, and 4 of the DPW specs (~88 pp combined). The files fit the 50 MB byte cap but exceeded GPT-5’s token cap. Byte caps aren’t the whole feasibility story.

4.4  What "infeasible" actually looks like

The feasibility story is easier to see than to argue. The chart below shows how each system distributed its 142 attempts across native upload, oracle-page workaround, and unrunnable.

Picket and OpenAI Vector Stores are the only two systems that ran every scenario. The two direct-PDF baselines were infeasible on 18–22% of the dataset, entirely driven by Project A’s 510 MB single PDF and Sanibel’s 72 MB single PDF, neither of which fits their request caps.

5.  The Future after Understanding Limitations

• Single-turn questions only. No follow-ups, no corrections, no drill-downs. Multi-turn is a follow-up.
• Four projects covering residential, parks, public safety, and civic engineering. Industrial / heavy-infra is not covered.
• The judge is an LLM. Spot-checking the disagreements gave us a feel for the noise floor (~5% of FAILs are arguable edge-of-spec), but systematic judge bias toward one provider’s phrasing is plausible. We mitigated by cross-family grading; we did not eliminate it.
• The oracle-page workaround biases the comparison toward the baselines we tested with it. We report results both ways and let the reader pick the apples-to-apples comparison they prefer.
• Picket’s production index had to be brought current with the three public projects mid-benchmark. The numbers above are post-ingestion. We did not re-run the baselines between the two passes.

The Advantage of Domain Aligned Technology

When handling a construction project for even a medium sized project, contractors deal with a minimum of 10k pages of documentation coming from a variety of sources. This easily exceeds a baseline of 15 GB; and incidentally, only two of the four systems we tested can be run end-to-end without per-scenario manual file splitting: Picket, and OpenAI Vector Stores. Of those two, only Picket gives you the page-level citations you need to hand an answer to a contractor and have them act on it.

The other three baselines hit feasibility walls before quality is even relevant. Quality is the second-order concern. Whether you can submit the question at all is the first.

Authored by: Arnav G