A1. Types of job
QuestionWhich types of job did you find and how many of each?
I found 10 types of job. Each type can include supporting lines, such as waste removal, surface reinstatement or traffic cones, which are part of the job rather than separate jobs.
Across all 999 jobs, the types split like this:
| Type of job | Jobs |
|---|---|
| Manhole cover replacement | 416 |
| Channel drain repair | 155 |
| Gully repair | 112 |
| Dig and pipe work | 72 |
| Tanker: pump out and dispose of waste | 64 |
| Drainage crew: jetting, CCTV, liner | 63 |
| Civils visit, labour only | 44 |
| Small civils repair (gully pot, chamber, bespoke cover) | 40 |
| Single engineer: plumbing, pumps | 21 |
| Manhole cover re-set (rebed) | 12 |
| Total | 999 |
For the 150 labelled jobs the type comes from the estimator's quote; for the other 849 from my system.
A2. Mislabelled classes
QuestionOur Engineers label the jobs, are there classes of jobs you can identify that are mislabeled?
Outright wrong labels are rare: I found 3. But about a quarter of the jobs labelled "Other" (73 of 267) clearly belong to a real class of work, so "Other" hides a large share of ordinary cover, channel and gully jobs.
This is what the 73 "Other" jobs really are:
| Real work | Jobs |
|---|---|
| Manhole cover replacement | 36 |
| Channel drain repair | 18 |
| Gully repair | 10 |
| Manhole cover re-set | 9 |
| Total | 73 |
The engineers knew what they were looking at: in 61 of the 73 the text names the object plainly ("broken manhole cover", "new gully lid", "missing aco grate").
Part of the cause is the form itself. Manhole, Gully and Channel types exist only under the Civils category. 30 of the 73 jobs were raised on a Drainage form, where "Other" was the only choice; 8 of those come from engineers who only ever use Drainage forms. The other 43 were raised on a Civils form, where the right type was available.
The 3 outright wrong labels all say "Defective Manhole" for something that is not a drainage manhole:
- J0934 is a gully: "2nd gully in from the gate… original grate is missing".
- J0712 is a water stopcock: "stopcock to isolate the water supply".
- J0628 is a water meter: "not a foul or storm drain. water meter".
Counts use the estimator's quotes for the 150 labelled jobs and my system for the other 849.
A3. Attributes that drive the price and the quote
QuestionThere are key "attributes" of jobs that drive a different price or a different quote, can you identify them and reason about them?
Within each type of job (A1), three groups of attributes change the quote: the part being fixed, extra lines added on top, and multipliers that scale everything.
1. The part being fixed
For a cover, three attributes pick the line from the rate card:
| Cover attribute | Values | Price, low → high |
|---|---|---|
| Size | 600x450, 600x600, 675x675, 750+ | £160 → £520 |
| Load class | light (A15/B125), heavy (C250/D400) | £185 → £330 (600x600) |
| Style | standard, chequer plate, recessed | £330 → £410 |
Gullies, channels and pipe work have fewer choices, because the rate card offers one type of grate and one type of channel:
| Type of job | Attribute | Values | Price, low → high |
|---|---|---|---|
| Gully | Pot broken | grate only, grate + new pot | £210 → £395 |
| Channel | Sump replaced | no, yes | £0 → £240 |
| Dig and pipe | How the pipe is fixed | patch liner, new pipe | £290 (liner) → £38 per 3 m of pipe + £22 per fitting + excavator £210 a day |
Tanker, drainage crew, visit, small civils repair and plumbing have no part to choose; they are priced by crew time and waste.
2. Extra lines added on top
These depend on where the job is and what it leaves behind:
| Attribute | Values | Price, low → high |
|---|---|---|
| Surface | block paving, concrete, asphalt | £0 → £660 (asphalt, median) |
| Waste type | general, contaminated or grease | £0 → £852 (median) |
| Job size | under 3 days, 3+ days or 8+ m of channel | £95 (general waste) → £320 (skip) |
3. Multipliers
These scale the lines already chosen:
| Attribute | Values | Price, low → high |
|---|---|---|
| Number of covers | 1, 2–3, 4+ | £1,870 → £8,466 (median quote) |
| Channel length | metres | £205 per metre |
| Out of hours | day, night or weekend | +36–39% on every crew line |
B. A classifier that labels a job only when it is very sure
QuestionWe would like a classifier with few false positives: one that labels a job only when it is very sure. How would you build it? Show a proof of concept, and tell us how many jobs it labels and how often it is right.
I built an MVP classifier that works on these principles:
- Code prepares the input. It turns each job's forms into clean text: empty forms and exact copies are dropped, every filled field and the free text are kept. No decisions are made at this stage.
- An LLM does the reading and the judgement. Claude Sonnet gets the text with a long prompt: 61 quoting rules, the rate card, three worked examples and the rules for confidence. It returns the type of job, the quote lines with quantities, and how sure it is. Code then checks every line against the rate card and adds up the prices.
- Three levels of confidence, set in the rules before any results were seen:
- Green: the rules give one answer from what the form states.
- Yellow: the form leaves a gap and the model makes an educated guess, for example no cover size given, so a standard 600x600 is assumed.
- Red: the form contradicts itself or doesn't say what work is needed, or a channel run is over 10 m, where the estimator's practice is unclear.
- It labels only green jobs and leaves the rest to the estimator: 298 of the 849 unlabelled jobs (35%).
On the green jobs it labelled, the classifier was right this often:
| Measure | Hold-out (17 green of 50) | Training (40 green of 97) |
|---|---|---|
| Manhole job: yes / no | 17 of 17 | 40 of 40 |
| Type of job | 17 of 17 | 38 of 40 |
| Exact label, including the cover kind | 14 of 17 | 36 of 40 |
| Main quote lines, F1 | 0.83 | 0.92 |
| Other quote lines, F1 | 0.79 | 0.85 |
The 3 hold-out misses are all the wrong cover kind on a correctly typed manhole job. 3 of the 100 training jobs are the worked examples in the prompt and are not scored.
Proof of concept: predictions.json (all 849 jobs, field confidence), quote_run.py, quote_rules.md.
Also tested: Haiku, Sonnet and Opus; with and without site photos (see D2).
What could improve it
Several ways to improve quality were not tested in this MVP:
- Other models, especially for reading the photos.
- Splitting the single prompt into a workflow of small steps, with a large model producing the final result once the small steps finish.
- Clearer rules on diagnosis versus work, night versus day, and how many metres of channel to replace. These rules can't be learned from the data alone; they are worth learning from the estimators and adding to the system.
C. Predicting quotes and their line items
QuestionAre you able to predict quotes and their line items? How would you measure accuracy?
Yes, I can predict the quotes: my system writes a full quote for every job, with rate-card lines and quantities (predictions.json). However, some limits in the data leave a relatively large gap to the prices in the real quotes.
Some of the gap comes from inconsistencies in the data. For example, one real quote (J0222) charges 150 loads of waste removal for half a day of work on one cover: £16,039 instead of about £1,884. This one quote is 19% of the system's money error on the training dataset (example 1).
Some comes from rules the forms don't reveal. For example, night work adds 36–39% to every crew line, but it is hard to tell from the form when it applies: of the 12 night quotes in the training dataset, only 3 forms mention out-of-hours work, while 3 other forms do mention it and were priced by day.
How I measure accuracy
A quote is compared with the estimator's real quote in four steps, from coarse to fine:
| Step | What is compared | Measure |
|---|---|---|
| 1. Manhole job | yes or no | share right |
| 2. Main lines | the part: cover, gully, channel, pipe lines | F1 on lines |
| 3. Other lines | labour, materials, waste, services, hire | F1 on lines |
| 4. Money | the quote total | mean and median absolute error (£) |
F1 on lines: precision is the share of the system's lines that are in the real quote, recall is the share of real lines it found. Quantities count in step 4, through the money.
Results on the jobs the system is sure about (green)
These are the jobs the classifier in B labels:
| Step | Hold-out (17) | Training (40) |
|---|---|---|
| 1. Manhole job right | 17 of 17 | 40 of 40 |
| 2. Main lines, F1 | 0.83 | 0.92 |
| 3. Other lines, F1 | 0.79 | 0.85 |
| 4. Mean absolute error | £571 | £440 |
| 4. Median absolute error | £306 | £246 |
Results on all jobs
Including the yellow and red jobs, the gap grows:
| Step | Hold-out (50) | Training (97) |
|---|---|---|
| 1. Manhole job right | 49 of 50 | 95 of 97 |
| 2. Main lines, F1 | 0.75 | 0.78 |
| 3. Other lines, F1 | 0.69 | 0.78 |
| 4. Mean absolute error | £1,379 | £764 |
| 4. Median absolute error | £744 | £351 |
A few large errors dominate: on the hold-out the 3 largest make up 46% of all money error. The hold-out quotes are also larger on average (£3,662 vs £2,767).
Where it falls short
- Inconsistencies in the engineer's forms. In 39 of the 849 unlabelled jobs the form contradicts itself (21), has impossible values (10) or doesn't say what work is needed (8); the system marks them red. 9 more jobs repeat another job's form.
- Decisions the form doesn't record: night work, whether a diagnostic visit or the repair itself is quoted, and how much of a long channel run is replaced.
- Inconsistencies in the real quotes, shown in full in Data inconsistencies.
- Two weak rules of my own. The rule for channel runs over 10 m rests on 2 training jobs, and the hold-out has a counter-example (J0421, £10,749 of error). One rule sentence about re-sets contradicts the pricing rows (10 jobs affected, all yellow).
D1. Summary
QuestionSummarise your work in short bullet points.
- A1. 10 types of job; the largest is manhole cover replacement, 416 of 999.
- A2. Outright wrong labels are rare (3), but a quarter of "Other" jobs (73 of 267) are really cover, channel or gully work. 30 of them were raised on Drainage forms, whose menu has no such type.
- A3. The price is driven by the part (cover size, load class, style), extra lines (surface, waste) and multipliers (number of covers, channel metres, night work +36–39%). Priority doesn't change the price.
- B. Claude Sonnet reads the forms and applies 61 written rules; it labels a job only when the rules give one answer from the form. It labels 298 of 849 jobs (35%); on the hold-out it labelled 17 of 50 and got the type right in all 17.
- C. A full quote for every job. On the confident (green) hold-out jobs: main lines F1 0.83, median price error £306. On all hold-out jobs: 0.75 and £744.
- Data inconsistencies: a quote with 150 loads of waste for half a day of work, light covers quoted where the form asks for a heavy one, 39 contradictory or impossible forms, 9 repeated forms.
- Limits: night work, diagnosis versus repair, and the length of channel to replace are not recorded in the forms.
- Cost: about $20 of API calls.
D2. Choices and their effect
QuestionWhich choices did you make, and what difference did they make?
- Code where no judgement is needed, the LLM where it is. I started with code that read the form fields: deterministic and consistent. But key facts often sit in free text (the size in the description, the surface in "Part required"), so the final system leaves the reading and the rules to the LLM. Code turns the forms into text, checks every line against the rate card and adds up the prices. On the training dataset this moved main lines F1 from 0.51 to 0.78, other lines from 0.61 to 0.78, and the mean price error from £1,904 to £764.
- Haiku, Sonnet and Opus. The three models were tested to see the trade-off between price and performance. On the same prompt in an earlier step they differed by 1–3 jobs out of 100, while the cost differed up to 27 times. The final system uses Sonnet, at half the price of Opus. Sonnet and Opus were not compared on the final rules.
- Clean data first. I excluded the four quotes with clear inconsistencies from the training dataset when writing the rules; they still count when the system is scored. Early versions were built on a "golden" subset of 36 training jobs with clean, unambiguous forms. A rule was kept only if the estimator follows it in most similar quotes, so one-off exceptions did not become rules. On the hold-out, never seen while building, main lines held up (F1 0.75 against 0.78 on the training dataset); other lines dropped more (0.69 against 0.78).
- Photos as input. Adding the site photos fixed 2 training jobs and broke 3, for about 50% more cost. The final system doesn't use images, which saves tokens without losing accuracy.
D3. How the AI's output was checked
QuestionHow did you check the AI's output? Where is your approach strong, and where is it weak?
- Objective comparison on the hold-out. The main check is a deterministic comparison with the estimator's real quotes on the hold-out dataset: 50 jobs never seen while building, scored once with the system frozen. This is the closest to a scientific test of whether the system works.
- Asking the AI for examples. While building, I checked the AI's reasoning by asking it for concrete examples, and found dozens of judgement mistakes. For example, two labelled quotes give a light cover where the engineer's form asks for a heavy one (example 2, example 3). I saw this as a serious inconsistency and an important signal; the AI treated it as minor, because only a few quotes had it.
- The weak part. With more time, I would spend it checking the data visually, to spot nuances the AI misses: AI is weaker on real-world scenes. Both I and the AI lack domain knowledge in drainage and civils work, and that is a limit of this approach.
D4. Files
QuestionI'll review several submissions, so please give me one starting page (an index.html or similar) that links to everything you want me to see.
This page is the starting point. Everything it refers to:
| File | What it is |
|---|---|
| predictions.json | answers for the 849 unlabelled jobs: manhole yes/no, type, quote lines with quantities, confidence, what the estimator should check |
| code/quote_run.py | sends each job's forms to the model with the rules and the rate card |
| code/build_predictions.py | turns the model's answers into predictions.json |
| code/holdout_eval.py | scores the frozen system on the hold-out |
| rules/quote_rules.md | the 61 quoting rules |
| rules/examples.md | the three worked examples in the prompt |
| Data inconsistencies | three examples shown in full: form, quote and photos |
| code.zip | everything above, to run locally next to the job pack |
Data inconsistencies
A few real quotes don't agree with their own forms or with the rest of the data. They lower the system's scores: with clean data the scores would be higher.
1. J0222: 150 loads of waste for half a day of work on one cover
The quote prices one cover fitted by a two-person crew in half a day, yet charges 150 loads of waste removal: £14,250, eight times the rest of the quote (£1,789). Every other quote with this line has 1 load; the largest jobs use one or two skips instead. This one quote is 19% of the system's money error on the training dataset.
Engineer's form
| Category / type | Civils / Defective Manhole |
| Request / priority | Quoted Works / P2 Moderate (1 week) |
| Description of exact location | entrance to site on right hand side |
| Any potential access issues | no |
| Can chamber opening be measured | Yes |
| Chamber opening length (mm) | 860mm |
| Chamber opening width (mm) | 700mm |
| Manhole frame depth (mm) | 100mm |
| Manufacturer grade | D400 - HGVs |
| Is heavier grade required | No |
| Surface type | Concrete |
| Part required | civil team to attend site to try and break free chamber 2nd stage interceptor lid hydraulic lifters breaker |
Estimator's quote: £16,039
| Code | Description | Qty | Unit price | Total |
|---|---|---|---|---|
| LAB-CIV-H | Civils gang (2 operatives + van), in hours, half day | 1 half day | £640 | £640 |
| SRV-UTIL | Buried utility plans (search before breaking ground) | 1 each | £85 | £85 |
| WST-GEN | Waste removal from site (spoil, broken frames, packaging) | 150 load | £95 | £14,250 |
| MAT-MORT | Rapid-set bedding mortar, tub | 8 tub | £68 | £544 |
| CVR-750-H | Cover & frame, solid top, 750 x 750 or larger, heavy duty (D400) | 1 each | £520 | £520 |
Photos
2. J0189: a light cover where the form asks for a heavy one
The cover is in a car park, the grade is D400, and the engineer recommends "a D400 rated lid and frame". The quote gives a light-duty cover (A15/B125).
Engineer's form
| Category / type | Civils / Defective Manhole |
| Request / priority | Reactive / P0 Danger (same day) |
| Description of exact location | car park (second chamber in rear [place] area ) |
| Any potential access issues | no |
| Can chamber opening be measured | Yes |
| Chamber opening length (mm) | 480mm circle |
| Chamber opening width (mm) | 480mm |
| Manhole frame depth (mm) | NA |
| Manufacturer grade | D400 - HGVs |
| Is heavier grade required | No |
| Surface type | Tarmac |
| Additional notes | circle cast [place] manhole chamber lid completely shattered in car park on site [place] have currently used a circle plastic chamber lid from rear [place] to cover exposed chamber in car park plastic lid is to be refitted to correct chamber in rear [place] after new cover and frame fitted on car park car park chamber left with barriers around but needs doing assap due to blocking off Tennant car spaces please note when arrived on site only rememisense that the original chamber lid was cast [place] it is suspected the original lid was not the correct rate and has shattered from cars driving over it due to being in a car park it is recommended to replace this with a D400 rated lid and frame when repair works are carried out please collect cones and barriers and return to depot again it is vital to refit plastic chamber lid to the correct chamber in yhe rear [place] Issue Type NA |
| Part required | civil [place] support to sort out |
Estimator's quote: £1,818
| Code | Description | Qty | Unit price | Total |
|---|---|---|---|---|
| LAB-CIV-H | Civils gang (2 operatives + van), in hours, half day | 1 half day | £640 | £640 |
| SRV-UTIL | Buried utility plans (search before breaking ground) | 1 each | £85 | £85 |
| WST-GEN | Waste removal from site (spoil, broken frames, packaging) | 1 load | £95 | £95 |
| MAT-MORT | Rapid-set bedding mortar, tub | 2 tub | £68 | £136 |
| MAT-CONC | Rapid flowable concrete, 25 kg bag | 1 bag | £24 | £24 |
| MAT-ASPH | Cold-lay rapid asphalt repair kit | 2 kit | £260 | £520 |
| MAT-PRIM | Primer for asphalt repair kit | 1 kit | £38 | £38 |
| MAT-BAND | Thermoplastic overbanding tape (seals joint round frame) | 1 roll | £95 | £95 |
| CVR-600-L | Cover & frame, solid top, 600 x 600, light duty (A15/B125) | 1 each | £185 | £185 |
Photos
3. J0381: a light cover where the form asks for a heavier grade
The cover is by a loading bay, the grade is C250, and the form says "Is heavier grade required: Yes". The quote gives a light-duty cover (A15/B125).
Engineer's form
| Category / type | Civils / Defective Manhole |
| Request / priority | PPM / P2 Moderate (1 week) |
| Description of exact location | see highlighted plans for exact location manhole chamber not actually on site plans to the right side of [business] unit loading bay/doors |
| Can chamber opening be measured | No |
| Manhole cover length (mm) | x |
| Manhole cover width (mm) | x |
| Manufacturer grade | C250 - Light Goods Vehicle |
| Is heavier grade required | Yes |
| Surface type | Tarmac |
| Additional notes | I've lost my tape measure so unable to get precise measurements manhole chamber approx 450mm diameter Issue Type tarmac all crumbled away and foul water manhole covers is loose. |
| Part required | frame and cover replacement |
Estimator's quote: £1,887
| Code | Description | Qty | Unit price | Total |
|---|---|---|---|---|
| LAB-CIV-H | Civils gang (2 operatives + van), in hours, half day | 1 half day | £640 | £640 |
| SRV-UTIL | Buried utility plans (search before breaking ground) | 1 each | £85 | £85 |
| SUN-CARR | Carriage / delivery of materials or plant | 1 each | £45 | £45 |
| WST-GEN | Waste removal from site (spoil, broken frames, packaging) | 1 load | £95 | £95 |
| MAT-MORT | Rapid-set bedding mortar, tub | 2 tub | £68 | £136 |
| MAT-CONC | Rapid flowable concrete, 25 kg bag | 2 bag | £24 | £48 |
| MAT-ASPH | Cold-lay rapid asphalt repair kit | 2 kit | £260 | £520 |
| MAT-PRIM | Primer for asphalt repair kit | 1 kit | £38 | £38 |
| MAT-BAND | Thermoplastic overbanding tape (seals joint round frame) | 1 roll | £95 | £95 |
| CVR-600-L | Cover & frame, solid top, 600 x 600, light duty (A15/B125) | 1 each | £185 | £185 |
Photos
