A Labeling Instrument for Legal OCR Model Trainers

A custom labeling interface to train a legal OCR model, with paralegals as the labelers: every correction became labeled training data

My role

Freelance designer

Year: 2014
Industry: Legal, machine learning
Team: Sole designer, with one engineer on the retraining side

Shadowed and interviewed three paralegals at a branch office, and drew a journey map of their work with them.
Designed and built a custom labeling interface to train the legal OCR model: paralegals labeled the low-confidence spans the model flagged, choosing from a four-type error taxonomy.
Worked with our engineer, who built the part that fed each fix back in to retrain the model.

86%→93% how sure the model was on its doubtful spots, across 1,250 training documents; more confident, not proven more accurate
4 of 6 stress votes from three paralegals went to checking the typed text by hand
4 mistake types the paralegals worked out with our engineer, the first clues the model learned from

Summary

Before a trial, each side has to hand over its records, a step called discovery, and it buries legal teams under thousands of scanned pages. OCR software, short for optical character recognition, turns a picture of a page into typed text. On badly worn medical records it misreads enough that people have to check it by hand, and the paralegals had no way to tell which pages would need that work.

I designed and built a custom labeling interface to train a legal OCR model. The paralegals were the labelers: they corrected the spans the model was least sure of, and each correction became labeled training data. The model grew more confident. Then the project stalled, not over the model but over who owned it.

Who was involved

Paralegals Three at a branch office, turning worn medical records into text. They set the line for which spots to check.
Our engineer The machine-learning engineer who built the retraining side and asked for a short, fixed list of mistake types.

The correction tool in a browser: a scanned emergency record on the left with every doubtful passage outlined in red, and a panel on the right listing each of those passages as an editable field beside a mistake-type menu. Screenshot of the finished correction tool. The document is a filed court exhibit; personal details are blacked out. — **The page, with the doubtful words circled.** Only the spots the software was less than 86% sure of are outlined in red, and the rest of the page is left alone.

Context

Paralegals preparing badly worn medical records for typing up had no way to tell which pages would need hand work. Anything smudged or dirty was set aside and retyped from scratch, so a clean page and a ruined one took the same effort.

Corrections were kept as Word files holding the retyped text and nothing else: not where on the page it sat, not what kind of mistake it was. So the model that made the mistake never found out it had been corrected. The existing process was also required and ours was an experiment, so whatever came next had to fit inside it.

Research

I shadowed three paralegals while they loaded scanners and set up OCR runs. At the end of each day I interviewed them, and together we drew a rough journey map, a picture of each step of the work and how it felt, tagging each task with a feeling, a thought and the technology in use.

I gave each of them two votes to place on the most stressful moment, and asked them to say why. Four went to checking the text by hand and two to waiting for the scan. When we tried Tesseract, a free OCR program, against the scanner software they already had, the paralegals decided where the line for hand checking sat: anything the software was less than 86% sure of, they wanted to see.

Assumption map

Research first

Each dot is one belief, placed by how much it mattered and how much we actually knew. The top right corner is where being wrong would hurt most and we knew least, so that is where the research went first.

Held up
Partly held
Didn't hold
Never tested

ID	Assumption	Certainty	Importance	Status
A1	Checking the typed text by hand is the most stressful part of the paralegals' work.	Medium	Critical	ValidatedIn the stress vote, three paralegals placed four of their six votes on checking by hand and two on waiting for the scan.
A2	Paralegals only need to check the spots the OCR software is unsure about, not the whole page.	Medium	Critical	Partially validatedComparing Tesseract with their own scanner software, the paralegals set the line themselves at 86%, but the time saved was never formally measured.
A3	The corrected text alone is enough for the model to learn from.	Medium	High	Didn't holdThe first round of corrections showed the fixed text was not enough, so the paralegals and the engineer added four mistake types.
A4	Feeding the paralegals' fixes back into the model will make it read better.	Medium	Critical	Partially validatedIts confidence on the doubtful spots rose from 86% to 93% across 1,250 documents, but it was never tested on fresh pages, so it got more sure, not proven more correct.
A5	The browser's own zoom is good enough for reading fine print on a worn scan.	Medium	Medium	Didn't holdBrowser zoom pulled the outlines off the words they belonged to, so zoom was built into the tool.
A6	The new tool has to fit inside the paralegals' required workflow without changing it.	High	High	Not testedThe team treated this as a fixed rule from the start because it had no authority to change the existing process.
A7	Checking by hand costs about 12 doubtful spots per document at 30 seconds each, at $35 an hour.	Medium	Medium	Not testedThis was the designer's estimate from watching sessions, not a formal time study.
A8	Getting the model to work was the main job, and the question of who owned it could wait.	Medium	Critical	Didn't holdOnce it ran, nobody had the standing to say who controlled the corrections or approved retraining, and the project stalled there.

A journey map of the paralegals' workflow: the steps of preparing scanned records for OCR, what they did at each, and where they said the work went wrong. — **The journey map.** Drawn with the three paralegals; the vote put four of six on checking by hand, though the map shows seven dots from a counting slip in the session.

Design

The correction panel headed Text Extractions, its rows holding the OCR's raw readings such as Pg4of34, po, i and wi, each beside a mistake-type menu, with the edge of the scanned document showing to its left. Screenshot of the finished correction panel. The folding and the see-through background are described from the working tool; neither shows in a still. — **The panel is the notepad.** One row for each doubtful passage, each with the OCR reading to fix and a mistake-type menu.

Tesseract rates how sure it is about each bit of text, and anything under the 86% line is outlined in red right on the page. The idea came from print design, where a proof is marked up on the page itself rather than in a list somewhere else.

Watching the paralegals type, I saw them look down to a notepad of lines and keywords, back up to the screen, and down again. So a panel beside the page is that notepad moved onto the screen, one row for each doubtful spot, with a menu of mistake types. It folds away when it isn't wanted, and its see-through background never fully hides the page.

Iteration

Two things I first took for granted didn't hold. The first round of corrections showed that the fixed text alone wasn't enough for the model to learn from, so the paralegals worked out four mistake types with our engineer: Misread, Incomplete, Bad format and Other. A misread is the model's fault and a bad format is the page's, so keeping them apart lets it learn the right lesson from each.

The browser's own zoom pulled the outlines off the words they belonged to, so zooming became a control inside the tool, and the outlines grow with the page. Each fix is saved as one small record tagged with the number of its spot, which is enough for the retraining step to put it back on the exact spot it came from.

The correction panel with the mistake-type menu open on one row, showing its four options: Misread, Incomplete, Bad format and Other. The fields beside it hold the software's raw readings such as po, Complaint; and aiey. Screenshot of the finished correction panel. The four labels are the list as it shipped; no reviewer could add a fifth. — **Four words, and one of them is Other.** Other lets the list stay short and still be honest about what it does not cover.

The same record with the zoom slider pushed to its right end. The page is magnified to several times its default size and the red outline still sits exactly on the word Complaint, with the correction panel unchanged beside it. Screenshot of the finished correction tool at full zoom. The document is a filed court exhibit; personal details are blacked out. — **The zoom had to be ours.** At full zoom the red outline still sits on the word it belongs to.

The correction panel beside the browser's developer console. The console logs each correction as a small record with four parts: index, originalDescription, correctedDescription and errorType, with values such as po corrected to smudge and typed Other. Screenshot of the finished correction tool with its console open. Each record carries the four parts shown; where the spot sits on the page and how sure the software was stay with the original readings and are matched back up by number. — **Each fix comes out ready to use.** The spot number, what Tesseract read, what the paralegal typed and the mistake type, saved as one record.

Outcomes

Across 1,250 training documents, the model's confidence on its doubtful spots rose from 86% to 93%. That is how sure it was, not proof it was right. We never set aside a batch of pages it hadn't learned from to test it on, so all I can say is that it got more sure of itself, not more correct.

The time saved was never formally measured. The cost it was weighed against, roughly 12 doubtful spots per document at 30 seconds each at $35 an hour, is my estimate from watching sessions, not a formal time study.

A four-item list of mistake types worked out by the paralegals and our engineer.
A built-in zoom, because the browser's zoom pulled the outlines off their words.

Retrospective: what stalled it

The most useful lesson came after it worked. Nobody had agreed who owned it. Once it was clearly running, the questions came up: who controls the paralegals' corrections, who's allowed to use the results for real work, who signs off on retraining the model? Nobody had the legal or technical standing to answer them, and the project stalled there. Getting the model right was half the job. Getting the people around it to agree was the other half, and nobody had planned for it.