Skip to content
nützliche.IT
AI in Practice2 min readBy Tyron Carlomagno

Local RAG in a Law Firm: From Pilot to Daily Work

A working RAG prototype is 30 % of the job. The decisive 70 % sit between pilot and production - and decide whether case workers actually open the tool on Monday morning.

RAG prototypes can be demoed in two weeks. Running RAG inside a law firm that genuinely uses it on Monday morning takes noticeably longer - and rarely fails because of the model.

What the Prototype Shows - and Hides

A pilot with 200 sample documents plausibly proves that retrieval-augmented generation works. It stays silent about everything that comes after:

  • Ingesting real data: case files with unusual naming, mixed formats, scanned PDFs with OCR artefacts.
  • Role and permission context: not every case worker may see every file.
  • Versioning and recency: a 2019 contract must not be cited as if it were the 2025 one.
  • Traceability: which passage produced which answer?

The 70 % Between Pilot and Production

Three blocks account for most of the effort in our experience:

  1. Ingest pipeline. Reliable extraction from DMS, network drives, and mail archives. Measure OCR quality, detect duplicates, segment text cleanly. Without this foundation the model choice is irrelevant.
  2. Access control. Every chunk row carries an access level checked at retrieval time. Better to show one document too few than one too many.
  3. Evaluation. A collection of real day-to-day questions with expected source citations. Without this benchmark you debate opinions.

Why Local AI in Law Firms Sounds Different in 2026

With client data, cloud inference is more than a data protection question: professions bound by confidentiality carry professional secrecy on top, and every external processor has to be brought in explicitly. At the same time, small models (7B-30B) have become production-capable on affordable hardware - Apple Silicon servers, mid-range GPUs, or even strong CPUs are sufficient for firms up to around 50 staff.

That shifts the question: no longer "cloud or nothing", but "which local setup matches the data volume and query load".

A Pragmatic Starting Point

Before ordering hardware, take stock:

  • Which ten questions do case workers ask their senior colleagues every day?
  • Where do the answers come from - case file, precedent, internal notes?
  • How long does the average search take today?

Those three answers form both the evaluation baseline and the business case. Everything else follows - not the other way round.

Call