AI Tech Inspire

Datasets and Tools to Test Document Forgery Detection on Medical Claims

Most fraud detection demos show doctored invoices or passports. But what happens when the document is a hospital bill or a discharge summary? That’s the question behind a recent request that caught the eye at AI Tech Inspire: build and evaluate a forgery detector for health-claim documents—fast, reliably, and with the right data.

What’s being asked, in plain bullets

  • A new team member is testing a document forgery detection agent for medical/insurance claims.
  • The agent uses GPT (specifically GPT-4.1) with a custom policy and prompt, ingesting base64-encoded images of documents (discharge summaries, hospital bills, prescriptions).
  • Primary goal: decide if a document is authentic or forged, focusing on image tampering, copy–move edits, and plausible fraud tactics.
  • Biggest blocker: data. Public datasets like DocTamper (CVPR 2023) don’t really cover medical/health-claim documents.
  • No known open dataset with paired authentic vs. forged health-claim reports.
  • Planned metrics: accuracy and recall; needs a balanced mix of genuine and tampered samples.
  • Explored approaches: synthetic generation via templates and programmatic tampering with OpenCV/Pillow.
  • Also considered: pretraining on general document forgery datasets or receipt corpora, then fine-tuning/evaluating on synthetic healthcare docs.
  • Looking for: any open dataset of forged medical/insurance claims, best practices to generate a realistic synthetic dataset, and tools for annotation (region-level masks or binary labels).

Why medical-claim forgery data is hard to find

Healthcare documents carry sensitive PHI, which makes open sharing rare and legally complex. That’s the core reason you won’t find a neat, public, “authentic vs. forged” dataset of claims and discharge summaries. Even when forms are standardized (e.g., CMS-1500 or UB-04 in the U.S.), the completed instances are not. This pushes teams toward synthetic pipelines that preserve structure but avoid real patient data.

Key takeaway: Expect to build your own synthetic set. Do it well, and you also get pixel-accurate ground truth for free.

A practical recipe for realistic synthetic health-claim docs

Think of this as a mini production line. The target is to mimic the full lifecycle of a claim document—from generation to real-world capture—then inject controlled tampering.

  • Start with public forms and templates: Use publicly available claim formats (e.g., CMS-1500, UB-04) and generic clinic/hospital templates. Generate variability with fields, fonts, logos, barcodes, and QR codes. Tools: ReportLab, WeasyPrint, LaTeX, or even template engines.
  • Create realistic captures: Export to PDF, then simulate print–scan cycles and phone camera captures. Apply perspective warp, lens blur, JPEG recompression, moiré, lighting gradients, and background clutter. This step matters more than it seems—many forensics cues live in these artifacts.
  • Inject tampering “attacks” in layers:
  • Content edits: change totals, dates, diagnosis codes, line items.
  • Copy–move: duplicate stamps, signatures, or line entries to inflate charges.
  • Splicing: transplant elements (e.g., hospital logo, physician signature) from another doc.
  • Resampling: scale/rotate patches to fit; leave subtle interpolation traces.
  • Recompression: multi-save JPEG sequences to blur trails.

Perform edits both pre-render (template-level) and post-capture (raster-level) to cover different attack surfaces. Libraries like OpenCV and Pillow handle most of this, while GPU-accelerated pipelines can lean on CUDA where helpful.

Tampering catalog to guide generation (and evaluation)

  • Numerical edits: totals, unit prices, taxes, currency symbols.
  • Textual edits: patient name, dates, physician, ICD/CPT codes, facility address.
  • Structural edits: duplicate or reorder table rows, cut-and-paste line items.
  • Signature/stamp manipulation: copy–move, opacity tweaks, resizing, fake seals.
  • Barcode/QR anomalies: mismatched data vs. rendered text, low-res reprints.
  • Layout consistency: spacing, kerning, font mismatches after splicing.

For an extra-hard split, include template transfer (tampering using a different hospital’s style) and multilingual elements, then hold out entire templates for validation.

Annotation pipeline: pick your fidelity (binary vs. pixel masks)

Binary labels (authentic vs. forged) are fast to create, but region-level masks unlock localization metrics and error analysis.

  • Auto-label from the generator: When you paste or edit a region, emit a matching mask. Save metadata in JSON (operation, bbox, mask_path, source_id).
  • Human-in-the-loop: For realism checks, review a subset in CVAT or Label Studio. Annotators can adjust masks and spot over-synthetic artifacts.
  • Versioning: Store images and labels with Hugging Face Datasets, DVC, or Git LFS and keep a manifest of generation seeds for reproducibility.

Modeling: pair an LLM with classical forensics

LLM vision models (e.g., GPT-4.1) can reason over layout, semantics, and inconsistencies. But purely visual forgery traces—like resampling or double JPEG compression—are often better captured by low-level forensics. A strong pattern is a multi-signal agent:

  • Stage 1: Classical detectors for copy–move (keypoints/patches), ELA/noise residuals, and barcode text decode vs. OCR consistency.
  • Stage 2: A vision-language model (e.g., GPT) consumes the image and signals from Stage 1 (summarized textually) to produce a verdict and rationale.
  • Stage 3: Policy layer with thresholds and risk scores; triage ambiguous cases.

If training a learned baseline, many teams reach for PyTorch or TensorFlow with a patch-based CNN/Vision Transformer for tamper localization, while keeping the LLM for high-level reasoning. Hosting data and models on Hugging Face makes iteration smoother.

Evaluation that goes beyond accuracy and recall

Accuracy and recall are a start, but consider these additions to avoid brittle deployments:

  • Precision/F1: essential when false positives have cost (e.g., claim delays).
  • AUROC/AUPRC: judge ranking under class imbalance.
  • Calibration: Brier score and reliability diagrams; calibrated risk scores aid triage.
  • Localization metrics: IoU/Dice for mask overlap when you have region labels.
  • FPR@TPR: e.g., False Positive Rate at 95% TPR; practical for operations.
  • Generalization splits: hold out entire templates, facilities, or languages.
  • Stress tests: compression level, lighting variance, print–scan depth, smartphone models.

Hardening with adversarial synthesis

Real adversaries don’t stop at cut-and-paste. Add modern attacks to your generator:

  • Inpainting to clean up edited totals or signatures using a diffusion model like Stable Diffusion.
  • OCR-aware edits: ensure the OCR output matches the visual tamper to reduce obvious inconsistencies.
  • Printer/camera fingerprinting: mix multiple device profiles so the model can’t overfit.

Privacy and compliance guardrails

Stick to synthetic or fully de-identified sources. If real documents are ever used, route them through a PHI-redaction pipeline and ensure contractual controls. Keep provenance for every image (source, ops, hashes), and separate training from evaluation by template and origin.

Suggested baseline stack (sketch)

Here’s a minimal, reproducible setup that teams can spin up quickly:

  • Generation: Template engine → PDF → rasterize → augment (print–scan/camera) → tamper ops (OpenCV/Pillow) → auto masks/JSON.
  • Forensics: ELA/noise residuals, copy–move via keypoints/patch-matching, barcode decode vs. OCR diff (Tesseract/EasyOCR).
  • LLM reasoning: A VLM like GPT-4.1 consumes the image plus structured cues; outputs verdict + explanation.
  • Metrics: Binary (AUPRC/FPR@TPR), localization (IoU), and calibration. Track by tamper type and capture mode.

Why this matters for developers

Document forensics drifts quickly: new templates, new capture devices, new fraud tricks. Building a generator-first evaluation dataset gives developers a living benchmark they can expand with each incident report. It also decouples low-level visual cues from high-level reasoning, which keeps the system transparent and maintainable.

At AI Tech Inspire, this trend stands out: teams that own their synthetic pipeline ship faster, localize failures better, and negotiate operational thresholds with stakeholders based on calibrated evidence—not vibes.

Open questions worth exploring

  • How much does a VLM’s verdict change when low-level forensics disagree? Should the policy default to caution?
  • Can lightweight mobile-side checks (e.g., barcode-text consistency) catch cheap forgeries before they hit the back end?
  • What’s the best way to simulate jurisdiction-specific forms and multilingual clinics at scale?

Resources to jumpstart the build

If you’ve seen a credible open dataset specific to health-claim forgeries, the community would benefit from a pointer. Until then, the most reliable path is a controllable, well-annotated synthetic pipeline with rigorous, real-world capture simulation.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.