NSV Pavement Inspection Platform

Project Overview

This project is a fully automated, AI-driven road surface distress detection platform purpose-built for the inspection workflows used by national highway authorities and the engineering firms that survey on their behalf. Modern highway monitoring relies on Network Survey Vehicles (NSVs) — instrumented vans that drive entire stretches of highway capturing high-resolution forward-facing video, GPS coordinates, speed, and chainage data overlaid directly onto the footage. Manually reviewing this footage to catalogue pavement defects is one of the most time-consuming and inconsistent tasks in highway asset management; a single survey day can yield many hours of video, and a human reviewer's accuracy degrades quickly under fatigue.

The platform replaces that manual review process with an end-to-end pipeline that ingests raw NSV footage, automatically identifies and classifies pavement distresses, measures their physical dimensions, geo-references each defect against the route's chainage, deduplicates repeat sightings of the same defect across consecutive frames, and produces auditable outputs suitable for regulatory reporting and maintenance planning. The result is a substantial reduction in survey-to-report turnaround time, more consistent defect classification, and a complete digital trail linking each finding back to the exact frame, GPS coordinate, and timestamp at which it was observed. From a business-value perspective, the platform converts an expensive manual labour bottleneck into a repeatable, software-driven service offering.

Key Features & Functionality

The platform delivers a complete operational workflow from video ingestion through report generation. Automated multi-class distress detection recognises fourteen distinct categories of pavement distress including alligator/fatigue cracking, block cracking, longitudinal wheel-path and non-wheel-path cracks, transverse and edge cracks, reflection and slippage cracks, shrinkage, seam joint distress, meandering cracks, potholes, failed patches, and disintegration/ravelling.

GPS and chainage geo-referencing automatically tags each detected defect with latitude, longitude, distance, and chainage values extracted directly from the in-video metadata banner, producing a fully spatial dataset without requiring separate sensor logs. Per-defect dimensional measurement computes length, width, and surface area for every detection, enabling severity grading and quantity-based maintenance estimation.

Intelligent deduplication consolidates a defect that appears across many consecutive frames into a single tracked entity, eliminating the inflated counts that simple per-frame detection produces. An annotated review video is emitted with bounding overlays, class labels, and confidence scores, allowing inspectors to visually verify findings. Structured exports are produced as CSV inventories and GeoJSON feature collections suitable for direct ingestion into GIS dashboards, asset management systems, or regulatory submissions.

In-application model training lets operators launch additional training runs against newly labelled data directly from the user interface, allowing the detection models to be continuously refined. Hardware-bound licensing locks each installation to the machine on which it is deployed, protecting commercial distribution and providing controlled expiry of customer entitlements.

Technology Stack

The platform is built primarily in Python and combines several leading open-source frameworks.

• Computer vision and deep learning: PyTorch, the Ultralytics YOLOv8 segmentation framework, ONNX Runtime (GPU), TorchVision, and OpenCV underpin the detection, segmentation, and image-processing layers.
• Image analysis utilities: scikit-image, SciPy, NumPy, and Shapely handle morphological operations, geometric reasoning, and shape analytics.
• Optical character recognition: EasyOCR and Tesseract are used in tandem to read the on-screen GPS/chainage banner from the video stream.
• Geospatial libraries: pyproj and geopy support coordinate handling and spatial joins.
• Tracking and analytics: The Supervision and lapx libraries support multi-object tracking; pandas powers data wrangling.
• User interface: PySide6 provides the desktop application shell, with PyQtGraph and Matplotlib for visualisation.
• Reporting and exports: openpyxl and ReportLab produce spreadsheet and PDF deliverables.
• Security and licensing: PyCryptodome implements the cryptographic license verification layer.
• Automation and packaging: Watchdog supports folder-watching workflows, while PyInstaller bundles the application into a self-contained Windows executable for client distribution.

Technical Complexity & Challenges

Several aspects of the project required non-trivial engineering effort and careful architectural choice.

Reliable detection in noisy field conditions. NSV footage is captured under widely varying lighting, weather, and pavement conditions. Distinguishing a hairline crack from a tyre mark, an oil stain, or a shadow boundary is genuinely hard. The platform addresses this with a layered filtering strategy: a region-of-interest segmentation step that restricts detection to the actual road surface, a contrast and edge-density check that suppresses smooth dark regions likely to be shadows rather than cracks, persistence requirements that demand a candidate appear across multiple frames before being accepted, and per-class confidence thresholds tuned for the distinct visual signatures of each defect type.

Temporal coherence and chainage-aware deduplication. A defect lasting one second of footage may appear in many sequential frames; without temporal reasoning the system would count it dozens of times. The platform maintains a lightweight tracking layer that associates detections across frames using spatial overlap and class-consistency rules, then performs a second deduplication pass that buckets detections by their geo-referenced chainage so the same physical defect is reported once, regardless of how long the camera lingered on it.

Pixel-to-world geometry without survey-grade calibration. The platform converts pixel measurements into real-world centimetres and millimetres using a row-dependent ground-sample-distance model derived from camera height and vertical field-of-view assumptions, with optional homography support when calibration is available. This delivers usable physical measurements without requiring per-vehicle sensor calibration data.

Robust OCR of the in-video data banner. Extracting structured GPS and chainage values from the burned-in video overlay required careful preprocessing — region-of-interest cropping, adaptive thresholding, character-allowlist constraints, and pattern-driven extraction — because the overlay typography, colours, and content vary between survey vendors.

Hardware-bound licensing. A custom cryptographic licensing layer binds each installation to a hardware fingerprint computed from MAC addresses, disk identifiers, and CPU identifiers, then verifies a vendor-issued, RSA-signed license file at every launch. This design preserves the firm's commercial control over distribution while operating fully offline.

Design & User Experience

The application is delivered as a native Windows desktop tool built on the Qt framework, deliberately chosen over a browser-based interface so operators can work with very large local video files without incurring upload latency or bandwidth cost. The interface is organised around a single primary task — process a video and review the results — and is intentionally restrained in visual complexity to remain usable by field engineers and survey technicians who may not be technologists.

The main window is divided into a left-hand navigation column showing live per-class detection counts, a central video player for playback of the annotated output, and a structured results table that lists each consolidated defect with its identifier, class, confidence, frame range, duration, dimensions, and geo-references. Two large action buttons drive the main workflows — loading a new video for processing and launching a fresh training run against new annotation data — while playback controls and a one-click "open output folder" action handle the routine file-management needs that come up after a run.

The licensing experience is designed to be self-explanatory for non-technical users: the first launch on an unlicensed machine generates a hardware-fingerprint file and presents a clear message instructing the operator to send the file to the vendor in exchange for a signed license. Once the license is in place, subsequent launches are silent. Throughout the application, error states are surfaced through native modal dialogs rather than buried in console output, ensuring field operators see clear, actionable messages rather than tracebacks.

Scale & Scope

The project comprises a complete training-and-inference pipeline rather than a single script. A dedicated dataset preparation tool ingests multiple CVAT/COCO annotation exports, harmonises their category dictionaries, deduplicates images that appear across multiple labelling tasks, produces a stratified train/validation split, and emits a manifest CSV documenting the provenance of every output file. A frame extraction utility converts arbitrary input video into the labelled-frame format expected by the training framework.

Two distinct inference pipelines target different deployment scenarios — a single-model GPU-accelerated path with road-mask filtering, tracking, and dedup; and a configuration-driven path supporting GPS metadata joins, geometry-aware measurements, and GeoJSON export. A modular source tree separates concerns into geometry, OCR, GPS-joining, deduplication, and rendering modules. A PySide6 desktop application orchestrates these pipelines through background worker threads, and a hardware-fingerprint generator plus RSA-signature verifier support the licensing flow.

The defect taxonomy spans fourteen pavement distress classes drawn from established highway pavement-condition standards. The system supports configurable processing parameters covering frame stride, confidence thresholds, region-of-interest behaviour, OCR cadence, tracking persistence requirements, and shadow-rejection sensitivity, allowing the same codebase to be tuned for different vehicle setups, road types, and survey conditions.

Business Impact & Use Case

The primary user community is the professional services ecosystem surrounding national-scale highway maintenance: survey contractors who collect NSV footage, consulting engineering firms that audit road conditions on behalf of public authorities, and the asset-management arms of the highway authorities themselves. In jurisdictions where condition surveys must be repeated at regulated intervals along thousands of kilometres of roadway, the cost of manual frame-by-frame video review is substantial and the variability of human classification creates audit risk.

The platform compresses a process measured in person-days into one measured in machine-hours, with consistent classification logic applied uniformly across every kilometre of footage. The geo-referenced outputs flow directly into the planning workflows downstream: a pothole detected at a specific chainage becomes a line item in a maintenance work order; a high concentration of alligator cracking on a particular stretch becomes a candidate for resurfacing; a longitudinal wheel-path crack progression observed across successive surveys becomes a leading indicator of structural fatigue. By producing a stable, machine-generated record of pavement condition that can be re-run identically against future surveys, the platform also enables longitudinal studies that were previously impractical.

For the firm offering the platform commercially, the hardware-bound licensing model supports a clean per-deployment revenue model, while the in-application training feature creates a natural upsell pathway: as customers accumulate locally-labelled defects, they can refine their own model variants without redistributing source code or core intellectual property.

Development Approach

The codebase is organised for clarity of intent rather than maximum abstraction, separating reusable utility modules from the orchestration scripts and user-facing application. Configuration is externalised into a YAML file that controls per-class confidence thresholds, geometric assumptions, OCR behaviour, and output locations, allowing operational parameters to be tuned without code changes. Both inference paths are driven by structured argument parsers so headless batch use and interactive UI use share a common entry surface.

Background work in the desktop application is dispatched onto Qt worker threads to keep the user interface responsive during long-running video processing and training operations, with explicit signal-based completion handling so errors in subprocess execution surface as user-visible status rather than silent failures. File-permission contention on output CSVs — a frequent issue when operators leave previous reports open in spreadsheet software — is handled gracefully by falling back to timestamped alternate filenames.

The training-data conversion tool emphasises auditability: it produces a manifest CSV documenting every output image's provenance, a separate warnings file capturing every ambiguity it encountered, and a deterministic train/validation split driven by a fixed random seed so experiments are reproducible. Image-content hashing is used as a tie-breaker when multiple candidate source files match a given annotation, ensuring stable behaviour across re-runs.

The packaging strategy targets reproducible client delivery: dependencies are pinned to specific versions, the application bundles cleanly into a standalone Windows executable via PyInstaller, and the licensing layer ensures distributions cannot be silently relocated to unauthorised hardware. Together these choices reflect a delivery-oriented engineering posture in which the product must run reliably on customer machines without ongoing developer involvement, while remaining maintainable enough that the underlying detection models and pipeline parameters can continue to evolve.