Generative AI / RAG Engineering

Industrial Knowledge Copilot

Enterprise RAG & Agentic AI for Industrial Knowledge Intelligence

Query-aware hybrid retrieval for industrial documentation. An evaluated search foundation for evidence-grounded technical assistance.

PythonSentence TransformersQdrantBM25RAG · TargetLLMs · TargetFastAPI · TargetDocker · Target

View Source on GitHub

Project Overview

Manuals, maintenance procedures, equipment specifications, alarm catalogs, and spare-part documentation contain answers. Engineers need a reliable way to find them without searching documents.

The retrieval engine combines Qdrant search, conditional BM25, and shared metadata filtering. The target RAG system adds LLM answers for equipment alarms, maintenance procedures, and spare parts.

Evidence-first retrieval

Select relevant technical evidence before constructing an answer.

Query-aware search

Adapt semantic and lexical retrieval to the question.

Measurable retrieval quality

Evaluate ranking with controlled development benchmarks.

System Architecture

Prepare the knowledge offline. Route each question deliberately. Return evidence with its sources. Evaluate retrieval separately.

Trace an example route

“When should the MX-300 hydraulic pump be inspected?”

DENSE ROUTE

A machine model supplies context; it does not, by itself, activate BM25.

01 / Sources

Synthetic documents

Canonical YAML → Markdown
Manuals · SOPs · guides

Offline preparation

02 / Structure

Section-aware chunks

Document IDs · heading paths
Equipment & revision metadata

Chunk text → BM25

03 / Representation

Title-aware embeddings

all-MiniLM-L6-v2
Document title + chunk context

Vectors + metadata → Qdrant

Input

User question

Query analysis

Identifiers & parts intent

Shared constraints

Candidate eligibility

Route selection

Dense only

Mode + constraints

Shared metadata eligibility before top-K

BYPASSED

BM25

Lexical retrieval

Terminology-sensitive
Eligible chunks only

ALWAYS USED

Qdrant

Dense vector search

Query embedding
Semantic similarity

Dense route

Preserve semantic order

Fusion & identifier boosting bypassed

Ranked source evidence

Implemented output

Ranked evidence

Chunk text · document ID
Section & source metadata

Planned

Grounded LLM answer

Evidence-supported response
Generated source citations

Offline assessment · outside the query-serving path

01 / Define

Benchmark cases

Queries + expected evidence

02 / Measure

Retrieval quality

Ranked results → Hit@K · MRR

03 / Verify behavior

Regression checks

Routing · constraints · metrics

Benchmark assessment and regression tests are complementary offline checks.

Data / evidence flowRouting controlOffline evaluationPlanned boundary

Inspect / C

Evidence retrieval

Implemented · local V2

Search eligible evidence semantically, and add lexical evidence only when the routing decision calls for it.

Engineering decision

Qdrant retrieves dense candidates. On the hybrid route, BM25 searches the same eligible chunk set before top-K truncation. Reciprocal rank fusion combines the lists; exact identifier boosting then adjusts the hybrid ranking.

Trade-off

Eligibility and ranking solve different problems. Fusion must not reintroduce documents excluded by another branch. The dense route retains its semantic order and bypasses fusion and identifier boosting.

Dense: Qdrant → ranked evidence. Hybrid: Qdrant + BM25 → RRF → identifier boost → ranked evidence. There is no learned reranker in this implementation.

SOURCE / hybrid_retriever.py · hybrid.py · identifier_boost.py · V2 report §44

Text architecture & development-stage notes

Knowledge preparation. Synthetic Markdown documents become section-aware chunks with metadata. Title-aware embeddings and metadata payloads populate Qdrant; chunk text supplies the BM25 index.

Query and evidence flow. Query analysis builds shared eligibility constraints and chooses dense or hybrid retrieval. Dense queries use Qdrant and retain semantic ranking. Hybrid queries use Qdrant and eligible BM25 candidates, followed by reciprocal rank fusion and exact identifier boosting. Ranked evidence retains source metadata.

Implemented versus planned. Evidence retrieval and source traceability are implemented. Evidence-grounded LLM responses and generated citations are planned. The route examples illustrate verified rules; they do not execute retrieval.

Separate development stages. This map describes local V2. Earlier in-memory, cosine-similarity experiments preceded Qdrant. Their results are not measurements of this later architecture. Development metrics and the reused 12-query diagnostic suite remain distinct; regression tests are offline checks, not production monitoring.

FIG. 01 / Industrial Knowledge CopilotLocal V2 retrieval architecture · select a subsystem to inspect

Core Capabilities

Implemented / Evaluated

Query-Aware Hybrid Retrieval

Semantic search with conditional lexical retrieval.

Implemented / Evaluated

Contextual Document Search

Document-title context and structured metadata.

Target Architecture

Evidence-Grounded Generation

LLM answers supported by retrieved technical documentation.

Target Architecture

Agentic Technical Investigation

Tool-assisted industrial troubleshooting workflows.

Implemented / Evaluated

Retrieval Evaluation

Hit@K, MRR, controlled experiments, and benchmark integrity.

Target Architecture

Backend and Deployment

FastAPI services and containerized deployment.

Investigation in Action

Identifier: HX-417
Equipment: MX-200
Strategy: Dense + BM25

Source evidenceIllustrative order

Troubleshooting guide / Relevant section

HX-417 · Diagnostic Checks

Lists hydraulic-pressure diagnostic checks and references SOP-MNT-001.

Equipment / identifier metadata: MX-200 · HX-417
Source file: TSG-MX200-001.md

Open synthetic source

Verified synthetic source documents. Display order is illustrative; no retrieval is executed, no scores are claimed, and no LLM answer is generated.

Retrieval Performance

Retrieval quality by evaluation suite

Hybrid Retrieval V2 engineering report · §§40–41
Latest local checkpoint · former held-out set reused for debugging

Development · 25 queries
Diagnostic · 12 queries

Hit@1

0–100%

Development Hit@1: 84%

Diagnostic Hit@1: 100%

Hit@3

0–100%

Development Hit@3: 92%

Diagnostic Hit@3: 100%

Hit@5

0–100%

Development Hit@5: 100%

Diagnostic Hit@5: 100%

MRR

0–1 scale

Development MRR: 0.891

Diagnostic MRR: 1.000

The 12-query suite informed V2 changes and is now diagnostic. Its perfect retrieval scores do not establish unseen-query generalization. Development and diagnostic results belong to this V2 checkpoint.

154 synthetic chunks · 90 automated tests passing at the local V2 checkpoint

Engineered Through Evaluation

Query-Aware Hybrid Retrieval

Finding: Naive hybrid fusion did not outperform dense retrieval.
Engineering change: Applied lexical retrieval selectively using query-aware routing and identifier handling.

Representation-Aware Retrieval

Finding: A procedural retrieval failure revealed missing document titles in embedding input.
Engineering change: Added document-title context to improve maintenance-procedure ranking.

Consistent Retrieval Constraints

Finding: Dense retrieval enforced metadata constraints; BM25 initially searched the full corpus.
Engineering change: Introduced shared candidate eligibility across retrieval branches.

Evaluation-Driven Iteration

Finding: Inspecting held-out failures makes that set part of the development process.
Engineering change: Relabel the reused 12-query suite as diagnostic and keep evaluation checkpoints separate.

Evaluation Integrity. Development and diagnostic benchmarks are explicitly distinguished from genuinely unseen evaluation.

Technology Stack

“Target” marks planned integration. RAG refers to the full retrieval-and-generation architecture.

AI & Retrieval

Sentence Transformers
Qdrant
BM25
LLMs · Target

Backend

Python
FastAPI · Target

Engineering

Pytest
Docker · Target
CI/CD · Target

Architecture

RAG · Target
Hybrid Retrieval
Metadata Filtering
Tool Orchestration · Target

Explore the Code

Explore the Engineering Behind the System

Explore the retrieval architecture, experiments, evaluation methodology, source code, automated tests, and engineering documentation.

View GitHub Repository Explore Architecture