From Flaky to Firepower: Game-Changing Updates to Frontline AI

For decades there has been fantastic potential for AI to help improve efficiency in manufacturing, and in many ways, AI has already helped companies achieve great improvements. Predictive models that consume sensor data and analyze condition-based trends are a great example.

But what about when equipment fails? The downtime caused by these failures is the largest source of avoidable cost for manufacturers, and yet the workers responsible for addressing these failures are still using outdated technology (or no technology at all) to get those assets back on line as fast as possible. Frontline AI that seeks to assist skilled labor in responding to downtime events (e.g. Datch’s Diagnostic Agent) is the next most obvious use case for AI to drive manufacturing efficiencies.

The issue is that this type of AI has missed the mark for manufacturers to date and a lot of first movers were burned by companies that over-promised and under-delivered. It was not until very recently that critical technological breakthroughs have made the promise achievable. Datch is on the forefront of these breakthroughs, leveraging a approach that helps manufacturers take advantage of game-changing efficiencies driven by frontline AI.

‍

What is the problem?

In today’s world, with intense competitive pressure and shrinking margins, it is essential for manufacturing companies to improve mean time between failures, minimize downtime, and increase operational efficiency. However, challenges such as an aging workforce, increasingly complex industrial assets, and rapidly increasing quantities of data make this easier said than done.

A popular method of improving efficiency by enhancing workers' access to information has been to use a RAG (Retrieval Augmented Generation) search solution. That is, to create an AI chatbot backed by a vector database loaded with embeddings from all the unstructured information in the company’s knowledge base. The intention here is that the chatbot will give every maintenance worker instant access to accurate and relevant contextualized knowledge. This kind of solution dazzles in a demo but utterly collapses in production.

‍

Why is the problem a problem?

Building a useful production RAG search system is a deceptively hard problem that crumbles under real-world pressure, and this is why many companies are rightfully skeptical of the impact that frontline AI can make. Here’s why:

• Scaling issues: As more data is added into a RAG system, it becomes increasingly harder to find the specific record that is relevant to answer a worker’s question. Datch’s internal analysis shows a clear inverse relationship between the number of records in the vector database and traditional RAG search answer quality. Answer quality drops most sharply when the dataset is large and heterogeneous; that is, if it contains a diverse mixture of logs, manuals, specifications, field notes, etc.

• False positives: More data increases the risk of false positive answers. For example, a reference from page 309 of a manual may describe a valid restart sequence for a critical machine; but page 2, which is never retrieved, states that the procedure applies to a different model of machine than what the worker asked about. The AI here would confidently present the wrong answer, potentially with disastrous results.
• Hallucinations: Unoptimized RAG is prone to hallucinations, confidently presenting wrong answers, invented from the Large Language Model’s (LLM) internal knowledge, and not based on any of the referenced documentation. Even a single hallucinated instruction could be disastrous if it results in someone breaking a critical piece of equipment.
• Missing context: RAG has no notion of which belt, mill, robot, or line the worker cares about. It has no context of who the worker is, where they are in the factory, or what their role is. Without this critical context, an answer that may be correct for one worker might be useless for a different worker.
• Unstable quality: AI is advancing at such a rapid pace that any system built today will be obsolete next year. A new LLM model might result in improved answer quality, but it might also result in new unforeseen hallucinations. It is very difficult to reliably measure RAG answer quality because every answer an LLM gives is slightly different. However, without a reliable answer quality metric, there is no way to ensure the system remains trustworthy as LLM models and data sources change over time.
• Answer quality ceiling: With sufficient prompt engineering a best-in-class RAG solution can be a helpful tool for some use cases. However, some questions are impossible for it to answer. For example, “How frequently has this alarm occurred in the last three months on this line?” RAG search cannot answer such a question because it cannot understand or aggregate information like this.
• Microsoft GraphRAG limitations: Microsoft’s GraphRAG is a popular technology with the promise of increasing RAG answer quality. It stitches subject-verb-object relations across different documents, and is indeed a proven method for improving RAG answer quality. However, it is still regular RAG vector recall search at heart, so the same scaling, quality ceiling, and hallucination challenges remain.

‍

What’s the solution?

Datch Cortex integrates several key proprietary technologies to address all the above challenges. The result is a maintenance tool thatdelivers high-quality answers workers need, not just in demos but in real production environments at scale:

• Custom ontology: Understanding the relationship between different data sources within the customer’s domain and encoding this conceptualization into a formal specification. This specification then becomes a guide for the AI to make sense of the data. That is, the ontology teaches the AI how to speak the organisation’s language.
• Knowledge graph: The Knowledge Graph is a manifestation of the ontology as a graph of connected data. The graph might include individual records of: alarm data, OEM manuals, troubleshooting guides, vendor relationships, spare parts, costs, work orders, worker HR records, resolution steps, and asset hierarchies, or anything else relevant for helping improve workers' access to useful information.

• Ingestion pipeline: This takes the raw structured and unstructured data and organizes it into a knowledge graph. The pipeline also improves the underlying data quality by appropriately translating multiple languages, as well as converting, indexing, embedding, enriching, filtering, and linking records.
• Graph traversal algorithms: These are specific ways of navigating through the connected knowledge graph to answer critical questions that workers need answered. Datch Cortex optimizes its multi-hop graph traversal query strategy to optimally answer the specific question being asked.
• Contextual understanding: Understanding the workers’ role, location, relevant assets, work history, native language, expertise, preferences, and more. This enables the system to generate answers grounded in the worker’s specific context, not just generic data retrieval.
• Learning loop: A structured framework for integrating worker feedback, and measuring and improving AI answer quality over time. Using the learning loop framework, Datch routinely improves answer quality by applying custom graph traversal strategies, refining prompts for specific queries, enabling chain-of-thought reasoning, tuning vector and full-text search, re-ranking results, selecting LLMs based on question type, and optimizing context window size. With this loop in place, we can confidently make improvements without risking regressions in answer quality. The result is a system that avoids hallucinations and false positives, even as the underlying data volume scales up.

‍

Why does the solution work?

Datch, equipped with rich context and a knowledge graph full of connected information, can proactively provide workers with the information they need to better fix and maintain industrial assets: OEM guidance, prior work orders, likely spare parts, specs, internal SMEs, and estimates of time to repair; all these are quickly and easily available to every worker. No LLM-prompting gymnastics required.

A learning loop flags and tracks any hallucinations to make sure workers can trust the answers they receive. Most crucially, as the data volume scales, the knowledge graph density also increases, resulting in better answer quality. This is the opposite of what occurs with a traditional RAG solution.

Datch’s knowledge graph-based maintenance tool transforms AI from a flashy demo into a serious production-ready diagnostic assistant that gets smarter over time. Datch is an indispensable tool for manufacturing teams that continues to push the boundaries of AI-powered troubleshooting.
‍

If you’d like to see it in action, please reach using the form below.