A single high-volume medical device now generates thousands of complaint records monthly, arriving through call centers, hospital portals, social media, and field reports. Many of these records lack detail, written by users who may not have witnessed device failure firsthand. Yet every complaint must be evaluated promptly. Under 21 CFR Part 803, manufacturers have 30 days to report events linked to death, serious injury, or recurring malfunctions. Certain events that require remedial action to prevent an unreasonable risk of substantial harm to public health must be reported within five work days.
Manual triage systems were not built for this scale. Senior reviewers handle all complaints, leading to backlogs and inconsistent assessments. Two experienced reviewers may classify the same vague report differently, or the same reviewer might alter their judgment based on the day. Before AI, triage was already the weakest point in post-market surveillance. With complaint volumes skyrocketing, the system has become unsustainable.
AI presents a clear solution—but it introduces risks alongside its advantages. When applied correctly, it eliminates gaps that manual processes have always contained. No complaint remains unexamined. For a process where regulatory deadlines begin before files are opened, this efficiency alone justifies the transition.
A reviewer processing 60 complaints daily may overlook connections between reports, such as a Tuesday complaint matching an issue from three months prior in another region. An algorithm identifies such links instantly. Patterns that would take a human team months to detect can emerge within days. Unlike human reviewers, models maintain consistency across thousands of records, applying the same logic to the 10,000th complaint as to the first.
However, human errors and AI errors fail in fundamentally different ways. A reviewer’s misjudgment is typically random and can be corrected through a second review or peer assessment. But when an AI misclassifies a complaint, the mistake is systematic. If a model learns that a specific event type is low priority, it will deprioritize every instance—silently and at scale—until someone detects the oversight. Since buried records are often ignored, the delay in discovery can be prolonged. And because the system operates as designed, no single person may recognize the failure until it surfaces during routine review.
Read Also: WHO approves historic 20% funding increase
AI’s hidden risks and systemic blind spots
Models also degrade over time. Products evolve, new failure modes emerge, and users adopt different ways to describe problems. A model trained on last year’s complaints may overlook this year’s risks. Meanwhile, reviewers who depend too heavily on an algorithm’s prioritization may weaken their own oversight, creating a cycle of over-reliance.
Later, a batch of pumps ships with a defect where alarms fail silently while the device continues operating. Complaints mention the silent alarm in passing, the model categorizes each as low priority, and the cluster remains buried for six weeks. No individual acted negligently—the system functioned exactly as programmed. Yet when the issue finally emerges during a standard review, the company must file a late MDR and explain to regulators why its triage system concealed the problem.
Regulators already demand AI validation and oversight
Many companies delay implementing AI governance until new, AI-specific regulations appear. Yet the necessary framework already exists, and regulators are already applying it. An AI triage tool functions as software within the quality management system. Under the Quality Management System Regulation, which incorporates ISO 13485:2016, clause 4.1.6 requires that any software used in the QMS be validated before deployment, with rigor matching the risk level. A tool influencing which complaints are read first, and which reportability decisions are made, poses significant risk.
Validation is not a formality. It involves defining the tool’s decision-making scope, testing it against records with known outcomes, and establishing acceptance criteria before live data processing begins. Inspection practices reinforce this requirement: the FDA’s current compliance program treats Medical Device Reporting as a priority in nearly every inspection. Investigators analyze complaint trends and reporting history to determine where to focus. If they inquire about complaint prioritization and learn an algorithm is involved, the follow-up questions are predictable: Show the validation records. Demonstrate ongoing monitoring. Identify who determined an event was not reportable.
Notified bodies ask the same questions: how was the system validated, and who is accountable for its output? Records must document the model’s suggestions, the human reviewer’s decisions, and the individual responsible for each call. If an override lacks a named decision-maker, auditors will classify it as a compliance failure.
Read Also: Africa Struggles With Mpox Surge and Strained Health Systems
Where AI stops and human judgment begins
This boundary must be clearly defined in procedures, with assigned roles and preserved validation records. If an investigator requests validation files for an AI tool, the response must include the model’s training data, performance metrics against known reportable events, and evidence of retraining after major changes, such as a new product launch or shifts in complaint language. Without these records, the tool’s use shifts from a compliance safeguard to a regulatory liability.
Continuous monitoring of the model’s output is mandatory. For example, a manufacturer might review a monthly sample of complaints the model flagged as low priority, with a qualified reviewer independently assessing them. This mirrors how production processes are monitored under ISO 13485:2016, where deviations require corrective action and documentation.
Defining the model’s role and human oversight
It may never decide what the items in the queue are. Table 1 in the FDA’s guidance on AI in medical devices, issued alongside the 2026 QMSR amendments, explicitly distinguishes between automatable and non-automatable functions. For instance, a model may recommend whether a complaint warrants investigation, but the human reviewer must confirm the determination and document their reasoning. If the model identifies a cluster of infusion pump alarms failing silently, a reviewer must still evaluate whether the pattern meets MDR criteria under 21 CFR §803.50.
Teams that can answer both get the speed, the pattern detection, and a queue where every record is read on arrival, without inheriting the silent failure mode. Teams that cannot answer will find their blind spot eventually, usually at the worst possible moment.
