--- name: device-evidence-and-testability-review description: Use this skill when a medical device manufacturer's product, systems, quality, regulatory, postmarket surveillance, or complaint team wants to know what evidence a device produces, what its quality and regulatory work needs that the device doesn't produce, and whether the device is built to be tested. It runs the Four Truths Data Inventory, the Evidence by Operation Checklist, and the testability mechanisms from "Put AI to Work First on Verification, Validation, and Regulatory Evidence"; identifies what complaint investigation, adverse-event reporting, corrections and recall effectiveness, and predetermined change control plan (PCCP) monitoring would need that the device doesn't produce; and recommends the next releases' changes and where AI assistance fits (observation, recommendation, or workflow assistance only, with human sign-off, and never as validated evidence without validation of the AI tool). Triggers on requests like "what evidence does our device produce," "why do our complaint files say cause could not be determined," "can we investigate complaints from device logs," "make our device produce its own compliance evidence," "run the Evidence by Operation Checklist," "is our device testable," "can AI help with our verification or complaint work," "what do we need for PCCP monitoring," or "how do we confirm a field correction reached every unit." --- # Device Evidence and Testability Review This skill is Labor as Code™ for the evidence and testability method in *Medical Device Connectivity*. Labor as Code is a skill that encapsulates a person's knowledge, judgment, taste, and skill in durable, inspectable, versioned form. The know-how is the habit of asking, for every quality and regulatory job, which device record should back it, whether normal operation produces that record, and whether anyone, person or AI, can put the device into a known state and check what it did. The method spans four chapters. "Every Device Produces Four Kinds of Evidence" teaches the Four Truths and the Four Truths Data Inventory. "Make Devices Produce Their Own Compliance Evidence" teaches Evidence by Operation and its checklist, and what a machine record can and can't prove. "Put AI to Work First on Verification, Validation, and Regulatory Evidence" teaches the regulatory and quality jobs that assume someone can examine the device, the five testability mechanisms, and where AI belongs in that work. "An AI-Ready Device Is Mostly a Well-Engineered, Interoperable Device" teaches the AI role spectrum and the provenance set. The inventory and checklist are printed in full in the appendix "Field Tools for Builders." Read the chapters for the argument and the worked cases. This skill runs the method on the user's own device. It never borrows the book's pump inventory, sterilizer checklist, glucose meter example, conceptual ratings, or composite manufacturer as the user's facts. **The skill finds gaps and recommends. People decide cause, reportability, and acceptability.** The skill states which changes the evidence supports and why. It never decides the cause of a complaint, whether an event is reportable, whether a correction is reportable, or whether a record is acceptable as regulatory evidence. The complaint investigator, a qualified person in regulatory affairs, and the quality system decide those. **This skill's own output is not validated evidence.** An AI running this skill is an AI tool. Its output is a draft for a person to check. It becomes quality system evidence only after the AI tool is validated as quality system software and a qualified person signs the record. ## Before running 1. **Apply the book's data rule first.** Use only information the user's organization permits in an AI tool it has approved. Never accept patient information, credentials, or network addresses. A complaint comes with patient identifiers removed. A log or journal export comes with network addresses removed. If the material holds any of these, stop, name where it is, and ask for a cleaned version. Don't accept another company's confidential documentation. 2. **Get the device.** Record the device class, product, software version, and whether it is in development or fielded. 3. **Get the material.** Ask for each of these and record which ones the user provides: - the device's log, event, and export documentation, and the interface documentation; - sample exported records or journal extracts, cleaned; - the list of records customers, surveyors, auditors, and investigators ask for; - a sample of recent complaint files, adverse-event report narratives, corrections or recall records, and CAPA records, cleaned, with the evaluation outcome where the file records it; - the test architecture: any test or simulation mode, test interface, replay tool, and how requirements and trace links are stored; - for AI-enabled device software, the PCCP and its monitoring plan, if one exists; - any AI tool the team uses or plans to use in quality or regulatory work, with its validation status. 4. **Ask who owns the result.** Name the product owner, quality lead, and regulatory lead. If the user doesn't know, write "owner to be confirmed." ## Step 1: Run the Four Truths Data Inventory Fill the four rows in the fixed order. For each row, list each item the device produces, where it lives today, and who needs it. 1. **Clinical truth:** evidence about the patient or specimen. 2. **Therapeutic truth:** evidence about an intervention performed on or for a patient. 3. **Operational truth:** evidence about the work the device performed and the surrounding workflow. 4. **Assurance truth:** evidence that the device itself can be trusted. Record "where it lives today" as one of: screen, paper, internal log, file, interface, another system, nowhere. Cite the document or sample record for each item. Write "not in the material" where you can't tell. Then record a conceptual rating for each truth (High, Medium, Low, or Very high) only if the user's team supplies it or confirms a proposed one. Mark the ratings "conceptual: not scores." Never copy a rating the book gives for a similar device. Flag every operational or assurance item that lives only on the screen, on paper, in the internal log, in someone's re-keying, or nowhere. The book calls each one an operational blind spot the product creates for every customer. ## Step 2: Run the Evidence by Operation Checklist Score the ten rows. An item passes only when normal operation produces it, in machine-readable form, and the institution can export it without asking the manufacturer. | Evidence | Pass when | | --- | --- | | Cycle or treatment records | Every cycle or treatment is recorded with identities, parameters, results, and outcome, with no operator transcription, and joins automatically to related records in other systems | | Software inventory | The device reports its software versions and components on request and after every change | | Self-test results | Every self-test is recorded with its time and result | | Calibration | Calibration state and history travel with the data they affect | | Update history | Every installation, failure, and rollback is recorded with its time and version | | Security events | Security events are logged in a format the institution's security monitoring tools can consume | | Utilization | Use and idle time are recorded per unit | | Failure records | Every fault is recorded with its error code and the device state at the time | | Configuration history | Every configuration change is recorded with who made it and when | | Overrides and exceptions | Every override is recorded with the operator, the time, and the reason given | For each row, name the device event that produces the record, who uses it, the status (pass, partial, fail, or not in the material), and the source. A record the device holds but only a field engineer with a service cable can read fails "export without asking the manufacturer." Check each passing record for provenance: source device, time, units, quality, device state, calibration, configuration, transformation history, and whether each value was measured, entered, inferred, or derived. Check for integrity protection (append-only, hash-chained or signed). A record that can be changed without a trace is weak evidence. State the limit once in the output. A machine record proves that a process ran with stated parameters. It doesn't prove the process worked. Qualification testing answers that, and Evidence by Operation doesn't replace it. ## Step 3: Check the testability mechanisms Check the device for each of the five mechanisms. The book states these as its doctrine, built from mechanisms that exist; it found no FDA guidance or recognized standard that requires them. Label them that way. 1. **A test and simulation mode** that runs the real software on simulated inputs and can never be mistaken for clinical use. 2. **A documented test interface** that puts the device into a named state on command. 3. **Deterministic replay from the device journal,** so the same recorded inputs produce the same outputs on a bench unit. This depends on the journal rules in "Network Failure Is a Normal Operating Condition": sequence numbers within an epoch, device time with a time-quality flag and monotonic counter, and records that carry their own associations. 4. **Structured logs** with identifiers, times, device state, and software version, readable by a machine. 5. **Machine-readable requirements and traceability.** The book names open specifications (OMG ReqIF 1.2, OMG SysML 2.0 and its Systems Modeling API, OASIS OSLC Requirements Management 2.1), none specific to medical devices. Record what the team uses. Never state that the team conforms to one. Mark each present, partial, or absent, with the source. Note for each absent mechanism which later step it blocks. ## Step 4: Map the regulatory and quality jobs to the missing evidence For each job below, record the evidence it needs from the device, whether the device produces it (citing Steps 1 to 3), and what the user's sample files show. The rules are cited as the book states them, dated. Mark every regulatory reading "for regulatory affairs to confirm." The book labels some readings as the author's own (for example, that a remote software fix is a correction); keep that label. 1. **Complaint investigation** (Quality Management System Regulation, in effect since February 2, 2026, incorporating ISO 13485:2016; 21 CFR 820.35). Needs event and alarm history, device state, software version, and the unique device identifier (UDI). Without it, the file documents the attempt instead of the cause. 2. **Adverse-event reporting** (21 CFR Part 803). The manufacturer evaluates the device and reports within 30 calendar days of learning of a reportable event; if it didn't evaluate the device, the report says why. Needs the device's own record for the event. Count, in the user's sample, how many files say the device wasn't evaluated or the cause couldn't be determined, and whether a device record would have changed that. Use only the user's files; never quote the book's tabulation as the user's rate. 3. **Corrections and removals, and recall effectiveness** (21 CFR Part 806; 21 CFR 7.42). Needs which units received the fix, and when. The effectiveness check the rule describes is the consignee's response; the book proposes that connected devices report fix status directly. Needs the software inventory and update history rows to pass. 4. **PCCP monitoring** for AI-enabled device software. The book cites FDA's final PCCP guidance as anticipating that manufacturers monitor real-world safety and performance, with a risk-based plan. Needs real-world performance by model version from fielded units, and the software inventory to know which version produced each result. Record whether the user's plan can be carried out with what the device reports. 5. **Also check, where they apply:** CAPA (structured service, complaint, and test data); postmarket surveillance and real-world evidence (device-generated data joined by UDI; the UDI rule makes the identifier readable by a person, not reported over a network); vulnerability monitoring under section 524B (software inventory across the installed base); design verification and submission (traced requirements and repeatable tests). Write each gap as: the job, the missing record, the device change that would produce it, and the Step 2 row or Step 3 mechanism it maps to. ## Step 5: Place each AI use on the role spectrum For each AI use the team runs or proposes, record its place on the AI role spectrum: observation, recommendation, workflow assistance, optimization, supervised control, autonomous control. Allow only the first three for this work. - **Observation:** summarizing field logs, complaint trends, and postmarket performance data. - **Recommendation:** proposing test cases, candidate causes, and trace links for an engineer to accept or reject. - **Workflow assistance:** drafting complaint records, evidence requests, report narratives, and test reports, and routing complaints. Check each use against these rules. Any use that breaks one is a finding. 1. A qualified person signs every record the AI helped produce. 2. The AI never decides cause, reportability, or acceptability. A step that determines acceptability with limited or no human review is out. 3. The AI tool is quality system software. It is validated under ISO 13485 clause 4.1.6 and FDA's computer software assurance guidance before anyone relies on it. Until then, its output is a draft, never validated evidence. 4. Record the model, version, and settings with every test or record the AI helped produce. 5. Accept only output that execution or a source link can check: a test that runs, a quote that opens to its log entry. Where no mechanical check exists, keep the AI at observation. 6. Every generated test traces to a requirement or the complaint, never to the device's current behavior. 7. When an AI runs a test, it drives a simulator or a bench unit in test mode, never a device in clinical use. 8. The AI tool that drafts tests is quality system software. The device software it tests stays under the device software rules. ## Step 6: Conclude with a recommendation Close with a short statement, not an evidence pack. 1. **The next releases' changes,** in order. Put first the changes that let the team back the records it files today: the Step 2 rows and Step 3 mechanisms whose absence leaves complaint and adverse-event files without a device evaluation, and unit-level fix status for corrections and recalls. Then what PCCP monitoring and vulnerability monitoring need. Then the rest of the checklist. Give each change the job it serves and the gap it closes. Mark the order "proposed: the product, quality, and regulatory leads to confirm." Put every failed Evidence by Operation row on the roadmap with a date field for the team to fill; never supply the date. 2. **Where AI fits now,** by use: its place on the spectrum, the person who signs, the check that catches a wrong answer, and whether the tool is validated. Say plainly which uses must wait for validation or for a device change (for example, complaint triage against device logs needs a retrievable journal first). 3. **The reasoning,** in three to five sentences tied to specific rows, mechanisms, and jobs. 4. **What would change it:** the evidence that would reorder the list, such as a complaint sample showing a different dominant gap. Where two orders are both defensible (for example, a testability mechanism that serves verification now against a software inventory feed that serves a pending correction), present both side by side, each with its reasoning and what would make it the right call. Where the material is too thin (no log documentation, no sample records, no complaint files), say so plainly, list what would settle it, and don't rank changes nobody can tie to a job. ## Output template ``` DEVICE EVIDENCE AND TESTABILITY REVIEW Device class and product: ____ Software version: ____ In development or fielded: ____ Owners: product ____ quality ____ regulatory ____ Material reviewed: [list, with what was not provided] 1. Four Truths Data Inventory [truth | items produced | where each lives today | who needs it | source] Conceptual ratings (team-supplied, not scores): __ / __ / __ / __ Operational blind spots: ____ 2. Evidence by Operation Checklist: 10 rows [evidence | producing event | who uses it | pass / partial / fail | source] Provenance and integrity findings: ____ Limit: a machine record shows a process ran with stated parameters; qualification testing shows whether it works. 3. Testability: [mechanism | present / partial / absent | source | blocks] 4. Regulatory and quality jobs (for regulatory affairs to confirm): [job | evidence needed | produced? | what sample files show | device change] 5. AI uses: [use | spectrum position | who signs | check | tool validated?] Findings against the AI rules: ____ RECOMMENDATION Next releases' changes, in order (proposed): ____ Where AI fits now: ____ What must wait, and for what: ____ Reasoning: ____ What would change it: ____ Not decided by this review: cause, reportability, acceptability, regulatory status. This output is a draft from an AI tool, not validated evidence. ``` ## Rules - **Never invent a device capability, log field, record, standard, conformance result, regulatory status, date, or number.** Where the material is silent, write "not in the material." - **Never borrow the book's facts.** The book's worked pump inventory, sterilizer checklist, glucose meter example, conceptual ratings, adverse-event tabulation, and composite manufacturer are examples, not the user's device or data. - **Keep three voices apart.** A regulatory or standards statement (dated, and marked nonbinding where it is guidance), a research finding, and the book's doctrine are each labeled as what they are. The testability list and the AI-first ordering are the author's thesis, not a requirement. - **Never state a cause or decide reportability,** even when the logs seem to show one. - **No patient information, credentials, or network addresses** in the input or the output. Use only what the organization permits in an approved AI tool. ## Checks before calling the work done - All four truth rows, ten checklist rows, and five mechanisms are present, each with a source or "not in the material." - Every regulatory reading is dated and marked for regulatory affairs to confirm. - No AI use sits right of workflow assistance, and each names who signs and the check that catches a wrong answer. - No AI output is described as validated evidence without a validated tool and a signature. - The conclusion gives ordered changes with reasoning, labeled options, or a plain statement that the material is too thin and what would settle it. ## What this skill does not do It doesn't investigate a complaint, decide a cause, decide reportability, or file anything with a regulator. It doesn't validate an AI tool, including the one running this skill. It doesn't replace qualification testing, verification, or the manufacturer's quality system records. And it doesn't treat a complete-looking record as proof: an AI-drafted file with an invented log entry is worse than a thin one, because it looks complete.