The Association for the Advancement of Medical Instrumentation (AAMI) recently hosted a summit focused on one of the most debated topics in medical device validation today: in-use testing of medical devices.
Conceptually, in-use testing evaluates a medical device under real-world or simulated conditions to ensure it remains safe, functional, and easy for people to operate correctly. At the summit, industry experts representing manufacturers, healthcare facilities (HCFs), test laboratories, and regulatory agencies gathered to discuss the expectations, challenges, and practical realities surrounding this process, including variances in what in-use testing means.
While the meeting did not seek to establish definitive answers, it highlighted substantial uncertainty within the industry and underscored the need for additional guidance, potentially in the form of an AAMI white paper or future consensus document.
Defining In-Use Testing: Do We All Mean the Same Thing?
The summit began with a straightforward question: What exactly is in-use testing?
While the term is understood on the whole, participants quickly recognized that there is no universally accepted definition. Often used interchangeably with concepts such as clinical testing, real-world testing, simulated use testing, and post-market evaluation, the term “in-use testing” has a lack of alignment that creates challenges for manufacturers attempting to develop validation strategies that satisfy regulatory expectations.
A recurring observation throughout the summit was that even regulators have differing interpretations about when in-use testing is necessary and the evidence it should provide.
In-Use Testing Versus Simulated Use Testing
One of the most heavily debated topics centered on the relationship between in-use testing and simulated use testing.
Manufacturers routinely rely on laboratory-based simulated use testing to evaluate cleaning, disinfection, sterilization, and reprocessing effectiveness. Researchers design these studies to represent worst-case clinical conditions while maintaining a controlled and reproducible environment.
Three main questions were raised repeatedly during the summit:
- If a simulated use study adequately captures real-world conditions, what additional value can be gained through in-use testing?
- What is the value of in-use testing when validation testing incorporates worst-case processing conditions?
- How can in-use testing’s logistical challenges be overcome for medical products that haven’t yet been cleared for market?
Regulatory representatives acknowledged that there can be benefits from clinical in-use assessments. However, discussions suggested that manufacturers may justify the exclusion of in-use clinical testing if they can demonstrate that their simulated use studies sufficiently account for real-world use conditions. They will need demonstrate that their validation approach appropriately represents clinical reality.
The Human Factors Challenge
The role of human factors was another significant theme.
Clinical environments introduce variables that are difficult, if not impossible, to replicate perfectly in a laboratory setting. Differences in user technique, adherence to instructions for use, workflow constraints, device handling practices, and environmental conditions all influence device performance and reprocessing outcomes.
Participants debated where human factors should be evaluated within a validation strategy. Suggestions included:
- During early validation activities.
- As part of simulated use studies.
- Through clinical in-use assessments.
- During post-market surveillance.
There was no consensus, but there was overarching agreement that human behavior significantly impacts device performance and shouldn’t be overlooked when assessing real-world safety and efficacy.
Practical Challenges of Conducting In-Use Testing
The summit highlighted practical limitations that complicate clinical in-use testing efforts.
Questions raised included:
- Can a single healthcare facility provide access to all relevant device types?
- How can testing be performed without disrupting clinical workflows?
- How can in-use testing capture the inherent variability that is found in a clinical setting without creating an overly burdensome study?
- What incentives exist for healthcare facilities to participate?
- How should manufacturers manage legal agreements and responsibilities with healthcare facilities?
- Is it reasonable to expect healthcare personnel to support extensive validation activities?
Healthcare facilities are primarily focused on patient care and operational efficiency, participants noted. Extensive testing requirements can create workflow disruptions. They can also increase resource burdens and potentially interfere with clinical priorities.
These practical realities make broad implementation of in-use testing difficult, particularly for highly specialized or complex devices. They can lead to situations where in-use testing no longer reflects actual clinical processes. Examples included placing investigational equipment in locations that differed from actual clinical workspaces or handling and transporting samples in ways that varied from clinical practice.
Device Complexity Matters
Diversity of medical devices and whether a single approach to in-use testing is appropriate was another key topic.
Simple reusable surgical instruments present different challenges than highly complex devices such as flexible endoscopes or automated cleaning systems.
Discussion participants noted that device complexity affects:
- Cleaning effectiveness
- Biofilm formation potential
- Accessibility of internal surfaces
- Introduction of contaminants
- Sampling methodologies
- Data interpretation
- Clinical relevance
As a result, the risks, objectives, and feasibility of in-use testing may vary significantly depending on the device category.
Biofilm, Device Condition, and Real-World Variability
The summit also explored the challenge of evaluating devices that enter testing with pre-existing wear, damage, or contamination.
A key question emerged:
How should manufacturers assess and account for pre-existing device damage and associated biofilm accumulation?
Unlike laboratory studies that often utilize pristine test articles, clinically used devices may contain residual wear. They could also have surface defects or historical contamination that influence test outcomes.
Participants recognized that understanding these variables is essential if in-use testing is intended to reflect real-world use conditions. However, significant questions remain regarding standardization and sampling strategies. How findings should be interpreted also remains unresolved.
Can Post-Market Surveillance Fill the Gap?
Several attendees proposed that aspects of in-use testing could potentially be addressed through robust post-market surveillance programs.
Real-world performance data, complaint investigations, adverse event reporting, and field observations may provide valuable insights into actual device use without requiring extensive prospective clinical testing programs.
While this concept generated interest, no consensus was reached regarding whether post-market data can fully replace clinical in-use studies in all circumstances.
Is In-Use Testing Truly “Real World”?
One of the more philosophical debates centered on whether in-use testing truly represents real-world conditions.
Some participants questioned if so-called in-use studies become highly controlled exercises that are essentially simulations conducted within a healthcare facility.
Is it still representative of true clinical practice if the testing environment is structured, monitored, and constrained to meet study requirements?
This discussion reinforced the need for greater clarity on the specific objectives of in-use testing and the evidence it is intended to generate.
Current Guidance Falls Short
A consistent theme throughout the summit was the lack of current, harmonized guidance.
Participants referenced legacy guidance documents dating back to the early 1990s and generally agreed that existing resources don’t adequately address the complexity of today’s reusable medical devices, let alone automated reprocessing technologies or evolving regulatory expectations. Additionally, it was recognized that the industry, as a whole, has improved the way that simulated use, validation, and human factors studies are designed and conducted, and that existing guidance doesn’t always account for this evolution.
This gap has contributed to differing interpretations among manufacturers, laboratories, healthcare facilities, and regulators.
Looking Ahead
Perhaps the most important outcome from the summit was recognition that the industry needs additional clarity.
There was significant support for development of an AAMI white paper or consensus-based guidance document that could:
- Provide a shared definition of in-use testing.
- Clarify when in-use testing is appropriate.
- Establish risk-based approaches to decisions related to in-use testing.
- Define relationships between in-use, simulated use, and worst-case validation.
- Address practical implementation challenges.
- Establish expectations for data interpretation.
- Consider the role of human factors and post-market surveillance.
Conclusion
The AAMI In-Use Testing of Medical Devices Summit highlighted both the value and the complexity of in-use testing. While regulators acknowledge potential benefits, significant questions remain regarding feasibility, clinical relevance, standardization, and the relationship between in-use testing, simulated use testing, and worst-case validation.
The main message coming from the event wasn’t that in-use testing should always be performed, but instead that the industry needs to develop a better understanding of when it adds value versus times when alternative approaches are sufficient. It’s also necessary to determine how expectations can be consistently applied across stakeholders.
As medical devices grow in complexity regulatory scrutiny increases, collaboration among manufacturers, healthcare facilities, testing laboratories, and regulators will be essential to establish a practical and scientifically sound path forward.