Why Software Cannot Be Tested Exhaustively
During software development, testing is typically conducted to verify that the developed product functions correctly according to specifications. However, it is a widely recognized fact that software, due to its inherent characteristics, cannot be subjected to exhaustive testing. This article explains the reasons for this phenomenon from a mathematical perspective and describes quality assurance strategies based on contemporary regulatory requirements.
The Complexity of Conditional Branches and Their Combinations
Software contains numerous “conditional branches” that determine the direction of processing based on conditions in specific situations. For example, if a single feature contains ten conditional branches, each evaluated as yes/no, there would theoretically be 2 to the 10th power, or 1,024, different combinations possible.
Considering a more complex practical example, software implemented in medical devices typically contains hundreds to thousands of conditional branches. If fifty conditional branches exist, the combinations would reach 2 to the 50th power, approximately 1.125 quadrillion combinations. Testing every single one of these combinations would be practically impossible even with modern computers, and the effort required would appropriately be described as astronomical in scale.
As shown above, even modest numbers of conditional branches lead to exponential growth in combinations, making it practically impossible to comprehensively test every case. This phenomenon is referred to as “test explosion” in software engineering and is recognized as a fundamental challenge in the discipline.
Evolution of Quality Approaches in Regulatory Requirements
Regulatory authorities worldwide, including the U.S. Food and Drug Administration (FDA), clearly recognize the limitations of development methodologies that rely solely on testing. In particular, for software embedded in medical devices, regulators strongly recommend an approach that incorporates quality from the design stage, rather than relying on post-development testing alone.
The FDA issued guidance on “Computer Software Assurance (CSA)” in 2016 and underwent a substantial update in 2024. This updated version emphasizes that ensuring software reliability requires quality management throughout the entire development process. This approach is referred to as “Design for Quality” or “Quality by Design” (QbD).
The International Council for Harmonisation (ICH) has also clearly stated its position in the Q14 guideline, emphasizing the importance of building quality into the design stage in pharmaceutical and software development. This is no longer merely a recommendation but has become a fundamental expectation in modern medical device and pharmaceutical regulation.
The European Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR) require detailed explanations of “quality assurance systems” in technical documentation, which are expected to include quality built in from the design stage. Specifically, compliance with IEC 62304 (Life Cycle Processes for Medical Device Software) and ISO 14971 (Risk Management for Medical Devices) has become mandatory in the development process.
Implementation of Quality Built-In at the Design Stage
Building quality into the design stage requires more than mere requirements definition. The following processes are critical: First, during the planning phase, it is essential to anticipate known issues, potential risks, and problems reported in similar existing products, and to develop specific countermeasures beforehand. This approach enables minimization of the aforementioned test explosion and facilitates realization of a more efficient and reliable development process.
Concretely, the following activities are encompassed: execution of risk analysis, hazard identification, requirements refinement, architecture design, and establishment of testing strategy. In particular, a risk-based testing approach is effective for obtaining maximum impact with limited testing resources. In other words, more detailed testing is conducted for high-risk functions, while simpler testing suffices for low-risk areas, which is a deliberate judgment.
From 2024 to 2025, regulatory authorities’ requirements show a clear trend toward further stringency. In particular, requirements regarding software security are being strengthened, and compliance with IEC 81001-5-1 (Security of Medical Device Networks) and the NIST Cybersecurity Framework is being required. Furthermore, for medical devices equipped with artificial intelligence and machine learning, validation based on FDA guidance (March 2024 edition) and ICH E42 (Artificial Intelligence and Machine Learning in Medical Device Software) is becoming mandatory.
Conclusion
Although the necessity of rigorous testing in software development is widely recognized, it is practically impossible to comprehensively test all conditions for the reasons outlined above. Consequently, building quality into the design stage becomes an extremely important strategy.
Particularly in the context of medical device development, rather than relying on simple testing, it is possible to create higher quality and more reliable software by foreseeing potential issues at the design stage and planning strategic countermeasures in advance. The key point to be emphasized in this article is that such a systematic approach is an essential condition not only for meeting regulatory authorities’ expectations but also for realizing patient safety. We strongly recommend utilizing these insights in practical implementation.
