To What Extent Should Cybersecurity Testing Be Conducted? (Part 1 of 3)

Introduction to Security Testing Requirements

Section 5.7 of IEC 81001-5-1 requires the implementation of the following security tests:

• Security requirements testing (Section 5.7.1)
• Threat mitigation testing (Section 5.7.2)
• Vulnerability testing (Section 5.7.3)
• Penetration testing (Section 5.7.4)

The fundamental question arises: Is the implementation of all these tests mandatory?

The answer is nuanced and depends on the risk profile and security level of the health software. However, IEC 81001-5-1 applies to all health software regardless of safety classification, meaning that unlike IEC 62304 which uses software safety classes, the cybersecurity testing strategy outlined in Section 5.7 generally applies comprehensively. Let us examine each testing category in detail.

1. Security Requirements Testing (Section 5.7.1)

Purpose and Scope

Security requirements testing verifies that the security requirements specified in the requirements specification have been properly implemented. This testing is fundamental to confirming that the software meets its security design objectives.

Testing Categories Based on Intended Use Environment

Based on the intended use environment, security requirements testing shall include the following types of tests:

Functional Testing of Security Requirements: This verifies that each security function operates as specified. For example, authentication mechanisms, access controls, encryption functions, and audit logging capabilities must be tested to confirm they perform as designed.

Performance and Scalability Testing: This examines whether security functions maintain their effectiveness under various load conditions. As system usage scales, security mechanisms such as encryption/decryption processing, authentication processing, and access control verification must continue to function properly without degradation that could introduce vulnerabilities.

Limit Testing (Testing That May Affect Security): This category explores system behavior at operational boundaries and includes several subcategories:

Boundary and Edge Condition Testing: Testing is conducted at the limits of acceptable input ranges, maximum concurrent users, maximum data volumes, and other operational boundaries to ensure security functions remain effective at these extremes.

Stress Testing: The system is subjected to loads exceeding normal operational conditions to verify that security functions do not fail or introduce vulnerabilities under stress. This includes testing behavior during memory exhaustion, CPU overload, network congestion, and storage capacity limits.

Malformed Input Testing: Intentionally malformed or improperly formatted data is provided to the system to verify that input validation and sanitization functions operate correctly and that the system handles such inputs securely without crashes or exploitable behavior.

Unexpected Input Testing: Inputs that are valid in format but unexpected in context are tested to ensure the system handles them securely. This verifies that the software responds appropriately to edge cases and unusual input sequences.

Testing for Software Services Used to Achieve Intended Functionality: For health software that relies on external software services, testing must verify the security of these integrations based on responsibility agreements between service providers, the manufacturer, and operators. This is particularly critical for modern architectures that depend on various service models:

Cloud Service Testing: General security testing for cloud infrastructure, including network security, data isolation, access controls, and compliance with security policies specific to cloud environments.

Software as a Service (SaaS) Testing: Testing focuses on application-level security, including authentication, authorization, data confidentiality, secure API usage, and proper handling of shared resources in multi-tenant environments.

Infrastructure as a Service (IaaS) Testing: Testing encompasses the security of virtual machines, network configurations, storage security, and the manufacturer’s responsibility for securing their own implementations on the infrastructure.

Platform as a Service (PaaS) Testing: Testing covers the security of the platform layer, including runtime environments, middleware security, platform API security, and the division of security responsibilities between the platform provider and the manufacturer.

Implementation Approach

These tests should be conducted for all security requirements specified in the requirements specification. The scope and depth of testing should be proportionate to the security level (SL) and capability security level (SL-C) determined through threat and risk analysis. Test plans must clearly document which security requirements are being tested, the testing methodology, acceptance criteria, and traceability back to the original security requirements.

2. Threat Mitigation Testing (Section 5.7.2)

Verification of Mitigation Effectiveness

Threat mitigation testing validates that each security control implemented to address identified threats operates effectively. This testing goes beyond functional verification to examine whether mitigations actually reduce risks as intended.

Verification Testing of Each Mitigation Measure: For each mitigation implemented to address specific threats, appropriate test content must be developed and executed to ensure the mitigation operates as designed. For example, if a rate-limiting mechanism was implemented to mitigate denial-of-service threats, testing would verify that the mechanism correctly limits request rates while maintaining legitimate service availability.

Obstruction Testing of Each Mitigation Measure: A critical component of threat mitigation testing involves attempting to circumvent or defeat the implemented security controls. Plans must be created and executed to actively try to bypass, disable, or undermine each mitigation measure. This adversarial testing approach helps identify weaknesses in the mitigation strategy that might not be apparent through positive verification testing alone. For instance, testing might attempt to bypass authentication controls, circumvent access restrictions, or find ways to disable security logging.

Verification That Mitigations Do Not Introduce Other Vulnerabilities: It is essential to ensure that the implementation of security controls does not inadvertently create new vulnerabilities elsewhere in the design. This is a common challenge in security engineering where fixing one problem can introduce others. For example, implementing encryption might introduce timing vulnerabilities if not implemented carefully, or adding input validation might introduce denial-of-service vulnerabilities if validation is computationally expensive. Testing must verify that the overall security posture improves and that mitigations work in harmony rather than creating new attack vectors.

Independent Testing Requirements

IEC 81001-5-1 Section 5.7.5 addresses the critical issue of managing conflicts of interest between testers and developers. The standard requires appropriate independence of testers from developers to ensure objectivity in assessing test results. The level of independence required increases with the security level of the system. Testing by the same individuals who developed the code may be acceptable for lower security levels, but higher security levels require testing by independent teams or even external organizations. This independence is essential to avoid the common pitfall of unconscious bias where developers may not effectively test their own assumptions and design decisions.

3. Vulnerability Testing (Section 5.7.3)

Foundation in Known Vulnerabilities

Testing for known vulnerabilities must be based, at a minimum, on recent content from established and industry-recognized public sources of known vulnerability information. This ensures that the software is protected against publicly disclosed security weaknesses that attackers are likely to exploit.

Public Vulnerability Databases: Manufacturers should regularly consult sources such as the Common Vulnerabilities and Exposures (CVE) database, the Common Weakness Enumeration (CWE) database maintained by MITRE, the National Vulnerability Database (NVD) maintained by NIST, and security advisories from software and component vendors. These databases provide comprehensive information about known security weaknesses and their potential impact.

Abuse Case Testing

When necessary, abuse case testing should be implemented. Unlike traditional use case testing which validates intended functionality, abuse case testing focuses on finding security issues by examining how the software responds to intentional misuse, malicious inputs, or unexpected inputs designed to exploit potential vulnerabilities.

Manual Abuse Case Testing: Security testers manually craft test cases that simulate attacker behavior, attempting to exploit potential vulnerabilities through creative misuse of the system. This requires security expertise and knowledge of common attack patterns. Manual testing is particularly valuable for identifying logic flaws and complex attack chains that automated tools might miss.

Automated Abuse Case Testing: Automated tools are employed to systematically test for common vulnerability patterns across all interfaces and protocols. These tools can efficiently test large attack surfaces and can run continuously throughout development.

Testing Scope – All External Interfaces and Protocols: Both manual and automated abuse case testing must be conducted against all external interfaces and protocols, including user interfaces, application programming interfaces (APIs), network protocols, file formats, hardware interfaces, and any other points where the software interacts with external entities.

Specialized Types of Abuse Case Testing

Certain sophisticated testing techniques fall under the category of specialized abuse case testing:

Fuzzing: Fuzzing is a systematic testing technique that involves providing intentionally malformed, abnormal, or random data in large volumes to software inputs and monitoring for crashes, unexpected behavior, or security failures. This technique is highly effective at discovering input validation vulnerabilities, buffer overflows, memory corruption issues, and other security flaws that might not be found through traditional testing approaches.

Network Traffic Load Testing: This specialized testing examines how the software handles abnormal network conditions, including excessive traffic volumes, malformed packets, unusual protocol variations, and other network-level attacks.

Capacity Testing: Testing explores system behavior when capacity limits are reached or exceeded, including disk space exhaustion, memory exhaustion, connection pool exhaustion, and other resource limitations that might be exploited by attackers.

Understanding Fuzzing

Since fuzzing is specifically mentioned in the standard and may be unfamiliar to those new to security testing, it merits detailed explanation.

What is Fuzzing?: Fuzzing is one of the software testing techniques used to discover vulnerabilities in software. Specifically, it involves deliberately sending large volumes of invalid data, abnormal data, or random data to a program and verifying whether the program crashes or exhibits unexpected behavior.

Purpose of Fuzzing: Fuzzing is primarily used to discover security vulnerabilities. Malicious attackers may exploit program vulnerabilities to penetrate systems or steal information. Therefore, it is crucial to conduct fuzzing during the development phase and fix vulnerabilities before the software is deployed.

Types of Fuzzing: There are primarily two types of fuzzing approaches, each with distinct advantages:

Black-Box Fuzzing: This method tests by inputting data from the outside without understanding the internal structure of the program. Black-box fuzzing is advantageous because it does not require access to source code and can test the software from an attacker’s perspective. However, it may be less efficient at discovering vulnerabilities that require specific input sequences or internal state conditions.

White-Box Fuzzing (or Grey-Box Fuzzing): This method analyzes the source code to understand the program’s internal structure, then targets testing at locations where vulnerabilities are likely to exist. White-box fuzzing is more efficient at finding complex vulnerabilities because it can use code coverage information to guide test input generation. This approach often uses instrumentation to monitor code execution paths and adapt the fuzzing strategy to explore previously untested code paths.

Automation and Efficiency: Fuzzing is often performed using automated tools, enabling efficient discovery of vulnerabilities. Modern fuzzing tools have become increasingly sophisticated, incorporating techniques such as coverage-guided fuzzing (where the fuzzer uses code coverage information to evolve more effective test inputs), grammar-based fuzzing (where the fuzzer understands the structure of valid inputs and generates meaningful variations), and symbolic execution (where the fuzzer uses formal methods to systematically explore program execution paths).

Integration into Development Processes: Continuous fuzzing, where fuzzing tools run automatically throughout the development process, has become a best practice. This allows vulnerabilities to be discovered and fixed early when remediation is less costly and disruptive.

Mandatory Nature of Testing Activities

While IEC 81001-5-1 does not explicitly use the concept of software safety classes like IEC 62304, all requirements generally apply to all health software. However, the standard does recognize that a risk-based approach can be applied through the concept of Security Levels (SL) as defined in IEC/TR 60601-4-5. The extent and rigor of security testing should be commensurate with the security level and the potential impact of security breaches.

For software in lower security levels (SL-C 1 or 2), certain streamlined approaches may be acceptable, but the comprehensive security testing strategy outlined in Section 5.7 remains fundamental. Organizations may also reference Annex G of IEC 81001-5-1, which provides guidance for “transitional health software” – legacy software being brought into compliance – where full implementation of architectural documentation may be deferred but the full security testing strategy of Section 5.7 must still be implemented.

The independence requirements for testing (Section 5.7.5) are particularly important. As security level increases, greater independence between testers and developers becomes necessary to ensure objectivity and avoid confirmation bias. For the highest security levels, independent third-party testing, including penetration testing by external security firms, becomes essential.

Regulatory Context and State of the Art

IEC 81001-5-1 was published in December 2021 and has rapidly gained recognition as the state of the art for medical device cybersecurity. The standard has been designated as a harmonized standard under the EU Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR), with harmonization effective from May 2024. In the United States, IEC 81001-5-1 is recognized as a consensus standard by the FDA and is explicitly referenced in the FDA’s premarket cybersecurity guidance as an acceptable Secure Product Development Framework (SPDF).

Manufacturers developing medical device software should implement these testing requirements not only for regulatory compliance but as essential engineering practice for developing secure, trustworthy health software. As cybersecurity threats continue to evolve, rigorous security testing throughout the software lifecycle has become indispensable for protecting patient safety and maintaining the integrity of healthcare systems.

Conclusion to Part 1

This article has examined three of the four security testing categories required by IEC 81001-5-1: security requirements testing, threat mitigation testing, and vulnerability testing. These testing activities form a comprehensive approach to verifying that security controls have been properly implemented, function as intended, and protect against known threats and vulnerabilities.

In the next installment, we will explore penetration testing (Section 5.7.4), examine the requirements for tester independence (Section 5.7.5), and discuss practical implementation strategies for integrating these testing activities into the software development lifecycle. We will also address how manufacturers can balance the significant testing burden with resource constraints while maintaining appropriate security assurance.

[Continued in Part 2]