Formative Evaluation and Summative Evaluation

Usability engineering for medical devices is a systematic process to ensure product ease of use and safety. Within this process, formative evaluation and summative evaluation play particularly important roles. Formative evaluation focuses on improvement during development, while summative evaluation handles final verification—they are complementary evaluation methods. Understanding these evaluation methods is essential for developing safer and more user-friendly medical devices.

Origins and Essence of Formative Evaluation

The concept of formative evaluation originated from the field of education. It was proposed by American educational psychologist Benjamin S. Bloom as part of his theory of “mastery learning,” aimed at enhancing each student’s learning achievement.

To understand this concept, consider the example of periodic tests in schools. The traditional evaluation method of checking learning comprehension only through final exams often results in a situation where, by the time insufficient understanding is discovered, the class has already moved on to the next unit, making effective improvement difficult. In contrast, formative evaluation—which involves administering daily quizzes to check comprehension—enables real-time understanding of students’ comprehension levels, allowing for immediate supplementary lessons or adjustments to teaching methods as needed.

Application to Medical Device Development

This educational approach has been applied to the medical device development process. In medical device development, formative evaluation refers to the continuous process of evaluation and improvement conducted before the product is completed. For example, when a prototype of a new medical device is created, actual users—healthcare professionals—are asked to trial it, and their feedback on usability and areas for improvement is collected. Based on this feedback, the design is refined, and the evaluation is conducted again. By repeating this cycle, the product evolves into a safer and more user-friendly device.

This process is similar to a chef repeatedly tasting and adjusting soup while cooking. If the chef cooks without tasting until the end, problems such as excessive saltiness or insufficient broth may arise. In the case of medical devices, if serious usability problems are discovered after completion, it could lead to severe situations affecting patient safety.

It should be noted that in the context of usability engineering for medical devices, formative evaluation is conducted in accordance with IEC 62366-1 (Medical devices—Part 1: Application of usability engineering to medical devices), which provides the framework for identifying use-related hazards, use errors, and critical tasks during the design and development phase. While the design verification activities required by ISO 13485 serve a similar purpose of confirming that design outputs meet design inputs, formative usability evaluation specifically focuses on the user interface evaluation to explore design strengths, weaknesses, and unanticipated use errors.

The Role of Summative Evaluation

Summative evaluation, on the other hand, is an evaluation conducted on the final product after development is completed. Like a final exam, it is an important process to confirm that all established requirements are reliably met. For instance, it objectively verifies whether regulatory requirements for safety are satisfied, whether expected performance is reliably demonstrated, and whether intended users can use the product safely, based on objective data.

In the context of usability engineering, summative evaluation is conducted in accordance with IEC 62366-1, which defines it as “user interface evaluation conducted at the end of the user interface development to obtain objective evidence that the user interface can be used safely.” The FDA requires human factors validation testing (which is synonymous with summative evaluation) to be conducted with at least 15 participants per user group for medical device submissions. The summative evaluation must demonstrate that use-related risks are acceptable and that the user interface can be used safely under realistic conditions.

While summative evaluation in usability engineering corresponds functionally to design validation activities required by ISO 13485 (confirming that the product meets user needs and intended use under actual or simulated use conditions), it specifically focuses on obtaining objective evidence of safe use through evaluation of hazard-related use scenarios.

Relationship Between the Two

Formative evaluation and summative evaluation complement each other. By discovering and improving problems during the development process through formative evaluation, the pass rate in the final summative evaluation can be increased. Additionally, issues discovered in summative evaluation are utilized as perspectives for formative evaluation in the development of next-generation products.

Comparison of Formative and Summative Evaluation

Aspect	Formative Evaluation	Summative Evaluation
Timing	During design and development	At the end of user interface development
Purpose	Explore design strengths, weaknesses, and unanticipated use errors; guide improvement	Obtain objective evidence that the user interface can be used safely
Regulatory Framework	IEC 62366-1 (formative evaluation); similar to ISO 13485 design verification in principle	IEC 62366-1 (summative evaluation); similar to ISO 13485 design validation in principle
Methods	Think-aloud protocol, heuristic evaluation, cognitive walkthrough, iterative testing with small numbers (1-10 participants)	Usability testing under simulated use conditions with representative users (FDA requires minimum 15 per user group)
Frequency	Multiple iterations throughout development	Typically once at the end, covering hazard-related use scenarios
Documentation	Results inform design changes; identify new hazards and use errors	Formal validation report with objective evidence; required for regulatory submissions
Participant Involvement	Encouraged to think aloud and provide feedback	Instructed to perform tasks without excessive communication to simulate realistic use
Regulatory Requirement	Not mandatory but strongly recommended by FDA and IEC 62366-1	Mandatory for demonstrating safe use and regulatory compliance

Toward Future Usability Engineering

In recent years, as medical devices become more sophisticated and complex, the importance of formative evaluation in usability engineering has increased. By incorporating the user’s perspective from early stages of development and conducting continuous improvement, safer and more effective medical devices can be developed efficiently. This approach of “evaluating while creating and creating while evaluating” is the core of usability engineering based on human-centered design principles, and has become an indispensable concept in future medical device development.

In usability engineering for medical devices, by appropriately combining formative evaluation and summative evaluation in accordance with IEC 62366-1 and other applicable standards, products that meet user needs and satisfy safety requirements can be efficiently developed. This evaluation process is expected to further develop as an important initiative toward realizing a better healthcare environment.

Key Regulatory References:

• IEC 62366-1:2015+AMD1:2020 – Medical devices—Part 1: Application of usability engineering to medical devices
• ISO 13485:2016 – Medical devices—Quality management systems—Requirements for regulatory purposes
• ISO 14971:2019 – Medical devices—Application of risk management to medical devices
• FDA Guidance: Applying Human Factors and Usability Engineering to Medical Devices (2016)