Challenges in Electronic Record Retention Requirements: Maintaining Authenticity During Long-term Preservation and System Migration

Introduction to Preservation Requirements

Section 3.1.3.(2) of the Japanese ER/ES Guidelines (Ministry of Health, Labour and Welfare Notification No. 0401022, April 1, 2005, “Regarding the Use of Electronic Records and Electronic Signatures in Applications for Approval or Permission of Pharmaceuticals, etc.”) contains the following requirement:

(2) When preserved electronic records are migrated to other electronic record media or formats, authenticity, legibility, and retention must be ensured for the electronic records after migration.

It should be noted that the revised “ER/ES Guidelines Commentary” was published by the Japan Association of Clinical Research Organizations in March 2024, requiring responses based on the latest technological trends, including the widespread adoption of cloud services.

Similar requirements are specified in FDA 21 CFR Part 11, but long-term preservation of electronic records presents technically challenging issues. Surprisingly, the challenges of record retention are not sufficiently understood in many cases, so I will provide a detailed explanation here.

Challenges in Metadata Inheritance

Consider, for example, the case where an electronic document is created in Microsoft Word. Metadata such as author, creation date/time, modifier, and modification date/time are recorded as property information in the Word file.

However, when converting these Word documents to PDF, depending on the conversion method, this metadata may not be completely inherited. While standard PDF conversion preserves basic metadata, detailed workflow information managed by Document Management Systems (DMS) or document management systems (approver, approval date/time, review history, etc.) is stored separately in relational databases and can be lost during simple conversion.

In other words, there is a risk that important information necessary to ensure authenticity may be lost during file format migration. The inheritance of the following information is particularly challenging:

• Author and creation date/time
• Modifier and modification date/time
• Approver and approval date/time
• Review history
• Version control information
• Audit trails

Complexity of Migration in Document Management Systems

In document management systems, Quality Management Systems (QMS), and other enterprise systems, this metadata and audit trails are recorded separately from the document files themselves in relational databases or metadata storage. While this structure enhances system efficiency and searchability, it poses significant challenges during system migration.

When replacing (upgrading) a system or migrating to a different system platform, it is necessary to properly migrate this metadata and audit trails to the new system and correctly associate them with the document bodies. This task is technically complex for the following reasons:

Differences in Data Structures Between Systems: Because the database schemas differ between old and new systems, data mapping and transformation are required.

Continuity of Audit Trails: Continuity of audit trails must be maintained before and after migration, completely preserving the history of who did what and when.

Ensuring Data Integrity: In recent years, regulatory authorities have strongly demanded data integrity based on ALCOA+ principles (Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, Available). These requirements must also be met during system migration.

Even if a system complies with FDA 21 CFR Part 11 or Japanese ER/ES Guidelines, compliance with regulatory requirements will be lost if authenticity cannot be ensured during system replacement.

Challenges in Disaster Recovery

Similar problems arise when systems must be reconstructed due to disasters such as earthquakes, fires, or floods.

If electronic records must be recovered manually in an emergency, there is a risk that important information such as audit trails and metadata may be lost. To prevent such situations, it is essential to establish a system that can properly execute recovery from backups.

This is why backup is positioned as an important requirement for authenticity in both the ER/ES Guidelines and Part 11. Backup does not simply mean data copying, but rather the preservation of complete records including metadata, audit trails, and electronic signature information.

Regulatory Authority Concerns

Regulatory authorities take an extremely cautious stance regarding operations without audit trails that may occur during system migration, migration, or disaster recovery during the retention period.

The reason is clear: if data falsification or fabrication occurs during these migration operations, verification becomes impossible if audit trails are not properly recorded. The following points are particularly emphasized:

• Documentation and validation of migration plans
• Data integrity verification before and after migration
• Audit trails of the entire migration process
• Clarification of migration responsibility and approval processes
• Establishment of rollback (reversion) plans

In recent years, the PIC/S (Pharmaceutical Inspection Convention and Pharmaceutical Inspection Co-operation Scheme) data integrity guidance has also emphasized ensuring data integrity during system migration.

Practical Countermeasures

To address retention challenges, the following practical measures are recommended:

Development of Migration Plans: From the planning stage of system migration, design and document methods for complete migration of data and metadata. It is important to implement validation of migration procedures as part of Computer System Validation (CSV).

Independent Management of Metadata: Establish mechanisms to store and manage important metadata and audit trails in ways that do not depend on file formats. For example, there are methods to output metadata in standard formats such as XML or JSON and store them with documents.

Establishment of Migration Verification Processes: Establish processes to strictly verify data integrity before and after migration. In addition to sample verification, full verification using statistical methods or hash value comparison should also be considered.

Strengthening Backup Strategies: Implement regular and comprehensive backups that capture not only data but also system configurations, metadata, and audit trails. Additionally, regularly conduct backup recovery tests to confirm that actual recovery is possible.

Implementation of Education and Training: Provide continuous education and training to system administrators and data owners on the importance of retention requirements and procedures for addressing them.

Application of Risk-Based Approaches: Rather than applying the same level of measures to all data, determine the priority of measures based on risk according to data importance and regulatory significance.

Summary and Observations

Among the three requirements of authenticity, legibility, and retention, the author considers retention to be the most challenging to address.

The reason is that ensuring retention requires a long-term perspective and continuous efforts, and preparation for highly uncertain factors such as system updates due to technological evolution and responses to unexpected disasters is necessary.

However, with proper planning and implementation, it is possible to address these challenges. What is particularly important is to design with retention in mind from the initial stages of system implementation and to continue regular reviews and improvements.

From a data integrity perspective, establishing a management system that satisfies ALCOA+ principles throughout the entire lifecycle of electronic records will become increasingly important in the future.

Comparison Table: Key Differences in Preservation Requirements

Aspect	Traditional (Paper)	Electronic Records	Challenge Level
Physical Degradation	Predictable, visible	Media degradation, format obsolescence	High
Metadata Preservation	Manually attached	Separately managed in databases	Very High
Migration Complexity	Simple copying	Requires validation, data mapping	Critical
Audit Trail Continuity	Physical stamps/signatures	Digital, system-dependent	High
Disaster Recovery	Original documents needed	Backup and restoration required	High
Regulatory Scrutiny	Moderate	Intensive (ALCOA+ compliance)	Very High

Key Regulatory References

• Japanese ER/ES Guidelines (2005, updated 2024)
• FDA 21 CFR Part 11
• PIC/S Data Integrity Guidance (PI 041-1)
• ALCOA+ Principles (Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, Available)
• ICH Q9 (Quality Risk Management)
• ISO/IEC 27001 (Information Security Management)