- Developing tools to evaluate both internal and external validity of NAMs, ensuring inclusion of high-quality evidence in risk assessments based on the weight of evidence principle
- Focus on relevance to human adverse health effects, ensuring that hazard assessments are objective, robust, transparent, and reproducible
- Internal validity tools will assess potential biases introduced by study design, conduct, reporting, or analysis, while external validity tools will evaluate the translatability of study data to human health effects, enhancing the reliability and applicability of risk assessment outcomes
Key messages
- It is critical that incorporation of New Approach Methodologies (NAMs ↗) data in chemical risk assessments is transparent and follows structured processes to make it possible for decision-makers to use assessments based on such data in their work.
- Study validity assessment constitutes a retrospective validation of existing data.
- The aim of this project is to create validity assessment tools for in vitro studies.
Overview
Implementation of systematic review (SR)/evidence-based principles in chemical risk assessments (CRAs ↗) increases the potential for rigour, objective, and transparent CRA conclusions, which is essential when summarising evidence for decision-making. Study validity assessment is a critical part of the SR principles. Study validity assessment constitutes a retrospective validation of existing data. It can be applied on both guideline and non-guideline studies. The aim of this project is to create validity assessment tools for in vitro studies that can be applied across chemical classes, sectors, and regulations.
We are creating a tool for the assessment of internal validity and a tool for the assessment of external validity. The tools for assessment of internal and external validity of in vitro studies will provide structured approaches for validity assessment. They will i) support objectivity, transparency and openness in the process of summarizing evidence for decision making, ii) support the incorporation of evidence from in vitro data in CRAs, and pave the way for the use of such data as stand-alone evidence (both guideline and non-guideline studies), iii) contribute to confidence in the use of NAMs data as evidence in chemical risk assessments, and iv) ensure that sufficient information about the quality of the evidence are available for the decision makers.
Another aim of this project is to increase the awareness, common understanding and knowledge on validity assessment in the CRA community, and to create online training materials.
Achievements & Results
- Establishment of project group roles and tasks, ensuring clarity and alignment among participants in systematic review principles, chemical risk assessment, toxicology, NAMs, and tool development.
- Establish a scientific advisory group with experts providing strategic guidance and support, ensuring synergy with related projects and avoiding duplication of efforts.
- Two protocols for the creation and user testing of the tool for evaluation of internal validity are published. Of the four studies that will be performed to create the tool, the first study is completes and published, the second study is completed, the manuscript is submitted, and the third study is ongoing.
- The protocol for the terminology project is published, and the study is ongoing.
- For the tool for evaluation of external validity, a manuscript describing the meaning of external validity for in vitro studies has been drafted.
Policy relevance
The tools will be designed to be applied across chemical classes, sectors, and regulations. They will i) support objectivity, transparency and openness in the process of summarising evidence for decision making, ii) support the incorporation of evidence from in vitro data in chemical risk assessments, and pave the way for the use of such data as stand-alone evidence (both guideline and non-guideline studies), iii) contribute to confidence in the use of NAMs data as evidence in CRAs, and iv) ensure that sufficient information about the quality of the evidence are available for the decision makers.
Related Publications
Harmonising the language of systematic review and chemical risk assessment improves transparency, fosters collaboration, and ensures that EU chemical regulations are aligned with public health protection goals.
Integrating systematic review methods into chemical risk assessments addresses regulatory gaps, strengthens evidence-based decisions, and enhances EU-wide policy consistency.
Clear definitions and communication of risk-related concepts empower policymakers to engage stakeholders and promote uniform compliance across Member States.
By identifying conceptual overlaps between systematic reviews and chemical risk assessments, this approach strengthens the scientific basis for regulatory decisions and risk communication.
Using cross-mapped terminologies between systematic reviews and chemical risk assessments enables standardisation and ensures clear, actionable guidance for consistent regulatory enforcement across the EU.
The focus on implementation of systematic review (SR) principles in chemical risk assessments (CRAs) is growing as it has the potential to advance the rigour and transparency of the CRAs. However, the SR and CRA communities use their own specific terminologies. Understanding the meaning of core SR and CRA terms and where they overlap is critical for application of SR methods and principles in CRAs. Moreover, it will increase the possibility for cross-sectorial collaboration, avoid misunderstandings, and improve communication among risk assessors, researchers, and policy makers.
We present a process for the cross-mapping of core CRA terms and core SR terms. Core terms for study appraisal, evidence synthesis and integration used in the SR and CRA communities will be included. The outcome will be an overview of how core SR terms map onto core CRA terms and vice versa, and a description of the relationship and conceptual overlap between the terms.
The cross-mapping is divided in three phases, where in the first phase the core SR and CRA terms will be identified. In the second phase, existing SR and CRA definitions will be mapped. In the third phase, descriptions of the relationship and conceptual overlap between the terms will be derived. The third phase will include weekly one-hour online meetings for SR and CRA experts.
A tool for evaluation of internal validity of in vitro studies called INVITES-IN is currently under development. The tool is designed specifically for cell culture studies.
This protocol describes the testing of the performance of INVITES-IN.
Data collected from the performance testing will be used for further refinement and development of the release version of INVITES-IN.
A tool for validity assessment of cell culture studies has been created as a part of the hazard assessment to be able to differentiate different studies according to validity.
A tool for evaluation of internal validity of in vitro studies called INVITES-IN is currently under development. The tool is designed specifically for cell culture studies.
This protocol describes the testing of the performance of INVITES-IN. By performance, we mean the extent to which results of using INVITES-IN are the same for different users (consistency), the amount of time and cognitive effort it takes to apply INVITES-IN (assessor workload), the precision and potential for systematic error in results of applying INVITES-IN (accuracy), and how easy it is to use INVITES-IN (user experience).
The participants in the user testing will be representative for the expected end-users of INVITES-IN which are persons preparing literature reviews including in vitro studies (e.g. in the context of chemical hazard and risk assessments or drug development). All end-users are expected to have experience with in vitro methods.
Data collected from the performance testing will be used for further refinement and development of the release version of INVITES-IN.
This protocol describes the design and development of a tool for evaluation of the internal validity of in vitro studies, which is needed to include the data as evidence in systematic reviews and chemical risk assessments.
The tool will be designed specifically to be applied to cell culture studies, including, but not restricted to, studies meeting the new approach methodology (NAM) definition.
This protocol describes the design and development of a tool for evaluation of the internal validity of in vitro studies, which is needed to include the data as evidence in systematic reviews and chemical risk assessments. The tool will be designed specifically to be applied to cell culture studies, including, but not restricted to, studies meeting the new approach methodology (NAM) definition. The tool is called INVITES-IN (IN VITro Experimental Studies INternal validity).
In this protocol, three of the four studies that will be performed to create the release version of INVITES-IN are described. In the first study, evaluation of existing assessment tools will be combined with focus group discussions to identify how characteristics of the design or conduct of an in vitro study can affect its internal validity. Bias domains and items considered to be of relevance for in
vitro studies will be identified. In the second study, group agreement on internal validity domains and items of importance for in vitro studies will be identified via a modified Delphi methodology. In the third study, the draft version of the tool will be created, based on the data on relevance and importance of bias domains and items collected in Studies 1 and 2. A separate protocol will be prepared for the fourth study, which includes the user testing and validation of the tool, and collection of users’ experience.