The rise of in vitro toxicity studies has created a pressing need for a reliable way to judge their internal validity and potential sources of bias.
Through a two-round Delphi survey and expert discussions, the study narrowed down a large pool of 405 methodological concepts to those considered essential for evaluating study quality.
Experts reached consensus that 372 items should be retained, highlighting substantial agreement across literature sources and topic-guided discussions.
These validated items will shape the INVITES-IN appraisal tool, showing that many principles used to assess human and animal research also apply to in vitro studies.
Context
In vitro toxicity studies are increasingly being included as evidence in systematic reviews and chemical risk assessments. INVITES-IN, a tool for assessing the internal validity of in vitro studies, is under development in a process consisting of four consecutive studies. Study One in the creation of INVITES-IN was the development of an “item bank” database of 405 concepts (“items”) of potential relevance for assessing the internal validity of in vitro toxicity studies. The items were gathered from both focus group discussions and a purposive literature sample. In this paper we present the second study in the creation of INVITES-IN, i.e. the methods and results for identifying items for consideration when assessing the potential for bias in an in vitro study.
Method
A two-round digital Delphi survey, followed by online Delphi panel discussions guided by a moderator, was performed. The Delphi participants were experienced with both in vitro models and systematic review methods.
Results
Fifteen experts completed both Delphi rounds, and thirteen participated in a guided Delphi panel discussion. Of the 405 items in the bank, the experts agreed that 372 should be considered when assessing the potential for bias in an in vitro study. Items gathered from both the literature sample and the focus group discussions (Study One) were considered to be important for the assessment of the potential for bias in an in vitro study; 83–100% of the items collected from the literature sample were identified to be important and 91% (127) of the new items discovered in the focus group discussions of Study One were identified to be important.
Discussion
The 372 retained items will be interpreted into a manageable set of study appraisal criteria and a supporting guidance that will constitute the INVITES-IN study appraisal tool. In terms of lessons for tool development, the high retention of items included in tools designed for assessment of human and animal studies to in vitro studies suggests that many validity concepts are generally applicable across multiple study designs. Therefore, tool development processes should benefit from drawing on assessment tools outside the immediate domain of interest. Tool development would also likely benefit from supplementing literature reviews with focus group discussions, as our results demonstrate that the use of focus group discussions with domain experts was a pragmatic and valuable approach to increasing coverage of items in a tool development process.
Conclusion
In conclusion, this study demonstrates the value of using rigorous methods to ensure a comprehensive dataset as the starting point for creation of an assessment tool, though the direct application of Delphi methods to item banks may be an unnecessary step in tool development.