Hello BRIGHT Run Family,
Hope you are enjoying Spring.
I have a research update for you.
In my February letter, I provided an overview of Clinical Practice Guidelines (CPGs). CPGs are evidence based recommendations that support clinicians in diagnosing conditions and delivering appropriate treatment and care.
Evidence is not simply classified as “good” or “bad.” Instead, it is graded according to levels that indicate the strength and reliability of each scientific study. Higher quality evidence gives clinicians and policymakers greater confidence when developing recommendations pertinent to a scenario or a question.
The process to develop a CPG is based on a step called systematic review of relevant literature. In a systematic review, the researchers conduct a comprehensive search for all potentially eligible studies, and in the course of doing so, exclude those that are ineligible. Then they assess and rank the eligible evidence based on their quality and perform a synthesis of evidence to complete the systematic review.
A comprehensive search can result in the retrieval of thousands of studies, and they must be screened to include the applicable studies only. The process of screening is done in two stages. Stage 1 – title/abstract screening (using only the title and abstract to include or exclude a study) and Stage 2 – full-text screening (studies that are included in Stage 1 are read fully for final inclusion or exclusion). Both stages are time consuming, if done manually, and are shown to benefit from the usage of AI.
In this context, using AI helps to identify included articles quickly. This is similar to the situation where you are looking for a few important files in a stack of thousands of files (see an example picture). Then, AI works with you to bring the important files to the top of the stack so that you will not need to go through the entire stack to find those. That can save you time.
However, would the AI tool save time if the important files are very difficult to identify? Can the tool miss identifying the file i.e., fail to bring it to the top without you going through the entire stack? Getting back to the context of CPGs now, would the AI tool be challenged if the CPG is very complex and have multiple clinical questions?
My colleagues Dr. Xiaomei Yao, Dr. Jonathan Sussman, and I recently tried to answer those questions for Stage 1 screening using a complex CPG on axillary management in breast cancer. Our team also included a group of students, from McMaster, who devoted their time to conduct 70 tests crucial for our study.
Our finding is that the AI tool (Covidence, a very popular AI-enabled platform) is a promising one (i.e., it saves time) but needs to be used with care as it might miss included articles in 20 per cent of the tests. (The study is here: The performance of Covidence: An artificial intelligence-based tool for title and abstract screening in a breast cancer evidence-based clinical practice guideline – ScienceDirect). This shows that human oversight is needed while engaging this tool, based on the version we studied. More oversight may be needed for more complex tasks.
While we increasingly notice the engagement of AI-enabled tools in our daily lives, the verification and validation of the tools in our context remain important. We are hoping that these tools will get better over time.
Thinking about time reminds me that we are approaching the 19th BRIGHT Run!!!
Stay well.
Best,
Ashirbani
Dr. Ashirbani Saha is the first holder of the BRIGHT Run Breast Cancer Learning Health System Chair, a permanent research position established by the BRIGHT Run in partnership with McMaster University.
