This transcript has been edited for clarity.
Hello, everyone. This is Dr Bishal Gyawali, from Queen’s University, Kingston, Canada. Today, I’m back with a new video in our series of Skills Lab: How to Interpret Clinical Research With Dr Bishal Gyawali.
In my last video, I discussed some background information to consider before you start reading a clinical research paper. Today, I’ll discuss where to start reading a paper. What’s the first section you usually see? The abstract.
The question is, do you always have to read the abstract? That’s an important question. When do you read the abstract? I don’t think you have to always read the abstract, but there are some occasions in which I read the abstract.
One is if it’s a new topic for you, to gauge whether or not you have an interest in that topic, and whether or not you want to read the full text. That’s when you read the abstract.
I don’t want anyone to make clinical decisions by reading the abstract alone. I discourage people from reading abstracts except on three occasions.
One is to get an idea about whether or not you want to read the whole paper. Before you spend time reading the whole paper, maybe you want to read the abstract to see if you are interested or if that is relevant to your topic.
Two, after you have read the full article and after you have made a scientific judgment about the article, then you can read the abstract to get some quick data points. What do I mean by that? For example, I see patients with colorectal cancer in my clinical practice and there is a new trial published. I have read the full article and I have made my own judgment about whether or not I should be using this drug. Let’s say I made an informed decision that this drug is going to be useful for some of my patients.
Now, I have a patient in front of me in the clinic, or maybe I’m going to see that patient tomorrow in the clinic, and I’m going to use that drug. In that case, I can use the abstract to get quick data points to answer the patient’s questions. For example, what would be my median survival if I take the drug vs I don’t take the drug? What would be the effect on my quality of life? What might be the major toxicities?
To answer these questions with quick data points, I will go back to the abstract because I have already read the full paper.
Third is for research purposes. I do a lot of meta-research where we gather data from different papers. In those cases, I read the abstract. I never make clinical or policy decisions based on reading the abstract alone.
I’ll give you a quick example of why I’m so critical about making decisions based on reading the abstract alone. One is because in some of the journals, conflict of interest information is so important, and they don’t provide that information within the abstract. You have to have access to the full text to see the conflict of interest of the authors.
That’s not the main point, though. The main point is there can be a large amount of spin in the abstract. For example, this is the conclusion of one of the abstracts from a phase 3 clinical trial of a new pancreatic cancer drug.
In the conclusion, you’ll see that there is a huge amount of spin. It says that although the hazard ratio was in favor of the drug, there was no statistically significant difference. The time from randomization to second disease progression or death showed a clear trend but was not alpha protected.
This is all jargon. If I were to summarize this conclusion in a single clear sentence, it would be that the drug failed to show any clinically meaningful benefit in patients. That would be the clear conclusion. But their conclusion has been spun in such a way that you will get an impression, if you read this conclusion from the abstract, that this was a really good drug — which is absolutely the opposite conclusion of the reality. Watch out for abstract conclusions, and don’t rely on them.
I have a suggestion about this for the journals. I think the conclusion of the abstract should not be written by the authors of the paper. Someone else should write this. An independent third party should be writing the conclusion of the abstract.
In fact, the Journal of Clinical Oncology, in their context section of original articles — and I’m going to give you an example here — has a section called “Relevance,” which is not written by the authors but by the associate editor of the journal who reviewed that paper. Similarly, I think it is important that journals have an independent expert write the conclusion of the abstract instead of the authors.
Talking about an independent expert writing the conclusion of the abstract, some journals publish editorials to accompany important trials. Those editorials can have a big influence on how we interpret the results of the trials. However, I encourage the audience to read those editorials only after you have read the paper for yourself.
Don’t trust the experts blindly. Don’t trust the majority opinion. In social media, for example, after a drug trial is published, there will be several tweets and people will have hot takes, but read the paper for yourself. There can be huge bias due to big names. You see these big names supporting the drug or saying things in support or against the drug, but don’t get influenced because of those big names. Beware of experts because experts don’t always give better advice; they just give more advice.
For example, in this trial, patients received a new drug vs placebo with the goal of improving muscle mass and muscle function for patients with cancer cachexia. As you can see, the results of the trial show that if you get the drug, then you gain some skeletal muscle mass — almost 1 kg — compared with losing almost 1 kg of muscle mass if you get placebo, which is statistically significant.
What is clinically meaningful for the patient? Does having 1 kg extra muscle mass mean that much more for the patients? No; it is the muscle function that is important. They looked into muscle function as well, and if you look at the data for muscle function in terms of hand grip strength, there is no difference.
These results mean that if you get the drug vs not getting the drug, you will have 1 kg extra skeletal muscle, but your ability to open the door or button up your shirt will not change. What sense does it make to get a drug that just improves your muscle mass but does not improve your muscle function?
That was my impression, but the conclusion of that paper was that this is a treatment option for patients with cancer cachexia. I was surprised. Why would they conclude like that? Based on my previous video, of course, I saw that the paper was written by the industry, as you can see here.
Then I was interested in the editorial, because the editorial is supposed to put things into context and provide correct inference. When I read the editorial, the editorial also said that this drug offers new hope for our patients. I was surprised. Why would the editorial also conclude in such a positive light?
Then I looked at the conflict of interest of the person who wrote the editorial, and I was surprised to find that the person has received fees from the exact same company whose drug was being discussed. The exact same company.
That got me intrigued, and we did a systematic study to see how prevalent this is and how it affects the interpretation. We published this paper in which we found that if the editorialist has received money from the same company, which we define as direct financial conflict, then the editorial is going to be unduly favorable for the product.
Our bottom-line expectation from journals or conference organizers is that at least the editorials for the journals and plenary discussions for the conferences, at a bare minimum, should not be receiving money from the same company whose drug is being discussed.
That means, don’t rely on the conclusion of the abstract. Don’t rely on editorials. There can be huge spin and bias. Interpret the data for yourself.
In the next video, I’ll go into detail about the introduction and methods.