I am presently reading through some interesting articles on the SBM site which seem to me to be relevat to this topic. There seems to be something I do not understand or am confused about which is probably why I seem to be ignoring some of your questions.
Which is why I said in my last post:

“Okay, at this stage I’m just thinking out loud and don’t expect a reply to this post. Maybe I need to keep my views on the back burner and see how it pans out as further information comes my way.”

I will come back later to see if I can make sense of all this though I probably won’t add a comment because old posts quickly go off the radar around here.
(It’s a pity we can’t get email notifications that bring us back to where we left off )

I’m starting to wonder whether you are arguing in good faith here because you didn’t read/have ignored my comment and you have quote mined the hell out of it and then thrown it up as a strawman.

There are very important pieces of info in studies that do not qualify for MAs. They are not RCTs. But then RCTs are not able to answer all clinically relevant questions (go back and read the bit about natural history). There are different EBM hierarchies for different clinical questions. In terms of treatment effectiveness the RCT and subsequent MS’s are supreme, as you keep correctly asserting to the wrong questions. But if the patients in the MA don’t match your patient then you still have to apply to a wider outlook.

Licensing would add a 4th or 5th layer of peer review. Not the first layer. It would be about the 3rd layer after initial design and possibly funding. And it is already what the FDA sometimes do. And you won’t explain what the marginal benefit of this would be. Plus the FDA can’t have the answer to a question that has just been posed by having a trial ‘licensed’ previously to answer a different question through you hypothetical 20/20 foresight licensing program.

Okay, at this stage I’m just thinking out loud and don’t expect a reply to this post. Maybe I need to keep my views on the back burner and see how it pans out as further information comes my way.

“You have to be able to critically evaluate your evidence when you are applying it to an individual patient right now. If the MA is based on crappy RCTs or isn’t quite relevant to your patient then it is not at the top of the evidence hierarchy and you’ll need to look for other flavours of evidence.”

Okay, I understand that he is probably criticising the quality of many MAs extending back into the past (hopefully, as you imply, this is no longer the case) rather than criticising MAs as a concept.

“But there are studies that are ignored by MA that are of immediate clinical utility and shouldn’t be ignored by a practicing clinician.”

So, what’s going on here. Are there no guidelines for what should be included? Is it just arbitrary and up to the person performing the MA to decide what should be included? Surely the MA should include all RCT without methodological flaws sufficient to invalidate the conclusions

“Cohort studies [of treated and untreated patients are] also important”

But as accurate as a methodologically sound RCT?

“To answer your question I don’t see a role for licensing of trials as this simply introduces a fourth or fifth layer of peer-review. The marginal cost-benefit of this is unclear to me.”

It would be the first layer, possibly making many of the other layers unnecessary. Surely it must be a good thing to start with methodoloigally sound (as far as that is possible) RCTs and go from there, rather than have to trawl through and evaluate each and every one of them for inclusion of your MA.

“FDA in some cases may already do something similar when it wants additiona info from a company before licensing a drug/implantable device.”

But it would already have the answer if the preceding trials were licensed

“You have to be able to critically evaluate your evidence when you are applying it to an individual patient right now. If the MA is based on crappy RCTs or isn’t quite relevant to your patient then it is not at the top of the evidence hierarchy and you’ll need to look for other flavours of evidence.”

It’s 1416 articles. Not necessarily studies. Not necessaily about this drug or about this disease. Not necessarily studies of effectiveness of any type. Not necessarily studies of humans even. Many studies of the pharmacodynamics etc. Not that many actual clinical trials. This is completely normal when doing a systematic review and subsequent MA. But there are studies that are ignored by MA that are of immediate clinical utility and shouldn’t be ignored by a practicing clinician.

Cohort studies in humans exposed to the drug are very important for clinicians to guage side-effects, for instance. They won’t be in an MA but they are still valuable decision making aids. Cohort studies in untreated patients give you the natural history (i.e. the harm in not treating)- also important, also not in an MA.

To answer your question I don’t see a role for licensing of trials as this simply introduces a fourth or fifth layer of peer-review. The marginal cost-benefit of this is unclear to me. Registration, on the other hand, I am a staunch defender of.

FDA in some cases may already do something similar when it wants additiona info from a company before licensing a drug/implantable device.

“They screened 1416 articles and ended up with 29 studies, 10 for effectiveness, the cream of the crop. 1416 articles is a lot of articles. I know they are not the crème de la crème, and is why I am not a fan on just relying on meta-analysis alone. That is a lot of ignored information.”

Why would you not ignore junk?
(And his third sentence confuses me no end! In fact that whole paragraph confuses me )

I suspect that maybe Mark is saying the same as I am, but he has not said so and in the article he comes across as demoting MA. I think he should have qualified his commentary somewhat to lessen this impression.

Okay I understand it can be messy, but is there really any excuse anymore for neglecting basics such as random allocation, double blinding, using a proper placebo control, using an N that has at least a chance of producing a reliable result etc.
Do you see any role at all for *licencing* trials?

Nothing is wrong with “prior probability”. Sorry for giving that5 impression. Zero PP probably means you shouldn’t bother subjecting the treatment to a clinical trial (unless the treatment is already in widespread use). A low PP means the evidence from your clinical trials must be correspondingly high (you’re mot going to overturmn science with a marginal result).

There were not 1416 trials. 1416 articles were identified by the systematic search. The point of these searches is to make them sensitive enough to pick up every one of the actual trials. What they mostly pick up is junk that’s not relevant and you have to sift out the good ones. When you do this stuff in the real world you get lots of editorials, case reports, news reports, reviews, opinion pieces, cohorts etc etc. You then pull the relevant RCTs out and meta-analyse them.

“According to Ben Goldacre, by any measure, it does not cost any more to do a trial properly. Do you disagree?”
A crappy trial is often less expensive than a really good trial. So Ben is sort of right. As much as I respect Ben’s opinions I’m not sure Ben has ever run a trial. But once again it’s not like these are getting screwed up on purpose. Hindsight being 20/20 and all that…

“It should be pretty straight forward for medical treatments surely. ”
Like I said previously. No, it’s not. The real world is messy. This stuff is actually hard to do. I imagine it’s much harder to do in an infectious disease setting given natural history and diagnostic issues.

And I think Mark and you are actually agreeing with one another. As with all analytic techniques meta-analysis is GIGO. You have to be able to critically evaluate your evidence when you are applying it to an individual patient right now. If the MA is based on crappy RCTs or isn’t quite relevant to your patient then it is not at the top of the evidence hierarchy and you’ll need to look for other flavours of evidence.

And what’s wrong with using prior probability in medicine?

“In order to get the money in the first place all such trials undergo extensive consultation and review. Much the same as anything humans do this sometimes doesn’t work out as well as it should. It’s not like we want to waste years of our lives on screwing things up.”

Out of 1416 trials only 29 were found to suitable for inclusion in a meta-analysis. I’m not sure how much “consultation and review” went into those 1416 trials, but it was definitely an enormous waste in anyone’s language.

“So basically your telling clinical triallists to do a better job. We’re trying- this stuff is actually difficult and sometimes the science can be less than ideal because of the logistics and reality intruding.”

According to Ben Goldacre, by any measure, it does not cost any more to do a trial properly. Do you disagree? It should be pretty straight forward for medical treatments surely.

“The mindset changes when you’re like Dr Crislip. You have a patient in front of you and you don’t necessarily have the perfect science to back up your treatment decisions. You have to make the best of what you’ve got, for that patient, at that time-point. This is the art and science of medicine and it’s not necessarily a ‘bad thing’.”

That may be the case, but that is not the way he has portrayed meta-analysis. He simply stated that they are one of many ways to assess rthe uselfullness of a therapy, and that 40% have been found to be unreliable. I had to go to the source to discover that these were metanalyses of (most likely) methodologically flawed RCTs. I still don’t know whether he still distusts meta-analyses as a concept or whether he accepts that those he referred to were flawed and hence not a true reflection of the concept of meta-analysis.

In fact, meta-analysis stands at the pinacle of SBM with RCTs close behind. But you have to hit the ball properly otherwise it will end up in the bunker instead of on the green. All the other criteria he mentions are merely assessments of “prior probability” in my opinion.

regards,
BillyJoe

In order to get the money in the first place all such trials undergo extensive consultation and review. Much the same as anything humans do this sometimes doesn’t work out as well as it should. It’s not like we want to waste years of our lives on screwing things up.

Meta-analysis is always going to be garbage in garbage out (GIGO).

So basically your telling clinical triallists to do a better job. We’re trying- this stuff is actually difficult and sometimes the science can be less than ideal because of the logistics and reality intruding. We’re not running mice around a cage. It’s real people, in the real world, running wild, and it gets messy.

The mindset changes when you’re like Dr Crislip. You have a patient in front of you and you don’t necessarily have the perfect science to back up your treatment decisions. You have to make the best of what you’ve got, for that patient, at that time-point. This is the art and science of medicine and it’s not necessarily a ‘bad thing’.

In the meantime guys like me keep trying to provide guys like Mark with better info.

Just one point which is sticking out like a sore thumb for me, re: the reliability of Cochrane reviews. You say of changing their conclusions;

‘perhaps. but I brings into question the result of their other meta’s’

This sounds very close to arguments along the lines of ’science has been wrong before’.

“You are largely correct except they are talking about meta-analysis before the large RCTs being reliable estimates or not. ”

Except that Mark Crislip is using that as an argument against meta-analysis. Of course meta-analyses are not going to be reliable if they are based on flawed RCTs. The solution is to perform sound RCTs. Those, and the meta-analyses based on them, are really the only reliable way to obtain evidence for the effectiveness of medical treatments.
All the other Hills criteria are merely a way of assessing “prior probability”and whether or not a treatment is worth the time, manpower, and expense of subjecting to a clinical trial.

“Mega trials are expensive to run afterall- and you have to wait many years for the outcome. And yes the subsequent meta-analyses of the mega trials wil be even more accurate. But again you have to spend mega bucks and wait many many years to know this.”

Yes, I understand: The bigger the trial the more reliable the result and the longer it takes to run that trial. On the other hand, a methodologically sound trial is no more expensive to run than a flawed trial. And a meta-nanlysis of a series of small but sound clinical trials would be equivalent to one large RCT.

“Secondly clinical trial registration became largely compulsory years ago. There are now a number of these world-wide. You’re not living in a fantasy world but you could google some of this stuff and rest assured that what you’re thinking is already the gold standard of behaviour in trials.”

I was referring to the *licencing* of trials by a panel of experts who would not grant the license unless the submitted trial protocol was methodologically sound (and, if the trial isn’t licenced it doesn’t get published). They would also have an educational role. This has never been done as far as I know.

You are largely correct except they are talking about meta-analysis before the large RCTs being reliable estimates or not. Mega trials are expensive to run afterall- and you have to wait many years for the outcome. And yes the subsequent meta-analyses of the mega trials wil be even more accurate. But again you have to spend mega bucks and wait many many years to know this.

Secondly clinical trial registration became largely compulsory years ago. There are now a number of these world-wide. You’re not living in a fantasy world but you could google some of this stuff and rest assured that what you’re thinking is already the gold standard of behaviour in trials.

I have read your article on Hills Criteria, but I have not yet read the comments so you may have already answered the following questions.

Who can argue with those criteria (except that “Plausibility” and “Coherence” seem to be the same thing).

However, regarding the question of the reliability of meta-analyses, you referenced the following study:
http://nejm.highwire.org/cgi/content/full/337/8/536#F1
This study compares large RCTs to the results of previous Meta-analyses.

I have to say I get quite a different message from the one you got.
If you look at this chart:

http://nejm.highwire.org/cgi/content/full/337/8/536/F1

The number of patients in the *large RCTs* mostly outnumber the total number of patients in the *meta-analyses*. And, since a meta-analysis is effectively a large RCT, they are effectively comparing larger *large RCTs* with smaller *large RCTs* (the smaller *meta-analyses*). All things being equal, you would EXPECT the *large RCTs* to be more accurate than the smaller *meta-analysis* (or at least to show significant differences from them)

Also (and I haven’t read it in detail), I would assume that the *large RCTs* were chosen for methodological soundness. Am I right? Whereas the preceding meta-analyses would very likely contain small RCTs including some with at least some methodological flaws (if only because they were performed longer ago when there was less scrutiny).

In summary, what I get from this is that RCTs need to be large and methodologically sound. If this is the case, the subsequent meta-analyses the are done on them will inevitably be similarly methodologically sound, but larger by at least an oder of magnitude and hence more reliable by virtue of having more reliably eliminated the effect of a chance result.

In other words, the take home message for me is that we should be doing methodologically sound large RCTs.
If we do, then the Meta-analyses we subsequently do on them will be even more reliable than the individual large RCTs.

Hence the suggestion in my last post of registering and licensing all clinical trials, the penalty of not doing so being the inability to get your trial published.
Maybe I’m living in a fantasy world though.

regards,
BillyJoe

“If the question is efficacy, the challenge studies are information in support”

The drug is going to be used in the general population, so surely we need to see how it performs out there. It could turn out that the drug is effective only if given immediately after exposure which would make it practically useless.

“The point is that there is more data to consider, as a clinician, to the results of a meta, which are wrong 35% of the time”

This must mean there is a way to assess a meta study that is more reliable than the meta study itself. I suppose I was asking what that method is.

“what is routine?”

Something that is done “routinely”, meaning not just for special
groups. At least that is my understanding. What is the “accepted” definition of “routine”.

“a failing on my part, I do not understand cost benefit analysis methodologies enough to be able to critique them”

Nevertheless a very important consideration in the context of recommending or not recommending the stockpiling of the drug (I know your article was about effectiveness, not whether it should be stockpiled)

“the point is that meta’s are limited in their applicability.”

I guess I was asking: in what way are meta’s limited in their applicability and, if so, what other method outperforms them?

“It is nice to have RCT, and drug companies are weasels”

Yes, 8 out of the10 studies were never released even on repeated request over a period of time by several different professionals. Yet they are supposed to show much better results than the two that were published – they refuse to publish the trials showing the better results! That’s hard to believe. And it was these results on which the decision to stockpile the drug were made.

“meta are a part of the argument, see the post on hills criteria.”

I see it is one of yours from a couple of weeks back – must have missed it by a couple of days! I’ll have a look as soon as I get the chance.

“meta’s are seen by many as a be all end all, esp the cochrane, reviews. They are just another, potentially flawed, mechanism to get at the ever changing ‘truth’ of how best to treat patients.”

Okay I need to read that post, but as a first pass:

My understanding is that RCTs were originally set up because it was realised that personal experience was far too unreliable a method on which to base decisions about investigations and treatments. Unfortunately, it seems that most who do RCTs actually do not know how to do them properly and, as a result most RCTs are too methodologicaaly flawed to provide the answer to the question being asked of them. Hence the meta-analysis (or systematic reviews). Obviously there could just as easily be flawed systematic reviews. For example they might include methodologically flawed RCTs, or they might have too rigid exclusion criteria resulting in the exclusion of trials that are sound. I suppose the result would also be affected by the “Bottom Drawer” effect.

However, it seems to me that the solution is to improve the quality of the systematic reviews and to *register* clinical trials (not sure if that is happening universally as yet).

Another solution that I have posted elsewhere, but that has never gotten any response from fellow posters, is that when trials are registered (a trial that is not registered doesn’t get published) they should also be *licensed* by an expert panel. The expert panels job would be twofold: to reject any trials with methodological flaws and to educate the researchers on the proper methodology. As with registration, any trial that is not *licensed* does not get published. Same for systematic reviews.
The total number of trials of tamiflu is over 1400, but only about 30 were fit to be included in the systematic review. That is an enormous waste. Surely the registration and licensing of clincal trials and systematic reviews would be a far better use of limited resources.

regards,
BillyJoe

https://wcd.coe.int/ViewDoc.jsp?id=1574973&Site=DC&BackColorInternet=F5CA75&BackColorIntranet=F5CA75&BackColorLogged=A9BACE

The COE was founded in 1949 by a group of individuals who also founded the European Coal and Steel Community (ESCS) and the European Economic Community (EEC), precursors to the EU. The COE has 47 members and was founded in 1949. The EU has 27 members and was formally established in 1993. The COE and EU complement each other and collaborate on a number of joint programs. Among other things, the organizations share the same flag, originated by COE.

What’s the relationship of COE to pharmaceuticals and vaccines? The European Directorate for the Quality of Medicines and Healthcare (EDQM) and the European Pharmacopoeia operate under the auspices of the Council of Europe.

http://www.edqm.eu/en/Homepage-628.html
http://www.coe.int/

If the question is efficacy, the challnge studies are information in support

“They screened 1416 articles and ended up with 29 studies, 10 for effectiveness, the cream of the crop. 1416 articles is a lot of articles.”
A very confusing paragraph for me: Does “cream of the crop” refer to the 29 studies or the 10? When you say “10 for effectiveness”, what were the other 19 trials testing for? Complications? Side-effects?

yes, complicaitons and side effects

“I know they are not the crème de la crème, and is why I am not a fan on just relying on meta-analysis alone. That is a lot of ignored information.”
You said that the 29 studies are the “cream of the crop” so what does “they are not the crème de la crème” refer to?
And why should you not ignore the 1387 studies that do not meet the criteria of methodolgical robustness and are therefore eliminated from the meta-analysis. Does that not increase the reliability of your results?

The point is that there is more data to consider, as a clinician, to the results of a meta, which are wrong 35% of the time

“The Cochran review states, “Because of the moderate effectiveness of neuraminidase inhibitors, we believe they should not be used in routine control of seasonal influenza.” The first half of statement is a fact, the second half is opinion…It is not a ‘we’ that I would use for deciding medical treatment. It is one thing to say that in healthy people from age 14 to 65 treatment is modestly effective, quite another to extrapolate that information to everyone, young and old, healthy and ill, pregnant or not.”
When the Cochrane reviewers states that they believe they should not be used in “routine” control of seasonal influenza, the use of the word “routiine” means they are not referring to their use in *special* cases – like the ones you mentioned, for example. It seems to me that are not actually disagreeing with you.

what is routine? A vague recommendation by 4 people.

“Whether olsetamivir is worth the cost and the breeding of resistance requires a complicated cost-effective analysis that I will never understand.”
Do you mean that it is a failing on your part, or do you mean that a cost/benefit analysis is too meaningless to be of any value?

a failing on my part, I do not understand cost benefit analysis methodologies enough to be able to critique them

“One of things I have learned in blogging and podcasting is how limited and unimpressive meta-analysis and structured reviews are. They do pool the best studies. But often it seems that the process of choosing the studies, much important and relevant information is not considered.”

I just don’t understand this. They “pool the best studies” but they “leave out important and relevant information”. What can that mean? Are you just disagreeing about the criteria for choosing the “best studies”, or do you think that studies with methodological flaws can still teach us something. If so, which ones and what can they teach us?

the point is that meta’s are limited in their applicability.

“Most of infectious diseases is not based on randomized, placebo controlled trials and does not need to be.”
Surely when Oseltemivir was first developed RCTs would have been essential to establish efficacy. The fact that the company hid 8 of 10 studies should surely raise a red flag.

It is nice to have RCT, and drug companies are weasels

“It is why you need to consider the entire medical literature.”
But surely a meta-analysis/systematic review does just that – it reviews all the trials and appropriately rejects the those with methodological flaws sufficient to invalidate them. If you think their criteria are too stringent and that they exclude worthwhile, you need to explain what your criteria would be and why you disagree with their criteria.

meta are a part of the argument, see the post on hills criteria.

“We will probably not have randomized controlled trials in the seriously ill.”
This is the problem isn’t it. Once the drug has come into widespread use and they “seem” to be effective, it becomes almost impossible to conduct a RCT, especially on seriously ill patients. Hence the importance of performing RCTs *before* they become widely used.

no argument there

“While we do not have randomized, controlled, clinical trials, we do have data to support the use of oseltamivir in ill patients. Not no data.”

The question is how reliable is that data.
I am reminded of the use of corticosteroids in premature babies to prevent blindness which seemed to be very useful until a RCT was done that showed it was actually harmful.

or steroids in sepsis as another example where trials keep giving contradictory results

Again see the Hills post

“The Cochrane review, which had used the published data in an earlier meta analysis, decided that they needed to be more rigorous this time. Why they didn’t bother, with their alleged mastery of the literature, to be this rigorous the first time I am uncertain”
Perhaps they learn from their mistakes. Perhaps, with experience, they improve their inclusion criteria to improve the reliability of their results.

perhaps. but I brings into question the result of their other meta’s

meta’s are seen by many as a be all end all, esp the cochrane, reviews. They are just another, potentially flawed, mechanism to get at the ever changing ‘truth’ of how best to treat patients.