RE: How good are scientific trials in variable illnesses?

Arlo Guthrie — Tue, 08 Sep 2020 17:21:21 GMT

Thank you Andreas Ege, and yes, totally agree, Ben Goldacre's are brilliant. Eye-opening and compelling reading.

RE: How good are scientific trials in variable illnesses?

Andreas Ege — Tue, 08 Sep 2020 16:02:07 GMT

Hello,

[quote userid="5464" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses/225294#225294"]most people hire a statistician to tell them how many patients they need to do the trial on in order to get a significant result.[/quote]

Unfortunately, I doubt that most studies do a power calculation, though it seems getting more frequent.

[quote userid="5464" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses/225294#225294"]I'm sure the system is pretty robust for veterinary, and more so for human drugs[/quote]

I very seriously doubt that. A worthy though scary read: Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients from Ben Goldacre. Didn't find much new in there but for the shear scale. I don't doubt for a minute that veterinary pharmaceutical behaviour is that different.

And don't get me started on the cascade system... ;-)

[quote userid="2100" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses"]how accurate are they when the disease is cyclical/variable?[/quote]

The short answer is, it depends on the study(ies), i.e. methods employed, quality, length of observation period, number of animals... Obviously, the more variable a problem is, the more complicated research does become - needing higher numbers of patients, longer observation periods and better control for compounding factors.

[quote userid="2100" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses"]I do wonder whether a reported effect could still be chance (or is even highly likely to be chance),[/quote]

In veterinary medicine the number of animals used usually is abysmally small, particularly in RCT. I mean, you get papers that examined the intervention in 8 dogs over 6 weeks and extrapolate recommendations for the whole dog population. Nutritional research seems particularly vulnerable to that. (A diet is complete and balanced by AAFCO regulations if 6 out 8 (!) dogs survive for 26 weeks without loosing more then 15% body weight, up to 2 dogs are allowed to drop out for "non-nutritional" reasons - and then they tell me they know the food is suitable for the majority of dogs for years.)

Bigger studies usually are done for human research using animals as models but obviously, these results are often only partially useful.

[quote userid="2100" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses"]I do wonder whether a reported effect could still be chance (or is even highly likely to be chance),[/quote]

Yes. That's one of the reasons the in medical science often so much loved p-value is of limited worth. If you think about it, the usual (arbitrarily) picked p=0.05 as being significant is higher than your chance of casting a 1 twice in a row with a (ideal) die (p=0.028 - it's significant, so it's not chance now, is it? ).

In the end, all these statistics primarily tell us that the success after some intervention is more likely due the intervention than good luck. It doesn't tell us it is not chance. And, obviously, it only tell's us how a given percentage of patients might react, not how the individual in front of us does.

[quote userid="2100" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses"]have a strong suspicion that the company doing the trials just got lucky. Is that possible in a controlled trial?[/quote]

Again, yes.

In addition, one can influence the results by the way one asks the question or chooses the methods (surrogate outcomes are a favourite if primary outcome is difficult to measure. Or if the study didn't give the right results for the primary but good results for a convincing surrogate).

Finally, they don't have to publish data they don't like. Nobody knows, how many studies are hibernating on hard drives, because the results weren't what people wanted to find. (If I remember right, some journals these days insist of having the study details before the study is done, but unwanted results then get published in very low impact journals nobody reads, or some similar system). And that ties neatly in with above, being lucky. If you repeat a study often enough, you have a good chance of getting lucky.

In the end, as Rob Davis writes, repeating and thus confirming or refuting results is very important. Unfortunately, it is also highly unpopular, both by scientists and journals, not the least because you might get laurels for the primary study but not for repeats.

The above mentioned book is a really good read about how pharmaceutical companies can and often do manipulate the system to their advantage.

RE: How good are scientific trials in variable illnesses?

Rob Davis — Mon, 07 Sep 2020 12:55:47 GMT

[quote userid="5464" url="~/001/veterinary-clinical/small-animal/orthopaedics/f/discussions/29343/how-good-are-scientific-trials-in-variable-illnesses/225294#225294"]In short, I think most people hire a statistician to tell them how many patients they need to do the trial on in order to get a significant result[/quote]

But a "significant" result is not necessarily a correct or representative result. It could be "lucky", even more so if not all trials written up. Ideally trials should be replicated to see if the same results are achieved.

RE: How good are scientific trials in variable illnesses?

Chris Geddes — Mon, 07 Sep 2020 12:34:46 GMT

In short, I think most people hire a statistician to tell them how many patients they need to do the trial on in order to get a significant result. Then peer reviewers have to agree before it's published. Google p values if you're suffering from insomnia. Re something getting licensed - it's the job of the regulator to determine that results are significant and not just lucky. I'm sure the system is pretty robust for veterinary, and more so for human drugs, though I'm also sure the odd mistake happens along the way (Thalidomide being the obvious example, though I'm sure things have improved since then).