Journal Club: Placebo Effect in Canine Epilepsy Trials

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Beats — Thu, 29 Sep 2016 21:27:45 GMT

[quote user="Virginia Campbell"]One study I think would be interesting would be to look at a lot of different blinded crossover trials to see if the active-then-placebo animals do better than the placebo-then-active animals...ie can you get a dog to associate getting an active pill with feeling better, then when it gets the placebo that smells and tastes the same, does it kick off an "expectation" in the dog that it will feel better which manifests as measurably improved results in the placebo phase compared to the placebo phase in the dogs that got placebo-then-active. This could prove a real placebo effect not just an owner-reported one. [/quote]

Interesting idea on testing for what I would also think of as a genuine placebo effect occurring - I think it's unlikely, but this suggestion is at least a plausible one that could be studied.

[quote user="Virginia Campbell"]Yes. This is why I would like to see the null hypothesis used as a starting point for studies were possible. I think it would encourage publication of more papers with negative findings as it appeals to the investigators' confirmation bias more ("See, we were right, our null hypothesis was supported!" rather than "Our hopes were dashed")[/quote]

I'm not sure I'm in agreement here, but might be reading your post or thinking it through wrong, so easiest to spell out my own thinking to see if conflicts:

I think the starting point should be a "hypothesis"

I think a "Null hypothesis" should only really come into play for statistical testing as the opposite of your hypothesis (whether this is a difference-hypothesis or equivalence-hypothesis) and you are setting up the null hypothesis then for purposes of statistical testing on the basis that if you can reject it with a certain level of probability that the results observed were not due to random variation, then acceptance of your original hypothesis (the "alternative hypothesis") can be made.

I would rail against doing such a study, failing to reject the "null hypothesis" and then claiming that this supports the null hypothesis, or indeed having the "null hypothesis" as what you were expecting to occur. Failing to reject a null hypothesis in a properly planned scientific experiment is not the same as proving/accepting/supporting the null hypothesis. If you anticipate there to be no difference and this is what you are trying to "prove" to whatever level of probability, then that is a question of equivalence testing and should be approached as such in my opinion. Statistical hypothesis testing is not a symmetrical thing that can just be flipped regarding the probability of the hypothesis being incorrectly accepted due to random variation (alpha or type1 error) when comparing a hypothesis-of-difference with a hypothesis-of-equivalence.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Arlo Guthrie — Thu, 29 Sep 2016 09:44:05 GMT

[quote user="Virginia Campbell"]There may also be some habituation to seizures of owners over time[/quote]

[quote user="Virginia Campbell"]but then over time you may become less stressy about it - the dog has had quite a few fits and not died, so perhaps your subconscious is more likely to accept the new reality and relax a bit and over time you may not be on high alert for seizures and identification frequency may peak then start to decline a little. [/quote]

A very accurate description of how we've adjusted to our daughter's seizures. Still on a permanent state of readiness, because of the risk of injury from a fall, but FAR more relaxed than the early years, and more likely (for example) to sleep through a nighttime seizure.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Anthony Dennison — Thu, 29 Sep 2016 09:35:47 GMT

Also, I'm afraid I haven't read the other studies yet, but what was the surgical implant and how can you have a washout from it? I'm guessing it required anaesthesia to place and remove, could that have caused any problems?

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Virginia Campbell — Wed, 28 Sep 2016 23:18:14 GMT

[quote user="Beats"] however some other suggestions could have been included such as positive publication bias -> it is unlikley that the authors on not observing an improvement with placebo from baseline in their previous 3 small scale trials would have chosen to publish this paper stating that they had failed to see an improvmeent from baseline with placebo in these, as such this paper only exists because of the observed positive result noted and it is unknown how many lack of observed placebo effect studies may exist in dogs had negative results for these undocumented findings been published to allow determination as to whether it is improbable that this positive-outcome paper exists.[/quote]

Yes. This is why I would like to see the null hypothesis used as a starting point for studies were possible. I think it would encourage publication of more papers with negative findings as it appeals to the investigators' confirmation bias more ("See, we were right, our null hypothesis was supported!" rather than "Our hopes were dashed")

I agree that the hypothesis itself should have been better stated - perhaps "response rates reported" rather than "identified", or "owner perceived as somebody else said upthread.

I'm not surprised that there is an apparent placebo effect [quote user="Beats"]I found it interesting that the human epilepsy trials seemed to find a much lower placebo effect - I wonder if this is reflective of better study design and larger scale studies etc, [/quote]

Yes...if you're a human you know if you'e had a seizure. If you're a dog owner expecting a treatment to work you may sleep more deeply and happily turn the radio up louder or the fan above the oven on at a higher setting, because you're not straining to hear Rover fitting in the living room as you might be if you expected him to be uncontrolled.

There may also be some habituation to seizures of owners over time..I expect in the first few months of having a epileptic dog you may get better at listening for them as you may recognise subtle cues that the dog is likely to fit soon (all the dogs had at least a years' worth of seizures before the study) so I would expect seizure identification to rise- but then over time you may become less stressy about it - the dog has had quite a few fits and not died, so perhaps your subconscious is more likely to accept the new reality and relax a bit and over time you may not be on high alert for seizures and identification frequency may peak then start to decline a little.

One study I think would be interesting would be to look at a lot of different blinded crossover trials to see if the active-then-placebo animals do better than the placebo-then-active animals...ie can you get a dog to associate getting an active pill with feeling better, then when it gets the placebo that smells and tastes the same, does it kick off an "expectation" in the dog that it will feel better which manifests as measurably improved results in the placebo phase compared to the placebo phase in the dogs that got placebo-then-active. This could prove a real placebo effect not just an owner-reported one.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Anthony Dennison — Tue, 27 Sep 2016 10:08:45 GMT

As I'm reading this between consults and may not time to go through it all, I'm afraid I'm going to be annoying and do it in bits.

I agree with Beats about the disconnected hypothesis - some authors when they have something different come up in their results compared to their original hypothesis they alter the hypothesis a bit so it matches, and you can usually tell when that's happened because the results say one thing and the discussion says another.

One of the inclusion criteria was a seizure frequency of at least 4 seizures a month - could this give bias towards owners who are desperate to find anything that works so will actively look for improvements? This is covered in the discussion quite well and the authors acknowledge that owner bias may have a greater influence than the placebo effect.

Using log-linear models for statistical analysis is useful as I would imagine the raw data was all over the place, applying a log makes it easier to compare.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

David Mills — Mon, 26 Sep 2016 23:31:39 GMT

I liked the paper. There are issues with its aims as set out well above by others, but I don't think it deters from the central issue.

Most (all?) of us use owner-reported incidence of seizures as the main measure for treatment effectiveness in treating canine epilepsy.

This paper - with small numbers - shows tentatively that an owner-perceived placebo effect occurs with treatment of seizures.

What it calls into question is the assessment of new anti-seizure medications without placebo. The Pexion data are a case in point - is it any better than nothing? And what position does that put us in as clinicians, recommending a therapy, that, although licensed, may be no better than a sugar pill? Is ignorance a defence?

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Beats — Thu, 22 Sep 2016 23:11:17 GMT

My random thoughts on reading this paper (which I found an interesting and worthwhile read):

I think there is a disconnect between the stated hypothesis and what the authors were wishing to communicate and discuss (which I thought they did well).

I personally think of a placebo effect as a therapeutically-beneficial effect, and I don't think this study demonstrates that dogs enjoy such a phenomenon.

I think it could have been clearer that this "meta-analysis" was a discussion on observations made by the authors during their previous 3 trials.

As a scientific paper, I think it suffers from not being repeatable by an independent investigator (there is no raw data to repeat the analysis on; 2 of 3 studies have not had detailed methodology produced elsewhere I think also).

There is a good discussion of the potential reasons behind the positive effect seen with placebo treatment in the 3 studies, to include regression to the mean, improved compliance to current therapy with owners daily documenting phenobarb/KBr administration etc, however some other suggestions could have been included such as positive publication bias -> it is unlikley that the authors on not observing an improvement with placebo from baseline in their previous 3 small scale trials would have chosen to publish this paper stating that they had failed to see an improvmeent from baseline with placebo in these, as such this paper only exists because of the observed positive result noted and it is unknown how many lack of observed placebo effect studies may exist in dogs had negative results for these undocumented findings been published to allow determination as to whether it is improbable that this positive-outcome paper exists.

No control exists w.r.t. the stated hypothesis (if I have understood this correctly - I'm not sure), thus no causality can be reliably inferred that the observed improvement to baseline was due to the placebo. To apportion the improvement to baseline noted with placebo to the administration of placebo, an appropriate scientific control would be no treatment (ignoring any ethical objections).

The hypothesis sounds improbable to me (if I have understood it to mean that the hypothesis is that dogs actually benefit from a placebo in trials), thus I would require quite a high burden of proof to convince me to accept it. If the hypothesis actually means simply that positive effects can be seen in trials when a placebo is given in dogs, then I would consider that quite likely (indeed my starting point would be to consider it to highly likely be the case).

There may also be bias from the authors of the studies being analysed performing this "meta-analysis" as they may be somewhat blinded to methodological flaws in the original studies.

I think there might be a significant trend to seizure reduction on enrolment in the trials as well as from baseline to placebo: All 3 studies reportedly had an enrolment criteria of at least 4 seizures per month or a history of cluster seizures while Figure 1 appears to show that the median baseline seziure frequency (i.e. after trial enrollment before getting active/placebo) was about 1/week in 2 of the studies and 0.4/week in the third study. By definition if the median is 1/week or less then, at least half are no longer fitting once a week and have had a decrease in seizure frequency. one would need to see the actual study data on patients to determine actual seizure frequency prior to enrolment and, again, I think a major deficiency of any scientific study is failure to provide the actual results rather than selected numbers supposedly summarisizing the point to be made. Certainly in the dietary modification trial (study C in FIgure 1), it appears that the vast majority of patients experienced a reduction in reported weekly seizure frequency on enrolment before any placebo treatment commenced.

More placebo-controlled trials with canine epilepsy have been perfomred since this analysis - superfically none appear to refute the suggestion that seizure frequency improves in placebo-treated group. Indeed, it is extremely hard to find any epilepsy trial in dogs that suggests a negative outcome for the investigated treatment - I found only 2 (one for levetiracetam compared to phenobarb as first line treatment where 5/6 of levetiracetam were withdrawn from trial due to high seizure frequency within 2-5mths, and one for EFAs). In contrast, a review of treatments in 2014 found all the contender drug studies to have been considered to have a positive effect even if the evidence was too poor to conclude that was necessarily down to the drug.

I got lost in the statistics section which went over my head - if anyone can enlighten me on the log-linear models any implications or caveats of these for the study interpretation and so on I'd be grateful.

I found it interesting that the human epilepsy trials seemed to find a much lower placebo effect - I wonder if this is reflective of better study design and larger scale studies etc, or just random variation suggesting nothing.

What do folks think of the levels of baseline-> placebo elevation in serum KBr (or phenobarb for diet trial) levels - at these differences would you expect to see an improvmeent in seizure control based on your clinical experience? It would be nice to see these broken down as raw data for patients as with standard errors in KBr might be nice to see if the "responder" patients were the ones with the more significantly incraesed KBr serum levels or not.

Is a major problem with performing placebo-controlled trials in UK for dogs that these would then require licensing as placebo administration is not act of "recognised" veteirnary practice? As such these are likely to require significant funding and as money talks be more open to other forms of bias?

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Francisco Gomez — Thu, 22 Sep 2016 08:54:15 GMT

My issue with this (and I hope I haven't missed anything in my ultra-fast reading) is that, for a prognostic study, the design doesn't account for the owner behaviour.

First, if the aim of the study is to check the placebo effect. Is this effect on the owner or on the patient? If it is on the patient, first we should, perhaps, learn first how the canine mind works to a much more detail that science has managed so far - (I can't see how a dog can make the connection between taking a sugar pill and not having fit otherwise) - So we must be talking about the effect on the owner. Then, the owner should have been subject of limitations of how they are included or not in the study (not just the dog) - For instance, do they work? how many hours do they spend at home? Are they next to the dog and awake each time a fit happens?

Finally, and as Neil points out, comparison like for like is pretty difficult when animals have been on other medications - but I suspect there would be an ethical dilemma of using known placebos on their own on animals with epilepsy - I.e. you know the dog will suffer because we aren't giving things that we have and we know that these things work.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Virginia Campbell — Thu, 22 Sep 2016 08:52:09 GMT

[quote user="Neil Wheadon"]The placebo's were in addition to existing medication such as phenobarbitone and bromide.[/quote]

As we get more effective treatments available in vet med this becomes more of a problem for clinical trials, as it's unethical to trial new drug against placebo, rather than known effective drug+new drug against known effective drug+placebo.

RE: Journal Club: Placebo Effect in Canine Epilepsy Trials

Neil Wheadon — Thu, 22 Sep 2016 02:51:14 GMT

OK, I'll go first

Issues

1) Done in 2009, before Keppra

2) The placebo's were in addition to existing medication such as phenobarbitone and bromide. I have read that maybe 50% of dogs are poorly controlled, so trying to compare like with like is pretty tricky here.

3) Fits based on owners observations. How many dogs fit without an owner being aware?

4) Finally the last part of the conclusion and I paste this

The main limitation to the present study is the low

number of dogs enrolled in each of the trials evaluated,

such that it is not possible to conclusively state the sig-

nificance of the placebo response observed. The lack of

ﬁnancial and administrative support that is required for

large-scale trials is perhaps the most consistent factor

that limits the ability to perform controlled clinical trials

in veterinary medicine.

Neil

PS I like this idea (The discussion of varying topics, not use of placebos)