Accuracy of tests

Re: Accuracy of tests

Bob Russell — Fri, 10 Dec 2010 13:32:14 GMT

I generally make my excuses and leave when I am bored by a conversation.

Re: Accuracy of tests

Kate Richardson — Fri, 10 Dec 2010 13:10:11 GMT

I think Martin be be slightly bored by the conversation???

Re: Accuracy of tests

Bob Russell — Fri, 10 Dec 2010 13:04:06 GMT

[quote user="Vikki Halliday"]

Martin,

????? What on earth is that picture about or have I been blonde and missed the point?

Vikki

[/quote]

Maybe I am really blonde instead of grey - I seem to have missed the point as well.

Re: Accuracy of tests

Vikki Halliday LLB — Fri, 10 Dec 2010 11:28:29 GMT

Martin,

????? What on earth is that picture about or have I been blonde and missed the point?

Vikki

Re: Accuracy of tests

Anonymous — Thu, 09 Dec 2010 11:30:32 GMT

Hi Alex

Do you ever think that, while when screening for e.g. cervical cancer one needs a cut-off point at which further investigation is done, when investigating a clinical case this is less of an issue and 'cut-off' points' are a bit of a more random concept? I appreciate that most clinicians intuitively think to themselves "Oh, variable 'x' is only just above the cut-off point, I'll be a bit suspicious" etc, but would there be any harm in trying to actually use something a bit more precise than this and use likelihood ratios or some other tool rather than choosing a cut-off point based on a ROC curve above which is 'positive' and below which is 'negative'?

My fav example of the meaninglessness of reported sensitivity and specificity values is in tests for canine pancreatitis. Have a look at http://www.idexx.co.uk/animalhealth/laboratory/speccpl/compchart.htm if you want a chuckle (and yes, I have told them that this kind of intentionally misleading visual display of data is outrageous - I was quite shocked actually, more what I expect from drug company literature ).

Re: Accuracy of tests

Martin Atkinson — Thu, 09 Dec 2010 10:35:14 GMT

Re: Accuracy of tests

Alex Allen — Thu, 09 Dec 2010 08:49:04 GMT

Hi Julian - the figures are purely hypothetical based on the assumption of a disease prevalence of 10% - this could be determined by a reference test or by a series of tests.

The examples show how different test performances can reveal different sens / spec / ppv and npv - it is these values the test designers deliberate over to see which would be the most practical test performance. For example in a test designed for initial cancer screening the sensitivity is the most important as follow up tests will then be used to eliminate the false positives - for example cervical smear testing. This deliberate choice to be over sensitivity is to reduce the risk of false negatives at the first test in a series, which could prove fatal.

In a less serious disease that could be left to advance before confirmation the test may use a lower sensitivity. ROC CURVES are used to decide what to use as your cut off values when determining at which point your test becomes positive or negative.

A simple tutorial on this is found at http://gim.unmc.edu/dxtests/ROC1.htm or (as mentioned previously - Mike Thrusfield's epidemiology textbook.

Re: Accuracy of tests

Alex Gough — Thu, 09 Dec 2010 08:28:06 GMT

It was also pointed out to me that you can have certainty in a test that is 100% specific, if you get a positive diagnosis. Apparently pregnancy testing in humans is 100% specific, unless you are a male with prostate cancer (which begs the obvious question...)

alex

Re: Accuracy of tests

Anonymous — Wed, 08 Dec 2010 23:32:36 GMT

[quote user="alex gough"]

Fair point, but I don't think it is unreasonable to say that radiography is not 100% sensitive and specific for fractures.

[/quote]

It is entirely reasonable to say that radiography is not 100% sensitive (perhaps non-displaced, hairline fracture) and specific (e.g. artifacts due to Mach lines etc.) for fractures; I would disagree if you said otherwise. This is why I gave a random example of something that radiography is 100% sensitve and 100% specific for.

[quote user="alex gough"]

Otherwise you are in effect saying that radiography is 100% accurate for fractures that I can see on xrays!

alex

[/quote]

Sort of. 'Accuracy' in the sense of diagnostic testing is generally defined something like:

"Accuracy is an indication of the extent to which an investigation or measurement conforms to the truth. Thus, if a set of scales records an animal's weight at 15kg and this is the actual weight of the animal, then the measurement is accurate." Thrusfield 1997, Veterinary Epidemiology 2nd ed., p.134-135.

Without getting too side-tracked, I'd have to think exactly what '100% accurate' actually means, or if this is simply a tautology? I'm going to assume you mean '100% sensitive and 100% specific' by this, in which case, yes - if what you are testing is fractures that you can see on a radiograph then radiography is 100% sensitive and 100% specific for them. 'Fractures that you can see on an x-ray' is a poor choice of something to test for I would argue, as this is not a clinical diagnosis at any normal level of refinement.

In general though, I think you're answering the question that I think you're trying to ask. Pretty much any test that is 100% sensitive and 100% specific will be a defining feature of what you're testing for. If it's blood tests you're after, then maybe something along the lines of a DNA test for gender (male defined as having a Y-chromosome), DNA testing done in accurate fashion (I'm of course not suggesting that being male is a 'disease'...).

On an aside, I had the privilege of attending a TB-testing training day in N. Ireland a couple of years back. The vet in charge of TB control at the Department pontificated on the 99.99% specificity of the comparative intradermal tuberculin test, and how we should advise farmers with a reactor who believed it didn't have the disease that there was only a 1 in 10,000 chance that it didn't and was being slaughtered needlessly. Ignoring the ludicrousness of the quoted specificity value, spot the problem (HINT: TB in the area I was testing was at a prevalence of approx 0.1% according to his preceding slide on the presentation, you can assume 100% sensitivity for the calculation if you want )

Re: Accuracy of tests

Julian Earl — Wed, 08 Dec 2010 22:53:18 GMT

[quote user="Alex Allen"]

For statistics fans out there here is a worked example from one of my thrilling talks ....there must be at least one more madman like me out there....

Population size = 500

Disease prevalence estimated at 10%

Test Example 1:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	48	2	96%	Sensitivity
DISEASE	Negative	40	410	91.1%	Specificity
		54.5%	99.5%
		Positive Predictive Value	Negative Predictive Value

Test Example 2:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	47	3	94%	Sensitivity
DISEASE	Negative	15	435	96.7%	Specificity
		75.8%	99.3%
		Positive Predictive Value	Negative Predictive Value

Test Example 3:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	45	5	90%	Sensitivity
DISEASE	Negative	2	448	99.6%	Specificity
		95.7%	98.9%
		Positive Predictive Value	Negative Predictive Value

The test with the highest sensitivity is not one that would be practical in clinics due to the high % of false positives. All three tests will be accurate if the test is negative but detection of disease varies considerably.

[/quote]

Alex, are those taken from three different tests and achieved results? If so, how is it known that those particular animals were positive? Infected deliberately? Or identified by some other test? It could easily lead to a circular argument and inaccurate premise if we're not careful - "this cat is positive because this test tells me; this test is accurate because it has told me that this positive cat is also infected..." etc.

I'm glad I'm not the only one to find this particular subject interesting although it can be quite challenging as well.

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 21:32:51 GMT

There is sort of a point to the hypothetical question btw, albeit one that I am trying to frame in the right words. It basically comes down to a discomfort with giving firm diagnoses. One referral centre I know has a section on their discharge sheets for "certainty of diagnosis". I have never seen them fill in 100% for this, they have a tendency to put "final", which I take to mean as good as it gets, and we don't intend running any further tests. Do we discuss the uncertainty of our diagnosis with the owners? Most of them dont like to think that a lot of expensive tests have been run and we still dont "know for sure". I would prefer to share the uncertainty in most cases, even if it is only a minor uncertainty, than look foolish later.

There is a chapter I think in Kirk's Current Veterinary Therapy XII, which I have seen the title for but never read, and would be intrigued to know if anyone has access to it, what the gist is, called something like, "Diagnosis in the face of uncertainty". I think we all deal with uncertainty all the time, whether it is the results of a FNA, a blood test or a MRI. And I guess therein lies one of the arts of veterinary medicine.

Sorry if I'm rambling a bit, I just find it interesting.

Alex

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 21:25:03 GMT

Fair point, but I don't think it is unreasonable to say that radiography is not 100% sensitive and specific for fractures. Otherwise you are in effect saying that radiography is 100% accurate for fractures that I can see on xrays!

alex

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 21:22:46 GMT

[quote user="Christopher Saul"]

[quote user="alex gough"]

Not that specific if there is prior treatment with steroids (which you may or may not be aware of).

Alex

[/quote]

It's a genuine question because I can think of no good examples. There must be some though. I think for the purposes of this discussion, maybe we should exclude the results of a physical examination, as a colleague noted that he felt he could diagnose an animal that was missing its head with 100% sensitivity and specificity!

Well you're just answering your own original quesion really: no there isnt a test which is 100% sens and spec!

[/quote]

Re: Accuracy of tests

Simon Neuhoff — Wed, 08 Dec 2010 21:07:39 GMT

[quote user="Christopher Saul"]

If diagnostic tests were 100% sensitive and specific then we as diagnosticians would be redundant. The art of veterinary medicine is partly about how we interpret test results in the light of signalement and clinical findings.

Would we really want such a test to exist anyway?

[/quote]

Not entirely true cause you would still need to know which test to choose. You could hardly run EVERY test in every case - although sometimes people seem to like to try!

Re: Accuracy of tests

Christopher Saul — Wed, 08 Dec 2010 18:40:06 GMT

Would we really want such a test to exist anyway?

Re: Accuracy of tests

Anonymous — Wed, 08 Dec 2010 18:33:05 GMT

[quote user="alex gough"]

[quote user="David Beattie"]

I'm not sure I've quite got what you mean?

Depends on what testing for?

For instance, I believe radiography (2 views) to be 100% sensitive and 100% specific for a displaced comminuted femoral fracture in a dog.

[/quote]

Yes, but widening it slightly, radiography is not 100% sensitive and specific for femoral fractures.

Alex

[/quote]

No, but by widening anything for which one is testing, you can clearly reach a situation where specificity +/or sensitivity are reduced, thus nullifying the point of your question.

In the general case what you are saying is:

a test for 'illness x' is 100% sensitive and 100% specific; this test is not however 100% specific for 'general illness'.

Re: Accuracy of tests

Christopher Saul — Wed, 08 Dec 2010 18:20:56 GMT

[quote user="alex gough"]

Not that specific if there is prior treatment with steroids (which you may or may not be aware of).

Alex

[/quote]

Well you're just answering your own original quesion really: no there isnt a test which is 100% sens and spec!

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 18:16:11 GMT

[quote user="Christopher Saul"]

If it's mecial tests you're after, I'm not sure it's possible to have a test that is both 100% sensitive and specific, though I suppose some get pretty close on one score eg ACTH stim is very specific for addison's.

[/quote]

Not that specific if there is prior treatment with steroids (which you may or may not be aware of).

Alex

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 18:14:06 GMT

Thanks Alex. That was sort of the point I was making, although the unwritten assumption in my original question was that we discussing the actual sensitivity and specificity (compared to reality), rather than compared to another test. However, practically speaking that is often the only way. I think certainty in veterinary medicine, or human medicine, is very rare, and often illusive. Of course we can perform a second "confirmatory" test, but if that second test isn't also 100% sensitive and specific then doubts remains. And sometimes that second test will be measuring a similar thing to the first one, so you can't just say that 2x90% certainties equals a 95% certainty (or whatever that figure would be). For example, a "positive" ACTH stimulation test, a "positive" urine cortisol:creatinine ratio and bilaterally enlarged adrenal glands might mean pituitary dependent hyperadrenocorticism, but might also mean a non-adrenal illness causing increased cortisol and adrenomegaly.

Alex

Re: Accuracy of tests

Alex Allen — Wed, 08 Dec 2010 18:02:02 GMT

The main problem with sens and spec of medical tests is that it is based on a comparison with a reference test. What the reference test's accuracy is can be overlooked...does the reference test for the same thing? If the reference test is still flawed in one way or another the other test's sens/spec of 100% does not necessarily mean it is 100% accurate.....just the same as the reference test.

Also beware of statistical manipulation - a test may appear to be very good when looking at just the sens and spec but the predictive values can reveal some shocking truths.... More recent studies into FIV/FeLV in houses test did just this where Sens, spec, NPV and PPV were all compared before any conclusion could be made. BUT as mentioned in the study the reference tests may identifying different things which may not be direct comparisons to the in house test.....

Next comes the question of what the postive result means in relation to the outcome...do you condemn to death a patient for having a one off positive or do you reinforce the test result by running additional tests. How the "positive" is interpreted is vital and often should only be used as a step in the diagnostic process. Tests are rarely considered black and white. Sensitivity versus PPV - see below

The old idea of euthansing a FELV positive cat without retesting has now (thankfully) been revised. I dread to think how many cats were euthanased back in the 1990s because they came up positive on a one off in-house test.....

For statistics fans out there here is a worked example from one of my thrilling talks ....there must be at least one more madman like me out there....

Population size = 500

Disease prevalence estimated at 10%

Test Example 1:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	48	2	96%	Sensitivity
DISEASE	Negative	40	410	91.1%	Specificity
		54.5%	99.5%
		Positive Predictive Value	Negative Predictive Value

Test Example 2:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	47	3	94%	Sensitivity
DISEASE	Negative	15	435	96.7%	Specificity
		75.8%	99.3%
		Positive Predictive Value	Negative Predictive Value

Test Example 3:

		TEST RESULTS
		Positive	Negative
DISEASE	Positive	45	5	90%	Sensitivity
DISEASE	Negative	2	448	99.6%	Specificity
		95.7%	98.9%
		Positive Predictive Value	Negative Predictive Value

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 18:01:01 GMT

[quote user="Christopher Saul"]

Continuing David's theme, I suppose physical examination is 100% sensitive and specific for limb amputation

[/quote]

This one is operator dependent. I suspect unless a full physical was carried out, it might not always be obvious that a pekingese being clutched to its owners ample bosom (male or female) has only three legs!

Alex

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 17:58:47 GMT

"I was just hibernating. Don't you guys ever take a pulse?"

Plenty of people get misdiagnosed as dead, (there is a society of dead people in India for people who have been wrongly declared as officially dead) and I am aware of horror stories of euthanased animals that are put in a bag in the freezer and are found the next morning frozen, having clawed their way out of the bag but not out of the freezer.

Alex

Re: Accuracy of tests

Alex Gough — Wed, 08 Dec 2010 17:55:23 GMT

[quote user="David Beattie"]

I'm not sure I've quite got what you mean?

Depends on what testing for?

For instance, I believe radiography (2 views) to be 100% sensitive and 100% specific for a displaced comminuted femoral fracture in a dog.

[/quote]

Yes, but widening it slightly, radiography is not 100% sensitive and specific for femoral fractures.

Alex

Re: Accuracy of tests

Utlendigur — Wed, 08 Dec 2010 17:49:29 GMT

[quote user="Vikki Halliday"]

Death certification?

[/quote]

Not always obvious - Hibernating tortoise?

Re: Accuracy of tests

Martin Atkinson — Wed, 08 Dec 2010 17:48:11 GMT

[quote user="Malcolm Ness"]

Provided you examine the right leg!

[/quote]

Ah but is that the right leg as related to the left leg or the wrong leg?