What does 'Clinically Proven' mean?

What does 'Clinically Proven' mean?

We often see the words 'Clinically Proven' on products, for example, a toothpaste is 'clinically proven to reduce tooth decay', or a mouthwash ‘clinically proven to reduce gum disease', but what does it actually mean?

The term 'clinically proven' conjures up images of noble scientists in white coats studying the contents of test tubes in a brightly lit laboratory before crying out “Eureka!” and happily telling some fat cat in a suit that their product works and can now be sold to the masses.

Sadly, the truth is somewhat different. The short answer is that 'clinically proven' actually means...

(wait for it)

(drum roll please)

Nothing…almost.

It is a virtually meaningless phrase. Nearly as meaningless as '8 shades whiter', but not quite that bad.

"Come on," I hear you say, "it must mean something, THEY can't say something is clinically proven if it's not, can they? Surely not?". I hear you end your plea in a diminuendoing whimper.

(I've no idea whoever THEY are, but for the purposes of this article let's say they are the big manufacturing corporates, the bad guys, evil money-grabbing suits trying to exploit the unsuspecting consumer out of their hard-earned cash.)

Well, that's true, sort of. No manufacturer can make outrageous claims in advertising, otherwise, the Advertising Standards Authority would get all cross and uppity, however, it is perfectly legitimate for companies to make claims about their product as long as they adhere to the Advertising Code. The Medicines and Healthcare Products Regulatory Agency (MHRA) 'A guide to what is a Medicinal Product' defines 'Clinically Proven' as:

'An implied claim that the product has met the appropriate efficacy test in relation to disease or an adverse condition.'

Roughly translated that means they just have to make it seem like they’ve done an experiment that proves the product works.

It’s hardly cast-iron proof is it?

Furthermore, and here prepare to get your mind blown, no scientific experiment has ever proven anything ever. EVER.

(I apologise for the overuse of italics in the above statement, but I think they’re justified.)

The most any experiment or study can say is: ‘the evidence suggests…’

It’s true. I bet you’re thinking you’ve been lied to all your life, which may be the case, but put away your conspiracy theories for a bit and read on:

In order to understand why an experiment can’t prove anything, we need to look at what a scientific experiment is, plus some other phrases such as null hypothesis and statistical significance. There’s plenty more detail we could go into, but let’s keep this light-hearted, heh?

Null Hypothesis

The Null Hypothesis (or H0 if you want to impress the ladies*) is a statement made in every experiment which states that doing the experiment is effectively a waste of time. For example, ‘The Null Hypothesis is that Toothpaste A has no effect on tooth decay’.

Now, experiments cost money (a lot of money) and businesses aren’t keen on wasting money so they’re really not going to be keen on spending a lot of money on an experiment that isn’t going to ‘prove’ that their product works.

The bosses of a company are far more interested in the what’s called the ‘Alternative Hypothesis’. In this case our alternative hypothesis could read: ‘Toothpaste A cures tooth decay and gum disease’.

Now the bosses are interested. They’re very happy investing money in an experiment that shows their product works. A cure for tooth decay and gum disease? Ker-ching!!! They’ll be richer than Midas in no time at all. Provided it does actually show that, and there’s the rub.

You can imagine the conversation:

Fat Cat: “So, I want you to design an experiment that shows our toothpaste cures tooth decay and gum disease.”

Brow-beaten Scientist: “Bloody hell. What?”

Fat Cat: “You heard sonny. Get it done. I’ve just paid a deposit on a Maserati.”

Brow-beaten Scientist: “Don’t you think it’s a bit of a tall order? You want to prove two of the most common diseases known to man are cured by a toothpaste? Can’t we tone it down a bit?”

Fat Cat: “Oh stop with the negative vibes, man. You can do it.” He grins in a supportive way.

Brow-beaten Scientist: “Eurgh, what’s wrong with your face? Uh, well for a start gum disease can be really tricky to stop, you need hygienist support, specialised brushes, possibly mouthwashes…”

Fat Cat: “Yeah, yeah. Forget the gum disease. What about just curing tooth decay?”

Brow-beaten Scientist: “Well, if people improved their diets by reducing their sugar intake and snacking frequency, that would have a far greater effect. Our toothpaste probably wouldn’t be necessary.”

Fat Cat: “What? Blimey. Keep schtum about the sugar thing. Can we put sugar in tooth paste?” Scribbles on his notepad.

Brow-beaten Scientist: “NO”

Fat Cat: “Ha ha. Only joking.” Draws a line through his writing. “Shall I cancel that deposit?”

Brow-beaten Scientist: “No, there’s no need for that. We just need to be a little more…vague.” He makes Obi Wan’s ‘these aren’t the droids you’re looking for’ motion with his hand.

Fat Cat: “Don’t do that, it’s not cool. What do you suggest?”

Brow-beaten Scientist: “I can design an experiment to show that our toothpaste is, how shall I say it? ‘effective in reducing tooth decay’. That should be suitably vague.”

Fat Cat: “Can you do that? That’ll be great! The punters will well go for that! How much do you need for this?”

Scientist: “Oh, I don’t know. How much is a Maserati?”

Fat Cat: Sighs and calls car show room to order another Maserati.

So if you do find if an experiment is mentioned by a company in relation to their product, you can safely assume the study shows the product in a good light, or to use the proper term, demonstrates a ‘high risk of bias’.

So now we have our two hypotheses:

Null Hypothesis: Toothpaste A has no effect upon tooth decay and;

Alternative Hypothesis: Toothpaste A is effective in reducing tooth decay.

The scientist needs to design the experiment to ‘prove’ his Alternative Hypothesis, and dismiss the Null Hypothesis.

Now we need to talk about levels of evidence. See the table below. You can see that the highest-level evidence is obtained from Randomised Controlled Studies (or Trial), and the lowest from expert opinions. Below these levels you’ve got your Mum’s opinion and the opinion of Dave your best mate from down the pub. I know your Mum and Dave would disagree with that, but there it is. Tough.

Levels of evidence

Figure 1 Levels of Evidence from https://www.cebma.org/wp-content/uploads/Levels-of-evidence.png

A Randomised Controlled Trial (RCT) compares the effect (Outcome) an action (or Intervention) has upon a group (Population) compared with a control group (Comparison).

Population

The choice of population can determine how impressive the results are or even the overall success of the trial. For example, if you choose a population for a toothpaste trial of adult patients with no fillings and no decay, you’re not going to get very impressive results because it’s unlikely that you will be able to show a difference. Likewise, if you choose a group of volunteers who regularly attend the dentist, they are less likely to have decay than irregular attenders. So, let’s choose a population of people at high risk of decay who don’t attend the dentist. Are you beginning to see how a trial can be biased?

Intervention

In this example we will keep it quite simple by saying that the intervention is using Toothpaste A for two minutes twice a day on an electric toothbrush.

Comparison

The comparison, or control group, can greatly influence the results of a trial. First of all, if there is no comparison, you can’t actually say whether your intervention is better or not because you’ve got no starting point. Additionally, in order to be ethical, the comparison must be a fair one and, in many cases, the current gold standard. For example, it will be quite easy to show that Toothpaste A is more effective at reducing decay compared to brushing your teeth with treacle.

In our example we may need two control groups because we don’t know how effective the toothbrush alone is in reducing tooth decay, so one control group could be just brushing with a toothbrush alone and another brushing with the current market leader in toothpastes.

Obviously, the owners of the company are going to be reluctant to invest millions in an RCT to show that their competitor has actually got a much better toothpaste than them!

In our example let’s assume that the scientist is ethical, and compares Toothpaste A with Toothpaste B, the current market leader from a competitor.

Outcome

Now comes the fun part (or boring part depending on how much you like statistics). Analysing the results. Obviously if Maseratis are going to get handed out we need irrefutable proof that Toothpaste A is far better at reducing decay than Toothpaste B. However, this is very tricky as there are lots of variables involved, such as the toothbrushing technique and diet of the patient in the trial or whether the operator or the patient are ‘blinded’**.

Another problem with a trial such as this is that tooth decay can take years to develop, the longer a trial takes, the more it costs. The companies will want quick results, so they may use a different type of study, such as a retrospective study, which isn’t as accurate (but who cares, so long as the punters are convinced).

Also, if you trial the toothpaste on ten patients and one of them gets decay, does this means it is any good? Maybe, but how do you know?

Essentially, it simply costs too much, takes too long and there are too many variables involved to ‘prove’ that the latest product the companies want to sell is effective.

So they cheat.

Well, not exactly. It’s called extrapolating data.

It works like this: We don’t have to learn everything from scratch in order to prove something. In our example, Toothpaste A contains Fluoride which has been shown to be effective in reducing tooth decay in many other studies. So the scientists use data from other studies and apply it to their brand new toothpaste, on the assumption that the results are transferable. Therefore, they can safely say their product is ‘clinically proven’ despite it not being around long enough to have had an effective study done directly on it.

Summary

So, in conclusion:

  • Nothing is clinically proven to do anything.
  • There is likely to be strong evidence to support a product is effective in what it claims to do, but the evidence may not come from a test specifically on that product.
  • Just because a product is effective at treating a certain condition, that may not mean it is suitable for you. If in doubt, get advice from a professional.

*It doesn’t. I’ve tried, don’t bother. Swot up on their favourite TV programs instead.

** This doesn’t have anything to do with eye-gouging in the name of science (unless that is what the study is about). It’s just a way of preventing bias by not influencing the patient.