The Problem with "Studies"

I’m a big fan of studies. I’ve been reading them, citing them, and utilizing them for decades. But you have to use them wisely. You have to be careful how you consume and think about them.

This’ll be a bit of a rant, so bear with me.

Something that always amuses me is when I see a health influencer talk about a study showing that some exercise modality or supplement or sauna session reduces the risk of heart disease by 24%. That cracks me up because the day before, and probably the day after, they will have been talking about some other measure you can take to lower your risk by 12% or 33% or 10%.

Now, I don’t know if this is how they intended to come off, but I will tell you that people read this or they watch it and they assume that these benefits are stackable. The average person thinks that if you do Tabata intervals on a bike three times a week, and use a sauna five times a week, and take a high-powered multivitamin every day, take krill oil morning and night, and get cold exposure, that these reductions in risk are additive. If you just do enough, if you do all the right things, read all the studies, and follow their recommendations, that you will become impervious to any degenerative health condition. You may even become immortal.

A lot of this thinking happens subconsciously. It’s more feeling than thinking, in fact. But when you consume so much health content from influencers who speak very calmly and authoritatively, it’s easy to feel like you can just add up the risk reductions and end up with the total approaching, or in many cases exceeding, 100%.

There are some problems with this approach. For one, the things you’re stacking are often contradictory. The study tells you that strength training reduces mortality risk by 20%, and another study tells you that taking ice baths reduces it by 10%. Can you just add those two and get 30%? No, and here’s why. They operate along different pathways. The benefit from strength training comes from hypertrophy, lean muscle mass, and a little bit of bone mass. Cold exposure actually counters that response. It impedes, or at least reduces, your maximum potential for muscle hypertrophy. Particularly if you do an ice bath after a workout, you will not gain as much lean mass as if you would’ve done it otherwise.

Or consider exercise and high-dose vitamin C. Most people would say high-dose vitamin C is probably pretty good for you, but it can actually impair the adaptations incurred from exercise. My point in all this is not to say that you can only pick one, that exercise is only good if you don’t take vitamin C, or that vitamin C is only good without exercise getting in the way. What I mean is that you can’t just add up the benefits you see in separate studies. They interact with, and at times contradict each other.

Another problem is that all these papers are done in different populations. The Tabata interval study done in Swedish male cyclists. The dark chocolate study done in Bolivian horticulturalists. The statin study done in men with a history of heart attacks. The results may not apply equally to everyone.

And hell, a great many of these studies aren’t even done in people at all. As Bret Weinstein has pointed out, they’re often done in mice, and not just normal mice but lab-engineered mice that are designed to have longer lifespans, longer telomeres, higher cancer risks, and who are generally just more responsive to interventions. This is actually why so many of the extremely promising-sounding animal trials for cancer drugs and other therapies never really pan out in people. They’re studying genetically homogenous engineered mice who have been designed to respond well to drugs, and then when you try to use them in dramatically heterogeneous human populations, the big mishmash of people from every corner of the world, it’s no surprise that the drugs don’t really work as well.

My point is not to despair and give up. It’s to temper your expectations. It’s to realize that there’s not gonna be any magic bullet in a bottle that you can take and have all your problems disappear. The reality is that how to be healthy is much simpler, but not very easy. You do have to work. You have to sweat, breathe hard, and muscles will have to burn. You’ll have to be uncomfortable sometimes, and when you do have comfort, you’ll have to have earned it.

Another funny thing that people get wrong is when studies talk about, say for example, vigorous exercise being linked to greater benefits for longevity than walking. These are purely numbers being shuffled around. They’re not looking at individuals who only did intense cardio versus individuals who only did walking. They’re looking at the mishmash of real humans doing a huge blend of all different activities and dying from any number of different causes at any different periods in life and ages, and trying to discern trends from the data.

What the stats say is that, all else being equal, this is associated with more or less this. The problem is that all else is never equal. You can’t control for all these variables, and lives lived certainly do not.

The study that the influencer cited showing that interval training reduces mortality by 20% was conducted in a different group of people than the study that found walking was linked to a 70% lower risk. Studies are just snapshots. They may be snapshots that span large groups of people, and they may suggest an element of truth, but they’re still just snapshots of the study population at that moment in time, given those parameters and study designs.

Something I talk about all the time is that in the end the only experiment that matters is yours. If you are the outlier in the study (or someone else in the study who’s a lot like you had an outlying result that differed from the average), your result is the one that matters for you. Not the average of everyone who participated.

Subscribe to our premium content to read the rest.