Magness Speaks — How to Spot Bad Science and Fads: Determining Whether an Idea is Worthwhile
This is a blog straight from email requests. It’s not exactly about training, but one of the most frequent and perhaps most important question I get asked is how the heck do you spot the good stuff from the crap…
A while back I read the book Bad Science: Quacks, Hacks, and Big Pharma Flacks, which is an excellent read, and then just started reading the book Proofiness because of a recommendation. Both books are in a similar vein in that they both touch on how people can get fooled into believing wrong things. One focuses on the use of statistics while the other goes over how “guru’s” use bad science to fool the masses.
While it’s not directly related to training or running, it plays a crucial indirect role. As a coach or an athlete, we are bombarded with different training philosophies every day. If that isn’t enough, many of us browse through the latest scientific journal findings to see what’s going on in that side. It’s hard enough filtering through it if it’s your job and you’re a coach. While it’s impossible to come up with some tried and true method of evaluating claims, I’d like to go over several ways to spot a bad idea, bad science, or a bad expert, first and then offer a few brief suggestions on how to avoid the trap.
Falling into the trap:
The crossfit phenomenon is interesting to me. I don’t want to get caught up in the efficacy of it, but instead focus on some of the claims made about endurance performance from them. Why? Because it serves as a perfect example of falling into the trap. Before I get a backlash from the crossfit people, my premise is that using only high intensity max work to race a distance event isn’t the optimal way to do it. CFE has claimed before that runners would be better off maximizing their running performance if they ditched the traditional way and trained their way, random high intensity, low volume work…
I was browsing through Tim Ferris latest book at Barnes and Noble the other day and read the portion that applies to my area of expertise, running. I was saddened to see that Tim jumped on the bandwagon and essentially fell for the trap in buying into crossfit endurance, pose, and the Barry Ross way of sprinting. Ferris is highly educated and while he does make his living finding “hacks” it made me wonder why intelligent people make that mistake. So keeping the examples in mind from Ferris book, let’s look at how people get hooked in and fooled:
1. Establish expertise
They’ve got to establish expertise in a persons mind. Whether it is through personal experience (“Hey I was once like you guys, but now I did this…”), connections with random professional athletes, degrees, or some other way, they’ve got to establish being an expert. And it’s not that hard to do, which is a post for another subject.
The next step is for whoever is presenting the idea or concept to be extremely enthusiastic. We equate enthusiasm with passion, confidence, and trust. The logic is that if this “expert” seems to genuinely believe what he is saying, it must be true. Enthusiasm and passion are good things, but not when it replaces knowledge.
3. Exploit people’s goals - Give them a magic bullet
Make big promises that people can reach those goals that always seem out of reach. Guarantee them that they can and the only reason they haven’t before is because they were given the wrong workouts/information.
4. Tell us what we want to hear - It’s easy
They’ll give people a shortcut. No need to run 80+mpw for an ultramarathon, just do 400’s! This is a common tactic in those wonderful infomercials or diets. Why did Atkins take off in popularity (before plummeting)? Because it was much more appealing to eat all the steak, fat, etc. you wanted than to eat a bunch of fruits/vegetables and the like.
5. Blame someone else
It’s not our fault we are out of shape, overweight, etc. If the guru can shift the blame from ourselves to someone else they gain an upper hand. There are many example sof this in the diet and exercise industry. Right now the trend towards blaming our obesity epidemic on the government dietary suggestions is an example. By shifting the blame to the outside, the guru is telling people what they want to hear. After all who wants to be responsible?
6. Go Against the accepted norms
In looking at the publication of scientific journal articles, there is an interesting trend. Whenever a new theory is tested and some positive results come back, there’s a swarm of papers on the theory trying to substantiate the fancy new theory and go against the previous norm. However, after a while, the tide turns and once this theory gains some ground, the trend switches to more and more papers trying to disprove the new idea. The point is that it’s human nature to like to be the first on a new trend. We like to go against the accepted norms. People will exploit this. Sometimes the norms are wrong, but often time they are the norms for a reason.
7. It’s a conspiracy
The previous two points bring us to this one. If the guru’s go against the norms they have to provide a reason why the norms are wrong. The easiest way to establish this is to develop a conspiracy. People love conspiracies. For example, a common tactic is to blame the government diet or exercise recommendations and say they are controlled by pharmaceutical, agricultural, or any other industry.
8. Hide behind Science and complexity
At a recent track meet I was having a conversation with a friend in college, who made the astute observation that if the coaches inserted random scientific terms to explain things, even if they were totally wrong, the runners seemed to buy into it more enthusiastically. That’s a very common reaction, we all do it. We associate science and complexity with being smart or correct. As I’ve said before…people trying to fool you go from simple to complex…good coaches translate complex things into simple understandable ideas.
The problem is most of us don’t have the knowledge or filter to figure out if what they are saying is correct or not. Even scientific experts don’t have the expertise at times, so how is a normal person going to? These guru’s will blend correct scientific terms (think: Neuromuscular, lactate threshold, VO2max, fascia, energy systems, etc.) and use them either incorrectly or out of context to explain something. Crossfit is a great example of this. They hide behind science, saying there way is based on science. Well, they use a lot of science in their lectures online, but the problem is they don’t understand it and use it wrong. The same can be said for Pose. They rely on a ton of science, and not all of it is bad, but when published in a reputable journal, the response from several biomechanist was nothing short of amusing.
9. Hide Behind numbers and statistics
The easiest way to trick someone is to throw in some numbers and statistics. Most of us (myself included) go a little brain dead when a bunch of stats are thrown out. We don’t ever question what they mean or how they go there. It’s human nature to just accept statistics. If you want to ramp things up even more, throw in a bunch of charts or graphs. Once again, these are very easy to manipulate in your favor (something as easy as changing the scale for example) and people rarely question them.
10. Rely on testimonial
Lastly, a common tactic is to rely on testimonials. It’s a tactic that is designed to make it plausible in the consumer’s mind that they too can reach the same results. After all, if all these “normal” people had success how can they be wrong? This line of thinking prevents us from delving deeper into the claims or legitimacy of it.
I don’t want to pick on any one group or get into the efficacy of any of them, but if you look at the latest trends or fads or whatever you want to call them in the exercise or diet world you can check off a number of the above tactics. Whether it’s atkins, Gary Taube’s book, Crossfit, Barry Ross’ sprinting ideas, High intensity interval training, Tabbata sprints, pose, chi running, supplement companies, SOMAX, or choose your own example, similarities between them all are readily noticeable.
How to avoid the trap
Now that we know how to spot the trap, how can we avoid it? There’s no easy way and this would require a separate post in itself but here are some useful tips in deciding whether or not the particular training method or gimmick is worthwhile or not.
The Stool Test
I’ve discussed the stool test previously, so I’ll just briefly go over it again. My thesis advisor, Jason Winchester, was the one who brought this concept to my attention and it’s a simple yet effective way to decide whether something is worthwhile.
Basically, you have 3 legs to a stool. If you have all 3, well then it works. If you only have 2, it could work but it depends on the strength of those 2 legs. If we only have one of the three, chances are it’s not going to work.
The three legs of the stool are
1.Practical- Does it work in the “real world.” What this means is have you tried and it works or have many others tried it and it works.
2. Research- Is there scientific research on it and does it confirm that it works.
3. Theory- Is there a legitimate, non-pseudoscientific, theory for why it might work.
Look towards the elites
Elite athletes aren’t perfect and don’t always get it right, but the chance that the majority of the successful ones are doing the right thing, is slimmer than looking at a bunch of recreational runners. Why, because elite runners generally have access to the best coaches and rely on maximizing performance as their job. There is more at stake for them to be right.
Secondly, many elites are always looking for an edge training wise. It’s not always the best of the best guys, but the ones a level below who are trying to make that jump. That’s why you’ll see an occasional pretty good athlete try crossfit or pose or some supplement. The key is not whether one guy tries it or not, but if they succeed and then several follow suit and succeed. It’s a copycat game and if someone has a lot of success with a particular training method, it will catch on. It’s why we went from a high interval training program to a higher volume one after the success of Lydiard’s athletes. Lydiard was bucking the norm but it worked, so it rapidly gained popularity.
Similarly, in the high jump, Dick Fosbury completely changed the game. The key again was that it worked and it was soon adapted by others. With fads that don’t work, you’ll see a handful of good athletes try them, but the results over the long term won’t be there and you won’t see more and more elites copying them.
Innovation is key, but the right innovation is even more important. If someone is doing something completely opposite from what the best do and claim their way is better, it’s doubtful. If they are still making that claim after a couple years, it’s even more doubtful because if it worked it would likely have caught on with someone as the results would be their (ala Lydiard or Fosbury). So look towards the elites as a general guide.
Know your history
Last, but not least, know your history. A ton of different training methods have been tried before. Often they are repackaged and sold as new or breakthrough methods. The reality is that they may be useful but there is a reason we have evolved past them. For instance, the very low volume tons of intervals has been tried several times throughout the past 100 years. It doesn’t mean that intervals are bad, it just means that if you know your history, you know that we used to do a ton of them, and through evolution of training that amount has settled down into a more reasonable place. It doesn’t mean that innovation has to stop. It just means don’t throw away 60 years of evolution of training. Use the lessons learned as your guide and evolve from here.
The general pattern in history with anything training related is that with each generation we have a slight blowback against what the previous generation goes. So it’s a constantly swinging back and forth pendulum that gets closer and closer to the center with each generation. So we started with extreme swings of all mileage or all very short fast stuff. Now, we argue over how much and when to do each and the variation between most good programs is more minimal than in the past. The point of all of this is that if you know what happened before and the road that has been traveled, you don’t have to make those same mistakes in figuring things out. You progress…
What’s the take away message here?
In this internet age, there’s a ton of information available. It’s almost too much. So come up with a model to figure out how to separate the worthwhile info and the crap info. I’ve generally found that at first it all kind of flows together, but if you keep at it, eventually you hit a point where it all clicks, and sorting through things isn’t that difficult. For coaches out there, my best piece of advice is to know the basic foundation and use that as your guide. That means, know the history of training, the basics of human movement/biomechanics, and exercise science. If you know how we basically work, then spotting crap science is a lot easier.
*note: a previous version of his post was originally published by Steve Magness on The Science of Running on May 10th, 2011. It has since been very slightly updated.