If you’ve been following along here on the HILARIOUS! Journey, you’ve probably picked up the sense that I’m a science geek. Given that all healthcare, hilarious or not, is supposed to be informed by science, no surprise, right?
However, I gotta tell you that whole “all medicine is rooted in science” thing is … a myth. The medical profession is working on that, but it ain’t soup yet.
Anyway, this week I’m not talking about just medical science, but science as a whole.
Let’s lay some groundwork first.
The Wikipedia definition of science is this: “is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.”
The Webster Dictionary definition is loooooong AF, this is the nut graf for this discussion: “knowledge when it relates to the physical world and its phenomena, the nature, constitution, and forces of matter, the qualities and functions of living tissues, etc.; - called also natural science, and physical science.”
Tl;dr = “science” explores, tests, and organizes knowledge so that it can be continually tested. Those explorations and tests are called “experiments” or “research,” the organizing of it is called “publishing,” and the testing is called “replication.”
So, what scientists do is come up with ideas for experiments, run those experiments, write up a report on the results, publish ‘em in a scientific journal, and then other scientists play around with the same experiment. Those experiments can serve up anything from new medical treatments to new consumer products to new industrial process to … almost anything.
Science is how we learn new stuff about the world. Animal, vegetable, mineral - everything gets the science treatment.
Most of the time I spend on the web is taken up with reading about science. I get frustrated regularly by the fact that most published science is paywalled, so you’ve either got to pony up somewhere between thirty and fifty bucks - or its equivalent in whatever national currency the publisher charges - to read an article. There’s a push for what’s called open access publishing, which means that anyone can read it, no cash payment required.
Since most science is funded by governments in some way, any citizen should be able to read it, right? We paid for it, we should be able to read what we already paid for without handing over another chunk of change.
But I’m not focusing on open access this week. I’m going to dive in on how science itself appears to be broken, or at least badly wounded, due to lies, damned lies, and statistics.
Now, I’m not super savvy when it comes to understanding statistics, so don’t think I’m planning on going all stats-wonky here. Hell, dudes, I’m working my way through a sixth grade statistics prep course online right now, and the struggle is real. But using statistics to crunch and interpret information - data - gathered in scientific experiments is kinda critically important to the whole exercise. So work with me here.
OK, OK, I can hear you saying, “Casey, you said there would be DATA THUGS in this thing!” And you’re right. And here they come!
I first tripped over the concept of data thuggery - damn, that’s a tasty phrase - when they were called research parasites by a couple of medical poobahs in the New England Journal of Medicine back in 2016. The article in NEJM was specifically talking about medical research teams working together symbiotically rather than just letting any Science Bob, Carol, Ted, or Alice parasite up all their hard work and use it for … whatever Bob and Carol and Ted and Alice want, research-wise.
The theory there boils down to “this is my ball, I paid for it, you can’t play with it,” to which others on the science playground say, “wait a minute, dude, that ball is something WE ALL PAID FOR, so give it here.” Given that science is paid for by governments, therefore citizens, data sets that are part of that science are … kinda public property?
Anyway, data thuggery. Get used to that phrase, I’m already in love with saying it.
FINALLY WE’RE AT THE THUGS - and here they are. Or at least the Big Two, as far as I’m concerned - James Heathers and Nick Brown. James came to my attention as the brains behind one of my favorite Twitter profiles, @JustSaysInMice, where you can get daily updates on what screaming headlines about medical breakthroughs are really … just in mice, not in humans. At least not yet. Nick came on my radar screen on James’ coattails. They’re both pretty terrific at busting on bogus science.
You don’t have to take MY word for it. Science magazine, the house organ of the American Association for the Advancement of Science, did a piece on Nick and James, “Meet the ‘data thugs’ out to expose shoddy and questionable research,” that did a great job of capturing just how much fun they’re having, goring scientific oxen all over ever’where.
Here’s a slice from that Science piece:
“The two watchdogs have been remarkably effective at uncovering problematic publications. So far, Brown estimates that the analyses he and Heathers have done, sometimes working independently and often with other collaborators, have led to corrections to dozens of papers and the full retraction of roughly 10 more. That total includes five papers retracted over the past year or so by Brian Wansink, a high-profile nutrition researcher at Cornell University.”
One of the pull quotes on that one is from Heathers, saying:
“In short, peer review misses all the hard stuff, and a worrying amount of the easy stuff.”
If you don’t know what “peer review” is, that’s evaluation of a person's work or performance by a group of people in the same occupation, profession, or industry. In other words, it’s science-professional “check my work.” Peer reviewers aren’t paid to review stuff, same way journal article authors aren’t paid. But scientific publishers have a profit margin that beats Apple’s. Go figure.
Anyway. Back to data thugs.
In case you’re thinking that data thuggery involves just snarking on the Twitters, au contraire, mon frère - it’s work.
These guys came up with something called GRIM, or granularity-related inconsistency of means, which boils down to “glorified adding up” in Heathers’ own words - GRIM simply asks do the numbers in this paper add up? It works in research studies with small sample sizes.
They also worked up something called SPRITE (sample parameter reconstruction via iterative techniques), which susses out “statistically possible data sets from the means and standard deviations reported in a study.” In plain language, they take the number sets in a study and run them to see if they add up statistically - do they make sense? Read the Science profile for a great capture of what SPRITE has done to headline-grabbing nutritional science studies - it’s hilarious. Also hilarious? The takedown of equally headline-grabbing gender studies.
Before you think that there are just two lone gunmen out there data-thugging it up in the science world, nope. These guys, particularly James Heathers, got my attention because their contributions aren’t just worthwhile, they’re funny and snarky AF. Which is my very favorite thing, as you’ve no doubt figured out by now.
No, there are data parasites, thugs, and detectives out there hunting down bad science, and forcing a rethink or straight-up retraction of a rising number of scientific papers. And there needs to be more of them.
There’s an emerging type of science called meta-research, or meta-science, that’s all about checking the work. And it needs to be supported. The challenge is the academic gladiator arena is a little, um, rough on folks who show up and say that someone’s work doesn’t add up.
In a piece for Nature’s World View, “What ‘data thugs’ really need,” Keith Baggerly talks about the risks involved to data detectives working to uncover bad science, pointing out that the detectives’ careers can get tanked if they fall afoul of academic politics in the process.
In another Nature World View piece, “A toast to the error detectors,” Simine Vizire calls for 2020 to be “the year in which we value those who ensure that science is self-correcting.” She says that:
“Researchers are often warned against pointing out errors — and sometimes kindness is used as justification. They are told to focus on improving their own research, or to state only the positive aspects of research done by others. If you don’t have anything nice to say, don’t say anything at all.”
I’M SORRY, WHAT? If a scientific study can be shown to be flawed, or just plain wrong, RETRACT IT. If it just floats out there in what’s called “the literature,” it gets baked into “what we think is true” and can echo for eons, maybe even costing lives, if it’s medical research.
Here’s a sobering example of just how much “wrong” is floating around out there in Science Land. In an article on eLife Sciences, “Meta-Research: A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals,” the authors conclude:
“We have identified 396 medical reversals spanning different types of medical disciplines, types of interventions, and populations. The de-adoption of these and other low-value medical practices will lead to cost savings and improvements in medical care.”
Medical reversal is totally a thing, and it needs to be. Dr. Vinay Prasad, an oncologist who I follow like a stalker on Twitter, has a whole website about it called, funnily enough, Medical Reversal, that he works on with a number of colleagues. He’s also got a book, “Ending Medical Reversal: Improving Outcomes, Saving Lives,” that he wrote with another MD I follow like a crazy person, Adam Cifu.
How does medical reversal, or “finding the stuff that’s just plain wrong, doesn’t work, or might be killing people,” work?
BECAUSE THE DATA THUGS ARE ON THE CASE. As are the data detectives, and the data parasites.
We need people - smart scientists, double bonus points if they can speak in plain language, triple bonus points if they’re funny - to get all up in the data thug life. Start checking the work. Find some data thug buddies, ‘cause there’s strength (and career preservation) in numbers.
If you’re listening to this, and thinking, “jeez, Casey, I’m not a scientist, I’m just a regular human who flunked math,” you can participate in the data thug life by getting on the Retraction Watch email list and spreading the word about what’s been proven wrong in science. I’m also including links to the Retraction Watch site and Twitter feed in the show notes.
You can follow any of the folks I’ve mentioned here on Twitter - all the links are in the show notes, kids, like I do.
My point here? Science is awesome, but since it’s a process involving humans, it’s not infallible. Nor is it sacrosanct, despite the opinions expressed by the Poobahs of the Academy about comportment and “the right way to do things.”
Sometimes you gotta make a ruckus to move things in the right direction.
So … bring on them data detectives, data parasites, and DO NOT FORGET to invite those data thugs.
Danny van Leeuwen, also known as Health Hats - with his diverse and prolific health experience, Danny uses his multiple hats to empower people as they travel toward their best health. To join Danny on that best health journey follow HIS Health Hats podcast on iTunes - it’s terrific!
Wanna be on this list? Become a Patron!
Open + close:
Music from https://filmmusic.io
"The Show Must Be Go" by Kevin MacLeod (https://incompetech.com)
License: CC BY (http://creativecommons.org/licenses/by/4.0/)
Podcast distribution rights: