While I would agree that the prevalence of the problem has been minimized in fMRI during the last 15 years, I disagree that our critique does not hold up. The root of our concern was that proper statistical correction(s) need to be completed in order for research results to be interpretable. I am totally biased, but I think that remains worthwhile.
When we published the salmon paper, approximately 25-35% of published fMRI results used uncorrected statistics. For myself and my co-authors, this was evidence of shaky science. The reader of a research paper could not say with certainty which results were legitimate and which might be false positives.
Hey, I know you got a lot of flack for the article. So, I just wanted to thank you for having the courage to publish it anyways and go through all of that for all of us.
I go back to the study frequently when looking at MRI studies, and it always holds up. It always reminds me to be careful with these things and to try to have other be careful with their results too. Though to me it's a bit of a lampooning, surprisingly it has been the best reminder for me to be more careful with my work.
So thank you for putting yourself through all that. To me, it was worth it.
Many thanks - appreciate the kind words. Thanks also for always working to work with care in your science. It makes all the difference.
Among other challenges, when we first submitted the poster to the Human Brain Mapping conference we got kicked out of consideration because the committee thought we were trolling. One person on the review committee said we actually had a good point and brought our poster back in for consideration. The salmon poster ended up being on a highlight slide at the closing session of the conference!
Thank you for publishing that paper, which I think greatly helped address this problem at the time, which you accurately describe. I guess things have to be taken in their historical context, and science is a community project which may not uniformly follow best practices, but work like this can help get everyone in line! It's unfortunate, and no fault of the authors, that the general public has run wild with referencing this work to reject fMRI as a experimental technique. There's plenty of different ways to criticize it today, for sure.
That was our paper! We showed that you can get false positives (significant brain activity in this case) if fMRI if you don't use the proper statistical corrections. We did win an Ig Nobel for that work in 2012 - it was a ton of fun.
(I mention this so more people can know the list exists, and hopefully email us more nominations when they see an unusually great and interesting comment.)
Interesting -- I just use https://news.ycombinator.com/best?h=168 for a weekly roundup, but that only tracks posts. Might need to supplement it with highlights or similar.
As the first author of the salmon paper, yes, this was exactly our point. fMRI can be an amazing tool, but if you are going to trust the results you need to have proper statistical corrections along the way.
As the first author on the salmon paper, yes, that was exactly our point. Researchers were capitalizing on chance in many cases as they failed to do effective corrections to the multiple comparisons problem. We argued with the dead fish that they should.
I'm the first author of the salmon fMRI paper, if you have any questions. Generally, how the investigators do their statistics can lead to implausible conclusions. Extraordinary claims should require extraordinary evidence.
Wow, thanks for the awesome work! It's my favorite neuroimaging study, and we have a print of the poster on our lab's wall.
How do you find the rigor of neuroimaging analyses have developed since that paper? I don't follow it much, but I've seen some quite wild looking stuff being published (e.g. predicting smallish datasets by feeding voxel activations into a huge ANN). Are my concerns about a new era of overfitting realistic?
That's amazing - you made my day with that statement.
I left neuroscience for the software world back in 2012, so I don't have a lot of data points since then. I know between 2009 and 2012 the field went from ~50% of papers doing the right statistical corrections to about ~90%, which is a huge step in the right direction. I hope those numbers are even better today.
The expense of MRI time means that studies include far fewer subjects than they might want/need. My opinion is that there are still significant challenges that go beyond correction for multiple comparisons, like data peeking and low-power experimental designs. I think that we should move to a mindset where we need replication and convergent evidence for major claims. Not a single study with 18 college freshman participants.
Pretty much what TeMPOraL said. You can scan pretty much anything with fMRI and find results if you don't use proper statistical corrections. I have found "significant" voxels in a pumpkin before while doing testing. Our argument was/is that scientists need to have appropriate rigor in their analyses, otherwise you can reach ridiculous conclusions - like a dead fish looking alive...
Thanks for the kind words. I am the first author of the "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction" paper. Happy to take any questions here. A link to the original poster: http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
It would be nice if the cost of an MRI was so low you would typically get a cheap one as part of your yearly physical and if anything popped up they could do it again in an expensive, high powered one to verify.
I've scanned about 300 people as part of my research career. The director of the imaging center reviewed every anatomical scan. From that group of 300 we informed about three people that they had an anomaly which should be examined by a doctor.
Yeah, that's what I'm talking about. Sure, it was 1% that needed further validation but that 1% is so much cheaper and easier to treat when its caught early vs. later on when it's noticed by the patient.
MRI's becoming commonplace, even if it were every 3 years instead of annually would be a useful tool to improve health outcomes across the board.
Marc Abrahams, organizer of the Ig Nobels, asked us for a salmon recipe to include in a cookbook they were publishing. We sent in a single page recipe for how to cook a salmon in an MRI scanner by overriding the safety protocols. That was fun to write.
My wife and I are so irritated about this. She bought a Cricut Maker last year to make masks for friends and family. Custom images were cut for every person she made a mask for. Now, nine months after we bought the machine, we will have to start paying $10/mo for the same privileges we enjoyed when we first got it.
Good question. Ultimately, in fMRI we are looking for changes in the local ratio of oxygenated to deoxygenated blood. As neurons begin firing in a brain region more oxygen-rich blood is dumped in by the circulatory system. If the salmon really was responding to the pictures there would need to be blood flow for these changes to be detectable using fMRI. Since the salmon had been prepared by the supermarket there was nothing to circulate blood around.
We just had the one salmon to scan, unfortunately. Regardless of the pre-existing condition of the salmon (zombie, rust, etc) there were no significant voxels of activity once we applied proper statistical corrections. This strongly points to correlated noise as the cause of the uncorrected results.
Thank you for responding.
I have been very curious on whether Rusts are active in salmon. I was not serious about zombie salmon though. One salmon is a rather small sample size regardless of the common sense reasoning around the subject at hand.
