I’ve known a lot of scientists, including a number who’ve quit in disgust, and this description of how so much of science works today matches what an awful lot of them will admit privately if they won’t state it openly.
I continue to view it as genuinely scandalous that “peer review” got hijacked as the way to determine who got money. It’s so rife for corruption, abuse, and conflict of interest it’s ridiculous. But it won’t get better until it’s reformed, and who, in the current system, has the cojones to demand real reform?
It’s about institutions, not the scientific method, far too much of the time.
{ 24 comments }
It’s all about being able to hand out other people’s money.
That’s not even the real problem.
The real problem is lack of transparency and lack of accountability. Things that would be considered absolutely reasonable in any other business or government funding context are treated like they’re gross insults to the people who control this money, and everybody else meekly complies because they’re intimidated by and/or revere the word “scientist,” like a priestly caste.
The fact that it’s actually science most of all that’s hurt by this flies right over people’s heads.
There is no question that there is great potential for corruption in the peer review of grant proposals and papers for publication. However, reviewers do not just get to say no. They have to outline reasons for rejecting a paper or grant proposal. Most journals send papers to multiple people to review. When reviewers say a paper should not be published. it is ultimately up to the editor to decide if he believes the reviewers or the authors. With grant proposals, the funding agency has to evaluate, and most funding agencies have some scientist to help in that evaluation process.
Scientists who have trouble getting papers published or getting grant proposals, frequently blame corruption in the peer review process. My opinion is that there is some corruption, but it is much less than what many people claim.
While the current process has problems, the question is how would you improve it? In the internet age, anyone can post their papers on the web, even if they cannot find a journal that will accept them. You cannot suppress the ideas.
I’ve talked to too many scientists in too many unrelated fields; it appears that the peer review process is broken because you -may- have to send it to multiple “peers,” but the multiple peers will often all be people who hold to one prevailing opinion and whose incomes are dependent on not rocking the boat. It is -inherently- corrupt in that fashion.
There are multiple solutions to the problem that have been proposed. One which has been mentioned repeatedly is to require peer review teams to be multidisciplinary and to include people who are clearly disinterested financially and otherwise; mathematicians, statisticians, computer scientists, etc. can all evaluate data in a disinterested fashion.
Another is a formal appeal process by which researchers who believe they are being subjected to professional or intellectual discrimination can be heard and have their claims evaluated by, again, disinterested parties with no vested interest in the question.
Because everybody knows that in certain fields, the “anonymous” multiple peers all really know each other and know and recognize each others’ work, including the work of scientists they don’t like. The anonymity is paper thin and the enforcement mechanism for ensuring that is weak.
Multiple proposals for review have been made, Mike, by multiple scientists. The ones I’ve outlined here are among the proposals. All are routinely ignored, so far as I can see. And the excuses are quick: it would slow the system down, it would be “affirmative action for bad ideas,” and other excuses, all of which look to me like nothing but excuses to add transparency and accountability to a process that is neither at the moment.
When you lack real transparency and real accountability, that is INHERENTLY corrupting, and those who are part of it have a built-in conflict of interest which causes them to resist any reforms. It is always so, in any human endeavor. ANY human endeavor. The notion that scientists are somehow immune to this, because they are “pure” because science itself is an inherently noble and intellectual calling is fatuous. Corruption is HUMAN, and all scientists are, last I checked, human.
I share the opinion that controversial work in controversial fields can meet with some steep institutional barriers. However, part of the problem with peer review is that there are already too many people outside the field evaluating proposals. To an outsider it may seem like anyone with a Ph.D. should be reasonably equipped to evaluate most proposals that cross their desks, but this is absolutely untrue. Science is too broad and too deep these days for that to be the case. To give just one example, I once had a grant rejected because the reviewer questioned that Raman spectroscopy could be done using near infrared excitation. The person was obviously not in the field, but knew enough from basic physics to know that scattering goes as 1 over the 4th power of the wavelength. Therefore the longer the wavelength, the less scattering. However, the reviewer obviously didn’t know that in most biological specimens, the gain in scattering efficiency at lower wavelengths is offset by the sharp increase in absorption, and that people have successfully been doing near infrared Raman work for 20+ years.
There also already are formal protest channels within federal grant funding agencies. And although you never know exactly who among a grant panel gave your paper the close review, at least at NIH the makeup of the panels themselves are public knowledge.
In very small fields like climate science there may really be a “brotherhood” or whatever. But in large fields like cancer research, neuroscience, etc., you honestly can’t know very many of the players personally. And if you look at the RFPs from most funding agencies, they’re pretty broad. Like I don’t submit my work to a Raman call, because there aren’t any. I usually end up submitting to NIBIB or some general biooptics call at NSF. That means that there’s almost zero chance that my proposal will be read by somebody I know, or even somebody I’ve heard of. On the last panel I submitted to, I was familiar with the work of about two of the 30 panelists.
None of that is to say that peer review can’t be improved, even substantially improved, but I would not call it “broken,” either.
I don’t see how the suggestions you’ve proposed increase either transparency or accountability, though. And whether you’d even want those qualities in a peer review system. For example, when I get asked to review a paper by some hotshot in the field – I’m able to give that paper an honest review, even if it ends up being negative, because there’s no transparency. The authors get to see the text of my review (and have the chance to respond both to me and to the editor) without my name being revealed. If the process were transparent, I would worry that judging this person harshly might bite me in the ass next funding cycle. Accountability is another sort of vaporous concept. Accountable to whom, and for what? Will the science police come and take my membership card away if I accidentally miss an error in a paper or proposal I’m reviewing?
Again, not trying to say that peer review is even close to perfect, or that there aren’t ways in which it can be improved. I would say that the main way it could be improved is for grant competitors to get a chance to respond to reviews instantaneously, before funding decisions are made. Usually when you submit a proposal, 3 people from the grant panel read it closely and their thoughts are shared with the larger panel who then discuss the more meritorious proposals in depth to decide where funding lines should be drawn. It would be helpful if the grantees could see the comments of the close-read reviewers in advance of the larger panel meeting, and get a chance to submit a 1 or 2 page response to the comments to correct any obvious errors (like the “can’t do Raman in the NIR” fallacy).
Generally speaking grantees can resubmit a revised proposal to the same panel that responds to the reviewer concerns – but at that point the larger panel is probably already slanted against you and it’s a needlessly uphill battle.
The accountability and transparency mostly needs to be to the taxpayers who are doing the paying most of all, and taxpayers have substantial reason to make sure their money isn’t being ill-spent on bad ideas or being locked up by a priesthood that locks out qualified scientific dissenters from receiving funding.
If there is now a system in place where a grant applicant believes he is being discriminated against for personal reasons, or for reasons of financial or other conflict of interest, I am a bit surprised to learn it; no such system existed years ago when I first heard about proposals for reform (probably around 2003).
As for the multidisciplinary question: making teams multidisciplinary will of course involve mistakes being made, and some things being slowed down because a reviewer didn’t know something basic. But I think that’s inherently better than allowing nothing but an insular team of people all “in the know” control everything.
And as I’ve said many times, mathematicians, statisticians, computer scientists, et. al. are qualified to review things based on those aspects.
Ultimately we should not allow a tiny minority of researchers in rarified field to hold absolute sway, they must be accountable by independent audit of their work in some fashion.
I can refer you to several scientists more intimately familiar with the system and its ins and outs for more specific details on what exactly should be done; what I continually observe is that most or all proposals for reform pretty much instantly get shot down with condescending responses, and outrage at the notion that taxpayers should be able to ask hard-nosed questions about any of this stuff.
It’s certainly not my intention to shoot anyone down or condescend. As I said, reforms can be made and are probably overdue. However, AFAIK complaints or petitions against unfair reviews have been around since at least as long as I’ve been aware of how grant agencies work – so from about 2002 onwards. It was my impression that such channels had been around forever, but I’ve never thought to ask.
The problem with asking a mathematician to evaluate, say, a biological grant proposal is that grant proposals by their very nature do not include a lot of numbers and hard data. You’re looking for funding to do work whose outcome is uncertain. Grant reviewers then look at the proposals and try and figure out which ones have the greatest chance of success based on a researcher’s proven track record and based on the validity of their a priori hypotheses. Often times a proposal will include some promising preliminary data – but it typically is on very small sample sizes. Therefore, asking a mathematician to comment on a proposal that is 98% biology and 2% math doesn’t seem especially wise.
Here’s an example from Talk-Polywell. The thread started with people who had found a book on super-conductivity by a disgruntled scientist who could not get funding. What’s cool is that the scientist showed up in and began discussing his ideas.
Wasn’t the Galileo controversy actually a failure of peer review?
Yours,
Wince
Zach: Oh, I did not intent to imply at all that you were being snotty or condescending or dismissive. Far from it. I have rarely (if ever) found you to behave in such fashion, and you have not done so here.
I was making generalizations based on what people in the trenches in multiple fields have said to me or written to me, or said or written in other forums. I’m not -in- the trenches, I only know what it looks like from the outside, and it looks riddled with holes for both monetary and just plain logical corruption to take hold.
I don’t pretend we can ever permanently fix anything, after all, we are talking about human beings here. Still, as taxpayers, I think more of us should be willing and even energized to step up and say “wait a minute, we’re paying for this” and not merely meekly accepting that them science folks is just way too smart for us so we shud jest shut up’n lissen.
Wince,
That’s a great link.
More a failure of censorship by the Inquisition.
The article leaves out some relevant information. Pope Urban VIII who was elected pope in 1623, was a friend of Galileo’s and had opposed the condemnation of Galileo in 1616. After Urban’s election Galileo wrote the “Dialogue Concerning the Two Chief World Systems”, which purported to be a balanced presentation of arguments for and against heliocentrism. The 1616 condemnation order prevented Galileo from holding or defending heliocentrism, but it did allow Galileo to discuss the heliocentrism hypothesis. The Dialogue was crafted to fit through that loop hole.
The Dialogue was approved for publication by the Inquisition, perhaps because of Galileo’s friendship with the Pope. After its publication, those opposed to Galileo and heliocentrism pressured the pope to take action, suggesting the pope was being soft on defending the church. Opponents pointed out that the main character in the Dialogue arguing against heliocentrism was named Simplicio. Galileo claimed he was named after Simplicius, a Aristotelian philosopher, which is Simplicio in Italian, but Simplicio frequently implies simpleton in Italian. Even though the book had been approved by the censors, opponents argued it was not a balanced presentation at all, but an advocacy of heliocentrism.
The problem Pope Urban had was that his own Inquisition censors had approved the publication of the book, so he had to claim that Galileo had tricked the Inquisition into approving the book.
Galileo was held under house arrest for the rest of life. That had a very chilling effect on science in Italy and in the rest of Europe that was under Catholic influence. The scientific revolution moved to protestant Northern Europe and England, where scientific theories could be published and discussed without worrying about the church.
mikeca,
> More a failure of censorship by the Inquisition.
Think more broadly.
The Inquisition served pretty much the same function as peer review does now – filtering out bad ideas. The notion of imprimatur
is not essentially different from being published in a peer reviewed journal.
Once you have noticed this similarity, what happened to Galileo looks almost exactly like what happened to Peter Deusberg. The politics are remarkably congruent.
Mc Kiernan,
Bing
Googleis my friend!Yours,
Wince
P.S. Correction made. I keep forgetting which search engine is currently my default.
Actually, Wince, in the 1600′s he was affectionately know as Peter Deoberg. His father was of course the famous physician, Deum de Deoberg.
Mikeca: You’re leaving a couple of other bits out. First, Copernicus had already published his heliocentric theory with the approval of a prior Pope, and not been condemned–indeed, that Pope (Leo XIII I think, but I can look it up if anyone cares) had encouraged scientists in the Church university system (the Church had founded every university in Europe at the time) to discuss it. A later Pope then got mad at Galileo.
The trial by the Cardinals came up with a document of dubious origin claiming Galileo had already been told that this theory was heresy and not to be taught, and accused him of ignoring THAT document. Which still no one can quite account for; Galileo apparently pissed a lot of people off and it may well have been forged, he claimed he’d never even seen it.
And in truth, his personal friend, the Pope, had punted the matter to the Cardinals because he, apparently, believed himself to have been personally insulted as “Simplicus.” And he mostly kept out of it, allowing the Cardinals to rule however they wanted. And they ruled on the basis of that dubious document, without even (apparently) realizing that by saying this theory was heresy, they had pretty much just condemned a prior Pope of the same thing, implicitly.
They convicted him on the basis of an apparently forged document.
Discussion of the heliocentric theory was allowed before Galileo’s trial, and afterwards.
It also bears noting that just by allowing it to be discussed, the Catholic Church was roundly criticized by Protestant leaders of the time.
Make of it all what you will; the popular mythology on Galileo appears to have been more important than what really transpired: a political battle and a Pope who’d felt snubbed and insulted. Shameful, but not -quite- how it’s usually portrayed.
In any case, I would grant Wince’s point that the whole matter appears eerily familiar to what happens to scientific dissenters today, only now it’s powerful scientists with money and pride on the line who are able to stifle dissent. In some fields, anyway. And it stinks to high heaven as an outside taxpayer watching it happen.
I read recently that in the AIDS wars, for example (since infamous Duesberg is brought up again, even though he is far from the only scientific dissenter banished), the establishment is just about to finally embark upon an experimental campaign where they allow some people diagnosed HIV+ to be treated without drugs, with extensive followup, to see how that goes. Gee, after we come up with enough HIV+ people who are 20 years+ drug free and alive, you’d think it’s just about time, wouldn’t you? The old guard is dying off or retiring with their incomes secure, there’s not so much money in the alleged pandemic any more, and it looks like oh, well, maybe some “rethinking” is in order. Such a surprise.
Mc Kiernan,
> Actually, Wince, in the 1600′s he was affectionately know as Peter Deoberg.
Your Google-fu is amazing! When I try Bing all I get is French!
Yours,
Wince
What happened to Galileo has to be view through the context of history. Martin Luther wrote the 95 Theses in 1517 to start the Protestant Reformation. The Catholic church responded with the Counter-Reformation starting with the Council of Trent (~1534) and the Roman Inquisition. The Roman Inquisition was started to suppress the spread of Protestantism in Italy.
Galileo advocacy of heliocentrism was viewed as simply another challenge to the authority of the pope, like Protestantism and he was brought before the Roman Inquisition.
Galileo was a pioneer of the scientific method, and one of the first to understand the relationship between theory and experimental science.
Peter Duesberg had little trouble getting his views published in the late 1980s and early 1990s, but as time went on, most other scientists in the field felt that Duesberg failed to present good evidence to support his theories and that the available evidence mostly supported the HIV-aids link.
This is clearly a highly charged issue with many public health consequences. If Duesberg is correct, then many people may be dying because they are using anti-HIV drugs. On the other hand, if Duesberg is wrong, then many people following his advice may be dying for lack of treatment.
I think it is completely misguided to equate what happened to Galileo with what has happened with Duesberg. Galileo was placed under house arrest for the rest of his life, and his book was placed on the banned book list for hundreds of years because he challenged the authority of the church. Although Duesberg has had trouble getting papers published and funding for his work, he is still a professor at UC Berkeley. Both Duesberg and his critics have tried to use the scientific method to convince the other side. The HIV-aids advocates have simply been much more successful at convincing scientists than Duesberg has been.
I don’t know what similarity you are referring to. Whether HIV causes AIDS or not is not a political question or religious question. It is a question of fact. One can design experiments to test the hypothesis that HIV causes AIDS or the Deusberg hypothesis. Most scientists believe that those experiments have been done, and they show the link between HIV and AIDS.
Wince,
You need Wiki. Need to remember latin. Look up Credo. See nomenclature.
Correctives here are: Dues berg is not Deus berg… although the pun applies as some think he is Deusberg.
Deum de Deo, lumen de lúmine, Deum verum de Deo vero, génitum, non factum, consubstantiálem Patri: per quem ómnia facta sunt.
Mike: Galileo advocacy of heliocentrism was viewed as simply another challenge to the authority of the pope, like Protestantism and he was brought before the Roman Inquisition.
I don’t understand why you put it this way. As has already been established, heliocentrism was condemned by Protestants, while the Pope had, prior to the Galileo incident, promoted it as worthy of discussion throughout Catholic universities and among Catholic scholars. The heliocentric theory was not a challenge to the Pope’s power.
And the Pope actually ASKED Galileo to present arguments for heliocentrism. Which the Church had already said was OK to discuss long before Galileo even came along. They’d said it was OK to discuss, repeatedly. Including telling people to look at and disseminate and discuss Copernicus’ heliocentric ideas.
Galileo got in trouble because he began making broad THEOLOGICAL statements about what his heliocentric theory meant, AND, he insulted the hell out of his former friend, the then-sitting Pope. And apparently, a forged document was used to accuse him of something he was innocent of. Probably, by people who didn’t like him and wanted to see him get in trouble.
Galileo did indeed live out his life in house arrest, which was wrong. But it was a luxury villa, where he was afforded lodgings, living expenses, and a paid assistant, and allowed to continue his work for the rest of his life under patronage of the Church and without persecution.
These days, those who challenge scientific authority don’t get arrested. They merely lose their jobs and have their ability to even get funding or published destroyed.
How this affair does not resemble to you the case of modern researchers who’ve been cast out of our universities and research institutes because powerful well-funded and politically connected researchers at various organizations don’t wish to tolerate dissent, I don’t know. It’s an obvious comparison to me, and fits many researchers not just one. Examples are manifold of modern researchers put through the wringer like this. Just look at Semmelweis, as someone who was persecuted but correct, which was a century ago but fairly modern and having nothing to do with any church. Are things really all that different today? I don’t think so.
Tom, I tried your link to the Polywell discussion but it just took me to the main forum page. Which thread is it? Do you have a more specific link?
Copernicus first privately circulated a manuscript explaining his theory in 1514, a few years before the start of the Protestant Reformation. By 1532 Copernicus had essentially finished “On the Revolutions of the Heavenly Spheres,” but was afraid to publish it.
In 1533 a friend of Copernicus gave lectures in Rome explaining the theory. Pope Clement VII heard the lectures and expressed interest in the theory. A few years later a cardinal wrote Copernicus encouraging him to publish his theory. Copernicus was still afraid to publish. It was not until just before his death that Copernicus finally allowed the publication. It was published in Nuremberg , Germany, which by the 1540s was a Protestant region. I doubt that publication required the Pope’s approval.
I think the treatment of Galileo was a result of the Counter Reformation and the Roman Inquisition which increased the influence of fundamentalist. Before the Counter Reformation got started, the Pope and many Cardinals were more open minded and did not not see heliocentrism as a threat to the church.
Well, Galileo was certainly challenging the Pope’s theological authority, because that’s what he was condemned of, not of heliocentrism itself.
It’s never been clear to me what exactly Copernicus himself was worried about, since as you said he’d been encouraged by Church leaders. But you probably put your finger on it: he lived in Germany, and German Protestants in the very early days of his career strongly condemned the theory. Yes, later they were cool with it, but they strongly condemned it early on, Calvin and Luther both, and Copernicus spent most of his life in Germany with that as the backdrop.
That by the 1540s Protestants were cool with it is good, but for MOST of his working career he’d witnessed a lot of controversy on it fairly close to home–and I only bring that up because it’s tiresome to keep hearing it was simply Catholics who viewed heliocentrism with hostility. Obviously, I’m biased, but still.
This all, again, gets us to the whole issue being primarily political at the time. I suspect that’s what Copernicus was really trying to avoid: the politics in general. Plus he may have been just plain insecure about his work. Luther started the Protestant ball rolling only a few years after Copernicus started privately circulating his theories informally; the Reformation would definitely have affected his view of how safe it was to publish openly for much of his life, since Protestant leaders in Germany condemned the idea. He probably wanted to keep out of the whole affair. It would not have seemed very clear at all in the 1520s or 1530s in Germany what might happen to him; Protestants and Catholics were killing each other, and Protestants killing other Protestants too. He probably wanted to lay low in general. I can’t say I blame him.
mikeca,
Here’s how to get my point:
First, put on your sociological pattern matching hat. Next, cease swimming in the sea of details, so you can see the forest rather than just the trees. The similarities between Galileo and more recent scientific heretics, like Semmelweis, anyone who disagreed with Lysenko under Stalin, Duesberg and Lomborg are readily apparent. Theological, political and scientific heretics are all treated pretty much the same, using pretty much the same methods and pretty much the same sorts of institutions. Humans don’t change.
Science is not reality. Science is the map. Reality is the territory. The map is not the territory. This is even true when the map is scale 1 to 1.
Scientific heretics claim the commonly held map is wrong. Some scientific heretics produce maps which match reality better than the commonly held map. This does not keep them from being persecuted as hetetics. Some scientific heretics produce maps which match reality worse.
All the details you have brought up, while for the most part true, are entirely beside this particular point.
Yours,
Wince
Wince,
There have always been scientific heretics. Einstein was a heretic. Einstein basically said the Newtonian physics was wrong, but Einstein proposed specific experiments that could be done to verify his theory. Many of those experiments have been performed and they do verify Einstein’s theory. Einstein is no long a heretic. He is the conventional wisdom.
By they way, they still teach Newtonian Physics because it is good enough to build bridges and buildings. You don’t need Einstein’s improvements for normal engineering here on Earth.
In astronomy there was the 1920 Shapley – Curtis debate about the size of our galaxy and the nature of spiral nebulae. Shapley had developed a new method of determining the size of our galaxy, and found it many times larger than previously believed, and that the sun was not near the center. Curtis had spent his life studying spiral nebulae and believed they were external galaxies made of of stars. If Shapley’s size of our galaxy was correct, then spiral nebulae would be unimaginably far away. Curtis advocated that our galaxy was much smaller and the sun was near the center.
This debate is famous in part because both sides were partially right and partially long. Shapley was right about the size of our galaxy and the sun location in it. Curtis was right about spiral nebulae being external galaxies, they just were further away than astronomers could imagine in 1920.
How was this debate settled? Largely by building larger telescopes and using them to gather more data. Shapley may have been a heretic in 1920. His size of our galaxy contradicted the obvious method of simply measuring star densities and distances. In 1920 Shapley had no good explanation for this discrepancy. In the 1930s interstellar dust was discovered, and then astronomers realized we cannot see most of our own galaxy because it is obscured by dust.
Science has always been advanced by heretics, but scientist challenge heretics to provide evidence for their theories. If they can present evidence, the heretics have become the accepted theory over and over.
As far as I can tell, Peter Duesberg has just not presented enough evidence to convince very many other scientist.
mikeca,
Now we are on the same page!
> Science has always been advanced by heretics, but scientist challenge heretics to provide evidence for their theories. If they can present evidence, the heretics have become the accepted theory over and over.
Exactly. Wish I’d said the same thing above.
Yours,
Wince
Comments on this entry are closed.