I know of loads of them (or at least their papers), even if I don’t personally know many. There are tens, (even hundreds!) of nominally qualified people for every full time research slot after all—a different problem. Not all of them try for academic positions of whatever stripe, but still there is a lot of weeding out to be done somewhere. Also, how would you handle decisions for better journals? Even restricting everyone to a paper every three years or so, it’s still not possible to read everyone in related fields. Someone has to be sifting through this, somehow.

Since any single person would be able to publish only one article every three years, you would need a lot of people who would be interested in publishing nonsense for that to become a problem.

Who do you think will those people be? Do you know personally a lot of people who you think would put the time and effort into writing some nonsense just because it will be published in an academic journal? I know no such people. (Of course, that’s just my “worthless” personal experience.)

Your proposal ensures that the debate slows down so considerably that much of its value is lost.

I do not believe that a person is able to generate an original idea that would be of wide interest more frequently than once every few years. The present high rate of publication is a product of the convoluted incentives of the academic system. Note that if a laboratory has 10 people working in it, then by rotating authorship they would be able to publish a paper every 3-4 months.

My experiences, as I’ve been at pains to point out, are equally worthless.

First, I fail to find where in your previous comments you point that your own experiences are worthless. More importantly, your arguments (as those of everyone) are constantly peppered with claims that are not backed up with “quantifiable data with proper controls”: For example, “paper from 2001 is cited in, say, cognitive neuroscience, it is a classic”, or “I do not believe that reasonable people find…”, etc. I could challenge almost every sentence that you make claiming that it is “worthless” without a thorough, and practically impossible, data collection effort.

My own work assumes that the scientific communities are relatively good at assuring quality, it does not directly argue for it. So I don’t think it will be of any relevance to you.

According to your own standards, if you assume that “scientific communities are relatively good at assuring quality” then you must cite some studies with “quantifiable data with proper controls” that show that. The citations in your articles would therefore be of interest in the context of our discussion. Do you have a paper of yours available online?

I do not believe that reasonable people find it offensive to be hold that their personal experiences are useful for evidentiary purposes (unless the claim is that there is something special about their personal experiences). My experiences, as I’ve been at pains to point out, are equally worthless.

My own work assumes that the scientific communities are relatively good at assuring quality, it does not directly argue for it. So I don’t think it will be of any relevance to you.

I won’t argue, but I do disagree: they are worthless. Glad you now see that I did not mean to be offensive in saying that.

It is not a matter of intent, or at least not only a matter of intent. The fact that you allow yourself to use this kind of language – and this kind of thinking – is telling, whether or not you had an explicit intent to be offensive.

I have in mind various adjectives that I could attach to your arguments – yet I choose carefully which ones to apply and which ones to keep to myself. Skepticism involves giving the benefit of every doubt to opinions that are contrary to those you hold dear. Injudicious use of strong and potentially offensive language is not compatible with skepticism.

Goldman’s Social Epistemology

Thanks for the reference – I’ll see if I can find it. Does this book, or any other work that you know, contain “quantifiable data with proper controls”? If so, what kind of data? Also, I am still interested: what is your own work in this area?

Regarding alternatives to peer review – I have made a proposal in the blog post to which I linked above:

My proposed system is, thus, to give each author a quota of publications – say one article every 3 years. The author can choose what work to put in that article – which is guaranteed to be published. Preferably, the article will still be refereed, but it would be up to the author to decide whether to incorporate any suggestions by the referees into the work. A researcher would then choose carefully, picking the best of their work of last 3 years (and incorporating any useful ideas of the referees) to produce the best paper he or she could come up with – with 3 years between publications it makes sense that there will be a lot of good work to choose from and a lot of time to write the paper well. The result will be more enjoyable and productive for the author, for his or her readers, and for the scientific community.

The shortcomings of personal observations are well known. This does not mean they are “worthless”. Rejecting such observations in the face of systematic observations to the contrary may be justifiable. Your attitude – blanket rejection without trying to address specifics, simply because the observations are unpleasant – demonstrates how easy it is to set the bar for evidence at a level that produces desired results.

Again I would like to express my interest in any work of yours or of others in the area. Can you provide citations?

As the politics of global warming show, if some powerful players could gain something by denying basic principles of physics, physics would become just as political as the most political areas of social science.

The only arenas where one could decide the correctness of a statement based solely on rigorously rational grounds are those of purely formal systems such as mathematics, or formal logic. When an attempt is made to apply the formal system to the real world (e.g., physics), non-rigorous considerations come into play and those can be, and to some extent must be, political.

Even when within working within purely formal systems, the decision to accept or reject a paper is not based on the truth of statements made in the paper alone. One can always reject papers that contain only true statements based on claims that they are obvious, irrelevant, unclear, incremental, do not cite the relevant literature, or any subset of these.

The conclusion is that peer-review is, and must be, primarily political.

When I reviewed papers I noticed that the scores I gave the papers were to a large extent arbitrary. It is always easy to set the bar high enough so that a paper would be found in need of revision or more work. It is also possible to set the bar sufficiently low so that only papers that are pure nonsense would be found wanting. Even in the best of cases, when the reviewer has no over-riding motives, personal characteristics of the reviewer (interests, beliefs, taste, and mood) determine where the bar is set in any specific case.

At the anecdotal level, I also had on one occasion seen how a paper with a fundamental error received passing scores by reviewers (and special treatment by the editor) – with the likely reason being that it was written by two prominent figures in the field.

Sortiton, worthless is what I meant. Worthless is what it is.

I’ll ignore this little bit of offensiveness. I cannot see how this kind of language (and the thinking it expresses) is compatible with the detached and unbiased approach you ascribe to scientists. It is definitely compatible with the kind of heightened emotions that I find is typical of scientific politics.

this is a topic on which I have read and written for a long time.

Interesting. In that case, I would like to look at your own work and that of others on this issue. Can you provide citations?

The system is worst at recognizing really new moves in the debate; that is its conservative bias.

This is a direct contradiction to the claim that the peer-review system encourages dissent. Dissent is certain to be seen as a “new move”.

Finally, there is no incompatibility between the dissent and conservatism claims. In order for dissent to be maximally effective, it needs to function against a background of taken for granted claims. The system is worst at recognizing really new moves in the debate; that is its conservative bias. Within the shared background, constrained dissent is normal; real consensus on anything (beyond quite broad claims) is relatively rare.

As two of the authors of a recent document advocating a one-state solution to the Arab-Israeli colonial conflict we emphatically intended to generate debate. Predictably, Zionists decried the proclamation as yet another proof of the unwavering devotion of Palestinian—and some radical Israeli—intellectuals to the “destruction of Israel.” Some pro-Palestinian activists accused us of forsaking immediate and critical Palestinian rights in the quest of a “utopian” dream.
Inspired in part by the South African Freedom Charter [1] and the Belfast Agreement [2], the much humbler One State Declaration, authored by a group of Palestinian, Israeli and international academics and activists, affirms that “The historic land of Palestine belongs to all who live in it and to those who were expelled or exiled from it since 1948, regardless of religion, ethnicity, national origin or current citizenship status.” It envisages a system of government founded on “the principle of equality in civil, political, social and cultural rights for all citizens.”
It is precisely this basic insistence on equality that is perceived by Zionists as an existential threat to Israel, undermining its inherently discriminatory foundations which privilege its Jewish citizens over all others. Israeli Prime Minister Ehud Olmert was refreshingly frank when he recently admitted that Israel was “finished” if it faced a struggle for equal rights by Palestinians.