Re: Why do you believe what you believe?
- From: Burkhard <b.schafer@xxxxxxxx>
- Date: Wed, 30 Sep 2009 10:12:40 -0700 (PDT)
On Sep 30, 8:02 am, Pulse <feralpu...@xxxxxxxxx> wrote:
<snip>
What makes specific secular texts more reliable than other contradictory
texts? Here I'm mostly interested in understanding why many consider
peer review journals so valuable as well as which (if any) other texts
can be considered of equal merit.
OK, here my part 2:
Essentially, any paper is only as good as the evidence that it gives
for the claims that are made. A good paper, academic or otherwise,
should state explicitly and fully the data that support the theory,
and how the conclusions are derived from the evidence. In this case I
can check for myself the validity of the claim if I have sufficient
knowledge of the field.
Ideally, the paper should also communicate the _limits_ of the
findings, be they a result of the methodology, or the data, or other
factors. Where possible, it may also report statistical evaluation of
the claims made, e.g.report known or potential error rates etc. This
however is subject specific and it depends on the discipline. For most
of my own work e.g. this simply would not apply.
If a paper has this information, in an ideal world it should not
matter whether it was peer reviewed or comes from the authors own
website. It has everything in it that is necessary to judge its
reliability.
Unfortunately, this ideal world runs into some practical problems
a) I might not have sufficient domain knowledge to really check for
correctness of the results myself.
b) I have neither the time nor resources to replicate the results
b) means that I can't check if the data is incorrect, either because
of a measuring error, an overlooked parameter etc, or because of
intentional fraud. Depending on the subject matter, it is not always
possible to give the reader access to the authentic data
So once again, I may have to outsource my decision making to other
people. This is where the peer review process comes in. Now, I'm on
record (in a peer reviewed journal no less, ;o)) that for all sorts of
reasons, the peer review process as we know it did a reasonably good
job in the past, but comes to the limits of its useful life. Problem
is that we have nothing better, so far. so some of what follows can be
a bit heretical.
Here is the traditional rationale: The peer reviewer(s) are a) experts
in the area themselves, taking care of problem a). They provide the
expertise to judge the content that I may be lacking, and this
corroborate independently the correctness of the paper. Moreover, the
reviewers are actually your competitors - they have an inbuilt
incentive NOT to let you pass if they can find a reason not to. This
is in particular the case for peer review in research grant
applications - if I recommend this grant, less money is available for
me. .. Now, with 12 years of experience in the system, i can tell you
that as a result, this process can be vicious. When working together
with industry partners for instance, they were always surprised how
nasty academics can be to each other - and that the reviewer remains
anonymous contributes to this. Just like the often much more agressive
tone on internet forums, peer reviewers often don't hold back.
This gives you a rational reason to put more trust into a peer
reviewed article - corroboration without fear or favour from other
domain experts.
The referees will spot obvious mistakes. They will also be immune to
one of the most common sources of research errors, confirmation bias:
a researcher who falls in love with his own theory, and ignores the
counter evidence. The referee is immune from this danger.
Now, one should note that different journals and different disciplines
have different traditions and methods of peer review, and this is one
of the problems: unless you check the journals "information for
authors" page, you may not know which system they are using. But in
theory, the trust you can put in a journal should depend on this, to
some extend at least. For instance, I found it....problematic...that
apparently, in biology peer review is not normally double blind, or so
I was told on TO. In my own discipline, this is normally the case,
though the extend to which journals go to "blind" the submission
varies (in one of the journals I edit, we remove e.g. self-referencing
footnotes, in another we don't) The fear is of course that a prominent
author gets a benefit of the doubt that a newby would not.
Some journals use 1, other 2 or three referees. Some allow the author
to suggest referees etc etc. So there is a problematic lack of
transparency. You can if you are really meticulous counterbalance this
to some extend. You can look for journal rankings, or their "impact
factor" to decide if a journal is likely to be a good one.
A bigger problem is that these days, there are more and more papers to
referee, and less and less funds. So the referee can not replicate the
results, or test them, really. It is difficult enough to get funding
for original research, but close to impossible to get it for mere
"checking" , and the referees typically do it in their own time, and
at their own expenses - as "good citizens of the academy".
But that dramatically limits the amount of rigour I can invest in
checking the paper. I will find gross mistakes. I wil also see if the
data suports the conclusion. But I can'tcarry out my own tests, and
even less discover inetntioanl fraud, unless the author is really
stupid.
As a result, there is at the moment a big debate of how much primary
data the authors ought to submit with the paper, for instance.
Another problem, similar to an issue I discussed previously, is the
increased specialisation: More and more journals for more and more
narrowly defined subjects. This drains the pool of truly independent
experts. If you struggle to get your stuff published, just start your
won journal and put your former PhDs on the board.
There are several other pressures on the system which we need not go
into here.
But as I said, it is difficult to find good alternatives. One thing
you you can and out to do is to go to a place like CiteSeer and see if
the article was cited by other researchers - either approvingly (good)
or rubbishing the claims (bad). This gives you peer review after the
point of publication.
Are there other sources than peer reviewed journals? As I said above,
what really matters is the content, not the process. It should be
published though, so that as many people as possible can subject it to
scrutiny. It may be that for some other reasons, we will need
increasingly in the future use other sources, simply because there is
economic and political pressure on academics to explore alternative
methods to share their information. For instance, research users like
private sector companies will often not subscribe to arcane academic
journals, and there is huge pressure on us to make our knowledge
available to them. Information contained in these new forms of
communication may be th eonly thing youget for a certain question.
One thing I advocated for instance (and use myself) are certain forms
of blogs. Academics who post their findings on a website, and then
have a limited list of other academics as commentators. This works
in effect like a more transparent and continuous peer review system:
You as reader can see what the peers actually think of the paper, not
just at the point of publication but also as time passes on and new
knowledge comes in. This seems to me to imitate some good features of
the peer review system (though it is not quite as adversarial, which
may be an issue) and has some additional benefits.
.
- References:
- Why do you believe what you believe?
- From: Pulse
- Why do you believe what you believe?
- Prev by Date: Re: Is Nashton a medical professional?
- Next by Date: Re: Not only are some posters in here USELESS..
- Previous by thread: Re: Why do you believe what you believe?
- Next by thread: Re: Why do you believe what you believe?
- Index(es):
Relevant Pages
|
Loading
