Re: FEX submission reviewers under close scrutiny
- From: "John D'Errico" <woodchips@xxxxxxxxxxxxxxxx>
- Date: Thu, 3 May 2007 09:49:59 -0400
Yair Altman wrote:
Both of you were apparently wasting your goodwill and time on an
immature self-absorbed hobbyist hiding behind anonymity. Some
people
simply fail to learn from constructive criticism.
To the point: reviewing is indeed subjective to a large degree.
While
it is truly the right of reviewers to give any rating they wish, I
suggest that a guideline in the FEX rating popup might be useful in
synchronizing ratings of different reviewers & different files.
Something like:
1=Poor: no help comment at all and/or program repeatedly crashes
and/or bad side-effects (e.g., "clear all")
...
5=Excellent: extensive help comment; extensively documented code;
supplied sample usage; non-trivial functionality
This won't stop rants such as the ones from this xy guy, but it
would
help us to decide whether to download file A or B.
Yair Altman (true name & email...)
I agree that a clearer desciption for
ratings would be useful. A problem is
there are many dimensions of any code.
The file exchange review system is a
simple peer review system. When you
post your work in a public place where
reviews are invited, you should/must
be willing to accept such a peer review
process.
Let me now state for the record that I
have written the most awful dreck in my
career. My first attempts at software
were pure spaghetti code, uncommented,
impossible to debug. I learned to write
(initially in fortran) from someone
who wrote that way as a matter of habit.
Then it took me years of unlearning how
to write such miserable dreck. I did
so by seeing what worked. I found that
if I commented my codes liberally, I
could return to them in the future and
fix bugs that crop up, or extend my
codes more easily. I found that modular
codes are easier to write and debug.
I found that when I wanted to use even
my own codes, the presence of good help
was a major virtue.
The simple fact is that had I had
someone sit me down 30 years ago and
explain to me what good code looked
like and how to make my codes better,
I would have been at least 10 or 20
years ahead of the game. My goals on
the file exchange are (in no special
order)
1. Provide useful, professionally
written tools of my own for others to
use.
2. Teach others to improve their codes
by my own examples.
3. Explain to others how they could
make their codes more useful to others,
hopefully short-circuiting the long
learning process that I went through.
4. Provide consistent, understandable
ratings for other individuals who need
to find a tool to solve a problem.
When I rate a file, I download it. I
read the website description. I try
to read the entire file, looking first
at the help. I look for the presence
(or lack) of several important things,
like checks for errors, an H1 line,
internal comments, help that explains
how to use the code and what the inputs
and outputs must be. Is the code
efficiently vectorized? Are large
variables preallocated where that is
appropriate? I'll look for obvious
problems. I'll look at what mlint
tells me, as it often picks up obvious
problems. Finally, I'll try to test it
out. I'll often spend 30 minutes to an
hour on a single file, and sometimes
more by the time I've finished writing
my review. I do try to give an author
a fair assessment of their work.
In some cases, I've even taken a file
that I felt had considerable merit,
but merely poor help, and wrote a new
version with help attached to it, then
sent it to the author. If I do this,
it is without ownership, the author is
welcome to post it as their own. In
more than one case when I have done so,
the author then went on to repair their
other works once they saw how easy it
was to write better help.
I have in some cases written my own
versions of a code, to see how I might
solve a problem. For codes that are
newly placed on the FEX, I will not
post a competing code. I will not post
an improved version of something that
the author has not had at least a year
to improve, and even then, unless I
can show some major improvements I
still won't post my own version. If
I see that the author of a code has
made the effort to improve their work
with new releases of a code, then I
might e-mail them with suggestions as
to how they might further improve their
codes.
How have I chosen to define the ratings
1-5 as listed on the FEX? A problem is,
as I said above, that there are several
ways to look at these codes.
5. (Excellent) - A code that does
something useful, that is not already
in existence on the FEX. This code must
do what it claims to do. Readable help
must be present, error checks, an H1
line, etc. Occasionally I'll let a minor
flaw or two pass for a code that I was
impressed by. If so, I'll state that
fact, and hope the author fixes it
anyway.
4. (Good) - A good code, with some
fixable problems that don't cause it
to fail. You should still be happy when
you receive a 4 rating from me.
3. (Fair) - A code with minor bugs,
not very descriptive, sparse help, etc.
A user who chooses to download this code
should expect it to work in general, but
expect it to fail occasionally, or perhaps
be slower than it really should. You may
find the help is not that clear.
2. (Needs improvement) - This is a code
that has serious problems, a major bug.
Users should beware when they use this
code. This is a code that might fall into
a "poor" rating, EXCEPT that I think there
is some virtue in this code. This is a
code that does something useful in my
eyes.
1. (Poor) - Major bugs. No help at all.
No comments. No error checking. Scripts
that clear your workspace for no good
reason. Heavily looped homework
assignments that don't offer anything
new or useful.
Yes, these categories are fuzzy. I'm
not confident that I will always get a
rating right, although I do try to be
consistent. Some might argue that there
should be a category for that exemplary
code, that is clearly a cut above. There
are a few of them. I'd like to have a
6 rating at times. (We are also trying
to get the Select designation working.)
When you as an author receive a rating
of 5 from me, its something that you
should be proud of. Occasionally, I'll
add a comment like "well done", "splendid",
or "superb". I'll even sometimes refer
a file to Doug as a pick of the week.
At the bottom end of the scale, this
is a big category. It encompasses files
that are pure dreck - a undocumented
script with one line of simple code
that does nothing useful, up to ones
that may do something, but are too
poor to fit into the category above.
I'd probably suggest that the author
rewrite these codes, then resubmit it
fresh after deleting the old version.
Finally, when I do rate a code as a 1
that is pure dreck, I'll state my
belief. I'll try to explain why I
think it is.
I will also be willing to re-review a
code when I see it has been repaired.
In some cases I'll even ask the
administrator to remove a review of
mine that I've been convinced was
wrong. The fact is, in a civilized
society, the peer review process of
the file exchange is an open debate
about the merits of a file. If "you"
feel that my review is in error, then
state your opinion, explaining why
you feel that to be true. Provide
your own rating for a file that you
feel to be more accurate. Back it up
with your name, and even better, your
true e-mail address. Give me and
others a valid reason to respect your
opinion. Having done this, you might
be surprised and find that I can be
convinced. I try to be fair in what
I've said, but I'll always be willing
to admit I was wrong when I find that
to be a fact.
John
.
- References:
- Prev by Date: Memory error (alot of parameters in workspace)
- Next by Date: Re: Integration using 'quad' Matlab commands
- Previous by thread: Re: FEX submission reviewers under close scrutiny
- Next by thread: Re: FEX submission reviewers under close scrutiny
- Index(es):
Relevant Pages
|
