Re: Metrics for automation ...

Paul Szymkowiak wrote:

....

When analyzing what you've measured to support change or achievement
goals, trends are generally more insightful and more useful indicators
than absolute numbers. Is the rate, volume etc increasing or decreasing
over time? Why is that? Is that good or bad? What can we do to change
that trend going forward?

....

As a starting point, try to base your approach on simple, easily
gathered primitive metrics; basically, keep collection as simple as you
possible can - but not simpler than it needs to be.

At a session at Agile 2006 Rachel Davies led a session in which we
experimented with creating robust metrics and then gaming them. My
conclusions were 1) ANY metric can be gamed handily; 2) in general, the
simpler the metric sounds, the easier it is to game; and 3) in this
business, we undervalue qualitative metrics and overvalue quantitative
ones. It's also to see that, usually due to reification (mentally
treating an abstract notion as though it were something concrete), we
end up comparing not only apples with oranges, but also apples with
tractors.

One of the easiest mistakes to make is to count test cases or bugs.
Test cases are ideas; bugs are ideas misimplemented. What's the
appropriate way to count an idea? There isn't one. When we count a
test case, we ignore everything about it except its identity. We
ignore the length of time it takes to write it, the length of time it
takes to run it, the risk that it is designed to detect, the
significance of that risk, the power of the test to reveal the bug, the
reliability of the oracle (the principle or mechanism by which we
recognize a problem), the probability of that risk coming to pass, the
variability in the impact of the risk--just to name a few. The same
sorts of things are true about counting bugs. And yet what do we try
to count most often?

In terms of assessing automation efforts, we often count something
about them, and presume that higher numbers are better. Yet test
automation is a tool; automated tests are tools. Car companies use
tools to build cars. Would we really assess the quality of a car by
counting the tools used to produce it? Would we willingly drive in one
where the tool count was the principal means of assessing the quality
of the car? Wouldn't we want to know something about the skill of the
people who were wielding the tools? Wouldn't we want to make some
observations about the item that the tools were producing? Wouldn't we
want to know something about the tools, not just how many there were?

If we're trying to report on progress towards a goal, I think
narratives and stories from skilled peole that we trust are going to
tell us what we need to know. Management ultimately wants to know
about the quality of the product. Telling them about risks, or about
the progress we're making towards identifying those risks, seems to me
to be the way to go. If they want something more numerical, we must
take responsibility for communicating the risks and drawbacks of that
approach. If they insist on numbers, let's use numbers that prompt
questions, rather than provide misleading answers.

---Michael B.

.

Relevant Pages

  • Re: [Full-disclosure] Risk measurements
    ... damn thing will escape with bugs, no matter how hard you try. ... The probability in risk management is mostly impossible since because ... you'll need to do some risk modeling to figure out ... It's like buying insurance ...
    (Full-Disclosure)
  • Re: Risk metrics
    ... We have updated this in OSSTMM 3.0. ... The OSSTMM has pulled out of RISK completely because it is so biased ... New metrics are quantification-based-- facts only from operations used ... > Vulnerability scans and pen tests are a snapshot. ...
    (Pen-Test)
  • Re: Disconcerting sight at Canadian Auto Show
    ... else volunteers to do the same thing. ... Risk assessment is my strong suit. ... Yes, the vehicle pulling out will likely be cited for failing to yield, but if the other car hadn't been speeding in the first place, the two would've been far less likely to meet. ... My use of the term "blind corner" implies that the speeding car isn't aware of it's existence, so it's quite impossible for that speeding driver to know all the risks invloved with his excessive speed. ...
    (rec.photo.digital)
  • Re: Hugely OT: sound pressure
    ... She'd ask how many people die in motorcycle accidents in the far east, ... It's a motorbike or nothing. ... you were in a car yourself. ... equipment and road quality to make that risk as low as it could be. ...
    (uk.comp.sys.mac)
  • Re: basic in-line skating question
    ... It's not safe to drive a car on them either. ... but I can make the level of risk and danger ... Skating out on the roads usually has additional sources of risk beyond ... with bicycling or car driving), ...
    (rec.sport.skating.inline)