Metrics for automation ...

I am a Test manager (often wear the hat of project manager), working
for an IT services company in India - we are a typical CMM shop and
have a Software Testing and Automation Delivery Unit. I have been asked
to collect metrics for the Test automation projects I run. These are
typically, GUI Regression based automated tests using compuware tool.
We, in the team follow a company internal framework that is a hybrid
Model combining the principles of Data driven and keyword driven
methodologies. We are constantly re-inventing ourselves and our
approaches for the automation. Our team is made up of a good mixture of
junior and senior test engineers- few them are new to automation and
new to programming. I would say that our test automation practice
(which is getting into a mass manufacturing mode - more and more
projects in pipe line) is nascent.

The problem I am facing here, is about identifying a set of metrics
that I will provide to management regarding the automation project I
run - on weekly ( daily data - consolidated weekly) basis. Here are
the management objectives of asking for metrics from Project managers
like me.
1. Track the progress of the project (something on the lines of number
of work units completed etc)
2. Productivity of the team ( this is most controversial - I am of
the opinion that we are like "a buggy application and put on an
automation block" or "A toddler who is just struggling to walk -
asked to run a 100 mtrs running event")
3. Above also can be interpreted as way to measure performance of the
team/individual members.
4. To show to client that we are improving on productivity -
producing more and more automation scripts (or work units) in a given
time period and hence churning out automation solutions quicker than
before.

My honest opinion is "I have challenges in presenting a set of
numbers (metrics) to the management in a way that makes real value and
sense in this context. Any ideas or help from members of this forum for
following questions -

"How do I articulate the idea that it is a challenge for me present
them a set of metrics so that management can get the kind of
information that they are looking for"

"In the context that I have mentioned - evolving practice, getting
into a mass production mode, "mix of junior and senior resources"
- are there any ideas of metrics that can help?"

"Automation work that we do is not a - routine task something that
can be explained in a simple flowchart or something that can be
compared with work output from an assembly line. How can I measure
something like number of work units delivered a day in such a
context?"

Shrini Kulkarni
http://shrinik.blogspot.com

.

Relevant Pages

  • Re: Quality Assurance for a very small company
    ... Your question is similar to some one asking "how do people commute ... Project management and release management practice -- Who decides ... Coming to Automation - you can deploy automation at multiple layers - ... GUI layers - where tools like Silk, QTP, TestPartner - COTS tools will ...
    (comp.software.testing)
  • Re: Eyes In The Sky
    ... "Production" in America is so high that the cost of labor is trivial. ... With increasing automation and robotic equipment, ... The cost of management is never cut....there will always be more people ...
    (misc.survivalism)
  • Re: Driverless Car ?!
    ... Automation that eliminates the current union contracts would help. ... are two sets of managements that require mindset changes: ... coroporate management and, more importantly, the union management. ...
    (sci.physics)
  • Re: I get it
    ... Minor nit in your screed: C&C is management. ... management) is a big thing and getting into the driver seat for automation. ... Of course manning reduction in the military is a good thing - it exposes ...
    (comp.lang.lisp)
  • Re: Metrics for automation ...
    ... with what and how collect as metrics in my context and I have mentioned ... the needs of management that is interested only in seeing numbers. ... I do not manage a development team - I manage a Test automation team - ... believer of the statements like "Defect free software delivery is ...
    (comp.software.testing)