Re: how fast can I sort on mainframe (using DFSORT)?
- From: betten@xxxxxxxxxx (David Betten)
- Date: 10 Mar 2008 06:59:40 -0700
Chris,
Since I joined the team as the performance lead a couple years ago,
Frank now defers these types of questions to me. Everything you (and all
the others that responded) have said are good points about DFSORT
performance. But the multitude of recommendations just reiterates the
point in my original response that there are so many factors, you could
spend weeks tuning things that don't improve anything. Just as I preached
during my 15+ years of batch tuning, I still say that before you start
tuning anything, you have to determine where you are spending time. Then
you start your tuning there. I always prefer to get some data (sysouts,
SMF, etc.) before I start making recommendations. That is why I asked
Pawel to send us a few sample sysouts as a start.
Have a nice day,
Dave Betten
DFSORT Development, Performance Lead
IBM Corporation
email: betten@xxxxxxxxxx
DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/
IBM Mainframe Discussion List <IBM-MAIN@xxxxxxxxxxx> wrote on 03/10/2008
12:48:05 AM:
Pawel,----------------------------------------------------------------------
I have been out of the sorting business for a while, but the target you
can go for is what I call 2T, where T is the time to read in the data.
The 2T comes from the fact that you have to read it in and then write it
out.
Writing to sort work files should not be the problem, hopefully the sort
is writing to several of them in parallel during the sort phase and
reading from them in parallel during the merge phase.
EXCP's are an accountant's measurement, not a tuner's measurement. Look
at your SMF data and see what the channel connect times are for SORTIN
and SORTOUT. If the elapsed time is significantly greater than those
two added together then you have to start looking for the bottleneck.
My first reaction to your 17M/second was a question. Is that data
sorted divided by total time? I assume it is. If so then you are
getting about 34M/second on input and output, and that is not a bad
number.
Tape and DASD inputs and output will give you different numbers. There
tends to be more interference with tape files. Fewer channels, no PAV.
Back when DASD was real 3390, 4M/second was the target, so you are doing
pretty good. I couch that with the fact that I have not run throughput
tests on the various DASD subsystems available today. 34M may be great
or it may be just average. I don't know.
The best thing you can do is give each sort enough memory. You will
need to ask Frank Yeager what the target amounts of memory are for a 10G
sort. My guess is about 17M to 24M. A 100G sort might need 50M to 75M.
A 1G sort might need 5M to 7M.
What you are attempting to do is make it so there is no need to do an
intermediate merge, or in the case of blockset sorting, that the final
merge remains efficient.
I would not attempt to get a 10G sort to run as a turnaround or incore
sort. It will put a large strain on RSM/ASM and not buy you that much.
I hope Frank weighs in on this.
Chris Blaicher
BMC Software, Inc.
These are my own personal comments and do not reflect the views of my
employer. Please do your own evaluation of any statements or
suggestions made.
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@xxxxxxxxxxx] On
Behalf Of Pawel Leszczynski
Sent: Sunday, March 09, 2008 4:06 PM
To: IBM-MAIN@xxxxxxxxxxx
Subject: how fast can I sort on mainframe (using DFSORT)?
Hello everybody,
I realize subject is VEEEERY broad and my question VEEEERY general,
but...
Recently in our shop we are reviewing our whole batch processing.
Most of the time of EOD processing is consumed by sort of many big
sequential
files. (One such file has approximately order of 10GB, 10mln records)
I listed few tens of such batch jobs (the longest-lasting ones)
and computed mean sorting rate.
It appeared to be about 1GB/min ~ 17MB/sec
I suppose it's very poor result(???).
Can you tell me how much I can improve this?
These batch jobs are little CPU-consuming (~10% of one CPU),
I suppose that major concern is to:
-limit EXCPS (1)
-increase throughput rate from DASD to central storage (2)
I realize that sorting whole file in central storage (hiperspace
sorting) would
eliminate need to use work files and EXCPS to them.
How much central storage is needed to handle in-storage sorting for
let's say 10 GB file???
How much can I improve (2)?
Can you tell me what is mean sorting rate in your installation?
Before starting I would just like to know if I can achieve substantial
improvement.
TIA,
Pawel Leszczynski
PKO BP SA
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to listserv@xxxxxxxxxxx with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to listserv@xxxxxxxxxxx with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
.
- References:
- Re: how fast can I sort on mainframe (using DFSORT)?
- From: Blaicher, Chris
- Re: how fast can I sort on mainframe (using DFSORT)?
- Prev by Date: DSN ENQ conflict - debugging after the fact.
- Next by Date: Re: DSN ENQ conflict - debugging after the fact.
- Previous by thread: Re: how fast can I sort on mainframe (using DFSORT)?
- Next by thread: Re: how fast can I sort on mainframe (using DFSORT)?
- Index(es):
Relevant Pages
|
