IDS performance problem


Hi,

Any help will be appreciated!

I'm dealing with an IDS 9.21.FC3 on HPUX 11.11 system, and two weeks ago the system
began to run soooo slowly on some peak hours. Of course my friend is aware that he
needs to upgrade, but that's not a possibility right now.

After 2 days of intense monitoring we saw that IDS is not able to take more than 30%
of the CPU (10 physical CPUS), and there are no disk issues (no more than 3-5% wio
as showed by sar -u).

I truly believe that the system is quite well configured, but...

What I've observed is that this is the system with most latch and buffer waits I've
ever seen, and the ready queue is 20 threads long in bad times, 90 threads long in
worst cases. Yes, we tried up to 20 CPUVPs with no results - no more than 30% real
CPU use. And this afternoon I've repeatedly seen this:

onstat -g wmx

Informix Dynamic Server 2000 Version 9.21.FC3 -- On-Line -- Up 6 days 20:53:27
-- 3967268 Kbytes

Mutexes with waiters:
mid addr name holder lkcnt waiter waittime
4179 c0000000d3e515e8 nsf.lock 178903 0 178727 11
178919 2
178878 17
178897 28
178336 5
178908 2
178887 5
178866 5
178914 6
178913 5
146757 1
178915 5
178673 1
178921 9
178399 11
168900 3
178534 1
178920 2
178901 8
177881 4
178402 3
178868 11
177845 16
178885 36
178882 8
19 0
178879 2
178924 0
46 0
18 0
178865 11
178918 2
4264 c0000000d4655358 hash PC_ID -1 0 178404 0


onstat -g wmx

Informix Dynamic Server 2000 Version 9.21.FC3 -- On-Line -- Up 6 days 21:01:42
-- 4172068 Kbytes

Mutexes with waiters:
mid addr name holder lkcnt waiter waittime
4179 c0000000d3e515e8 nsf.lock 179073 0 179123 10
179167 1
27 1
179166 4
178337 4
179174 3
178970 2
179173 1
179177 4
179081 13
179138 1
179064 2
179150 1
179141 1
179102 5
179114 6
179186 1
154345 1
179172 1
178901 1
179125 3

179176 8
178935 0
178912 8
46 0
179187 0
179183 4
179175 1
28 0
178923 5
47 0
179096 0
179098 0
179182 0
179142 0
4258 c0000000d4655028 hash PC_ID 178432 1 178867 0
178956 0
178387 0
4259 c0000000d46550b0 hash PC_ID 178902 1 179144 0
178698 0
178951 0
4268 c0000000d4655578 hash PC_ID 177879 1 179121 0
178985 0
4269 c0000000d4655600 hash PC_ID 178898 1 177319 0


and this:
onstat -g spi

Spin locks with waits:

Num Waits Num Loops Avg Loop/Wait Name
461 5736195 12442.94 mtcb vproc_sync_lock
1204 22413 18.62 mtcb sleeping_lock
3 10 3.33 mtcb tcb_queue_lock
29099 561272 19.29 mtcb mutex_list_lock
47 1167 24.83 mtcb cond_list_lock
634 30765 48.53 mtcb notify_lock
190 7806 41.08 class cl_lock, class = 0
178839 6820049 38.14 vproc vp_lock, id = 1
180588 7060191 39.10 vproc vp_lock, id = 3
160169 6678508 41.70 vproc vp_lock, id = 4
141997 6210570 43.74 vproc vp_lock, id = 5
117829 5194097 44.08 vproc vp_lock, id = 6
95753 4379701 45.74 vproc vp_lock, id = 7
73398 3546255 48.32 vproc vp_lock, id = 8
69551 3438488 49.44 vproc vp_lock, id = 9
1683 12302 7.31 vproc vp_lock, id = 16
2 8 4.00 vproc vp_lock, id = 17
1446 11229 7.77 vproc vp_lock, id = 18
1230 9728 7.91 vproc vp_lock, id = 19
1219 10492 8.61 vproc vp_lock, id = 20
1234 9749 7.90 vproc vp_lock, id = 21
1128 8123 7.20 vproc vp_lock, id = 22
1136 6817 6.00 vproc vp_lock, id = 23
1000 7134 7.13 vproc vp_lock, id = 24
1123 6982 6.22 vproc vp_lock, id = 25
947 9009 9.51 vproc vp_lock, id = 26
2 1 0.50 vproc vp_lock, id = 27
2 13 6.50 vproc vp_lock, id = 28
4 38 9.50 vproc vp_lock, id = 29
4 11 2.75 vproc vp_lock, id = 31
....
1017 6758 6.65 mutex lock, name = ddh chain
1583 12117 7.65 mutex lock, name = ddh chain
956 5481 5.73 mutex lock, name = ddh chain
109 2486 22.81 mutex lock, name = glob PC_ID
25 242 9.68 mutex lock, name = hash PC_ID
6 56 9.33 mutex lock, name = hash PC_ID
14 80 5.71 mutex lock, name = hash PC_ID
148 2020 13.65 mutex lock, name = hash PC_ID
213 3598 16.89 mutex lock, name = hash PC_ID
95 1550 16.32 mutex lock, name = hash PC_ID
79 545 6.90 mutex lock, name = hash PC_ID
84 1275 15.18 mutex lock, name = hash PC_ID
1040 17522 16.85 mutex lock, name = hash PC_ID
418 6249 14.95 mutex lock, name = hash PC_ID
8 63 7.88 mutex lock, name = hash PC_ID
39 914 23.44 mutex lock, name = hash PC_ID
451 7089 15.72 mutex lock, name = hash PC_ID
1179 23777 20.17 mutex lock, name = hash PC_ID
9825 258278 26.29 mutex lock, name = hash PC_ID
72 684 9.50 mutex lock, name = hash PC_ID
26 217 8.35 mutex lock, name = hash PC_ID
8 145 18.12 mutex lock, name = hash PC_ID
200 2679 13.39 mutex lock, name = hash PC_ID
64 1137 17.77 mutex lock, name = hash PC_ID
51 422 8.27 mutex lock, name = hash PC_ID
59 1162 19.69 mutex lock, name = hash PC_ID
5 8 1.60 mutex lock, name = hash PC_ID
6 28 4.67 mutex lock, name = hash PC_ID
681 10329 15.17 mutex lock, name = hash PC_ID
6 36 6.00 mutex lock, name = hash PC_ID
33 594 18.00 mutex lock, name = hash PC_ID
87 3207 36.86 mutex lock, name = hash PC_ID
2 16 8.00 mutex lock, name = hash PC_ID
65 2122 32.65 mutex lock, name = hash PC_ID
250 2996 11.98 mutex lock, name = Language
....
2 1 0.50 mutex lock, name = gfile
62863 2763501 43.96 mutex lock, name = vpc aio
97 771 7.95 mutex lock, name = iovp aio 0
2 1 0.50 mutex lock, name = vpc msc
191 1833 9.60 mutex lock, name = iovp aio 1
72 667 9.26 mutex lock, name = iovp aio 2
60 969 16.15 mutex lock, name = iovp aio 3
....
2 62 31.00 mutex lock, name = AIOreq
4 27 6.75 mutex lock, name = AIOreq
4 36 9.00 mutex lock, name = pt_100002
140994 952812 6.76 mutex lock, name = pt_5003a0
78 322 4.13 mutex lock, name = pt_5003a6
65 296 4.55 mutex lock, name = pt_5003a3
....
66179 387581 5.86 mutex lock, name = pt_400012
155345 1038908 6.69 mutex lock, name = pt_400013
8 19 2.38 mutex lock, name = pt_40004c
....
1 1 1.00 tcb lock, tid = 179928
1 63 63.00 tcb lock, tid = 179934
1 500 500.00 tcb lock, tid = 179940
342742 1917383918 5594.25 shmcb sh_lock
1046 2058 1.97 pool ID lock
399 4149835 10400.59 pool po_lock, name = pnlpool
712 15240 21.40 pool po_lock, name = dstpool
637 16008 25.13 pool po_lock, name = rsam
536 54877 102.38 pool po_lock, name = aio
2254 13078656 5802.42 pool po_lock, name = gls
46 1261 27.41 pool po_lock, name = resroutpool
4 8 2.00 pool hash[84] lock
....



Any thougts? Any idea?
Thanks in any case.
--
José Luis Matute
Responsable Division Sistemas
Contacto Local del Grupo de Usuarios de Informix en España

D&D Grupo Dydes
Poligono EUROPOLIS Edif. Al Andalus
Calle X, nº 4 y 6
28230 Las Rozas (Madrid)
Tf: +34 91 6407080 Fax: +34 91 6373280



sending to informix-list
.

Relevant Pages

  • RE: Problem with Windows domain users
    ... Problem with Windows domain users ... As of IDS 10.00.TC6 the situation was not good and I'm sorry for the ... bring this to the attention of IBM technical support some time ago. ... the localsystem user instead of the informix user? ...
    (comp.databases.informix)
  • Re: Losing Precision from FLOAT in DBD::Informix
    ... The versions of CSDK and IDS are also not ... STSM, Informix Database Engineering, IBM Data Management ... >> are declared FLOAT. ... >> - where and why is the precision lost? ...
    (perl.dbi.users)
  • Re: Re: Informix in the news! Part Deux...
    ... Maybe IDS will need to come to a point like derby for this to happen, ... Subject: Re: Informix in the news! ... > engaged with the vendors mentioned below as well as many others. ... they don't have to worry about the future of DB2, ...
    (comp.databases.informix)
  • Re: Shocked! Shocked, I tell you!
    ... Informix Dynamic Server is an IBM flagship database. ... In February, we announced IDS v10.0, the most ...
    (comp.databases.informix)
  • Re: Informix business grew by double digits in 2006 (Q1+Q2)
    ... Christine Normile wrote: ... Informix went from being a company ... applications certainly do not need all the features and functions of IDS. ... do you really think that cisco would buy informix if they ...
    (comp.databases.informix)