History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QUARTZ-669
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: James House
Reporter: Torsten Juergeleit
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Quartz Scheduler

Spinning-workaround in QuartzSchedulerThread.run() is expensive (cpu-wise) on multi-core systems

Created: 29/May/08 10:55 AM   Updated: 19/Aug/08 02:32 PM
Component/s: Core
Affects Version/s: 1.5.2
Fix Version/s: 1.6.1

Environment: Sun T2000 (T1 SPARC cpu with 8 cores, 32GB memory), Sun Solaris 10, Java 1.5.0_14, BEA WebLogic 9.2 MP2


 Description  « Hide
We're running multiple instances of an JEE application (which uses Quartz internally) on Sun's T2000 boxes. Here 2 Quartz schedulers instances (with a single worker thread each) are used to trigger 3 jobs in different intervalls (the shortest is 5 seconds and the longest is 4 times a day).

Now we're experiencing a significant basic load (aka white noise) on the 32 virtual cpus (aka strands) of our boxes

1. Via "mpstat" we can see that the first strand (cpu 0 -> which runs the kernel's process scheduler) has a unusual high number of cross-calls. Normally the process scheduler doesn't create so many cross-calls.

2. With "corestat" we can see that the first core (with cpu0) has a a significant higher load (~15%) than the other 7 cores


By using Dtrace to examine the kernel scheduler's thread management activities (enqueue / dequeue, on-cpu / off-cpu) we found that the threads with the following stack trace are the most expensive cpu-hoggers (~6000 calls within 10 seconds; each thread is only 60µsec in state "on-cpu"):

   libc.so.1`__pollsys+0x4
   libc.so.1`poll+0x7c
   libjvm.so`__1cIos_sleep6Fxi_i_+0x230
   libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
   libjvm.so`JVM_Sleep+0x260
   java/lang/Thread.sleep*
   org/quartz/core/QuartzSchedulerThread.run*


The culprit for these heavy short-running thread activities seems to be the spinning-workaround (sleeping the active thread for 10ms) in QuartzSchedulerThread.run() http://svn.opensymphony.com/fisheye/browse/~br=trunk/quartz/trunk/src/java/org/quartz/core/QuartzSchedulerThread.java?r=793#l283 .
Maybe the statement "don't worry though, this spinning doesn't even register 0.2% cpu usage on a pentium 4" found in the workaround's javadoc doesn't hold true for cpus with multi-core architecture.

Cheers,
Torsten

 All   Comments   Change History      Sort Order:
Torsten Juergeleit - [29/May/08 11:39 AM ]
The following is a quote from Sun's document „Developing and Tuning Applications on UltraSPARC T1 Chip Multithreading Systems" http://www.sun.com/blueprints/1205/819-5144.pdf :

„On a traditional processor, other LWPs in the dispatch queue will prevent the polling thread from burning CPU cycles. Polling in a CMT architecture is an undesirable option, as it occupies CPU cycles that could be used by other threads scheduled on a core. In a thread-rich environment the polling thread will also run far more often. "

Torsten Juergeleit - [04/Jun/08 09:40 AM ]
How about introducing a property for the value of the variable "spinInterval" (e.g. "org.quartz.scheduler.spinInterval") instead using a hard-coded value ("long spinInterval = 10;")?

In our case we would use a value of 1000 for this property (our shortest interval of job scheduling is every 5 seconds). This would greatly reduce CPU-hogging for us.

Jim Nichols - [18/Jun/08 01:31 PM ]
Based on the DTrace output, how can you be sure that the time is being spent spinning and not actually doing work in the run method? If you have a job running every 5 seconds, the scheduler thread is going to be very frequently in there starting the job.

Torsten Juergeleit - [19/Jun/08 04:29 AM ]
The following Dtrace script is used to collect every thread of a given process name (e.g. "java") when it's put on cpu or took off cpu by Solaris' process scheduler within 10 secods:

#!/usr/sbin/dtrace -qs

BEGIN {
        t[0] = timestamp;
        t[1] = timestamp;
        t[2] = timestamp;
        t[3] = timestamp;
        t[4] = timestamp;
        t[5] = timestamp;
        t[6] = timestamp;
        t[7] = timestamp;
        t[8] = timestamp;
        t[9] = timestamp;
        t[10] = timestamp;
        t[11] = timestamp;
        t[12] = timestamp;
        t[13] = timestamp;
        t[14] = timestamp;
        t[15] = timestamp;
        t[16] = timestamp;
        t[17] = timestamp;
        t[18] = timestamp;
        t[19] = timestamp;
        t[20] = timestamp;
        t[21] = timestamp;
        t[22] = timestamp;
        t[23] = timestamp;
        t[24] = timestamp;
        t[25] = timestamp;
        t[26] = timestamp;
        t[27] = timestamp;
        t[28] = timestamp;
        t[29] = timestamp;
        t[30] = timestamp;
        t[31] = timestamp;
}


sched:::on-cpu
/ execname == $$1 /
{
        printf("%12d %2d %20d %-10s: %d/%d %s %4d\n", timestamp/1000, cpu, timestamp - t[cpu] ,probename, pid, tid, curpsinfo->pr_psargs, curlwpsinfo->pr_pri);
        t[cpu] = timestamp;
}
sched:::off-cpu
 / curpsinfo->pr_fname == $$1 && timestamp - t[cpu] < 60000 /
{
        printf("%12d %2d %20d %-10s: %d/%d %s -> %s\n", timestamp/1000, cpu, timestamp - t[cpu], probename, pid, tid, curpsinfo->pr_psargs, args[1]->pr_psargs);
@st[jstack(15),pid,tid]=count();
        t[cpu] = timestamp;

}

tick-10s {exit(0);}

END {
trunc(@st,20);
printa(@st);
}


As the result this scripts prints an ordered list of stacktraces with the number of invocations. For a system with 6 JVMs running 3 Schedulers from Quartz 1.5.2 the script prints the following:



              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xe8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
      862 42 73

              libc.so.1`__pollsys+0x4
              libc.so.1`pselect+0x1c8
              libc.so.1`select+0xa0
              libjvm.so`__1cIos_sleep6Fxi_i_+0x170
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x21c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe4
              libjvm.so`_start+0x134
              libc.so.1`_lwp_start
      807 8 76

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2934 105 148

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2934 109 167

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2900 119 174

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2917 115 179

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2900 115 181

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2968 118 181

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
              StubRoutines (1)
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
     2883 125 182

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2968 116 187

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
              StubRoutines (1)
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
     2883 121 188

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2934 107 192

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2968 114 192

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2951 116 200

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
              StubRoutines (1)
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
     2883 123 202

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2900 117 208

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2951 118 232

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2917 117 233

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2917 119 238

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              C2IAdapter
              org/quartz/core/QuartzSchedulerThread.run*
              OSRAdapter
     2951 114 266


The numbers in the line after the stacktrace are PID, thread id and number of invocations (within 10 seconds). This indicates that 18 instances of QuartzSchedulerThread were on-cpu (spinning) from 14 to 26 times per seconds each.

For us (running multiple JVMs on boxes with 32 cpus) this spinning is cpu-wise way to expensive. The cpu used by Solaris' process scheduler is spoiled by scheduling QuartzScheduler thread for spinning only. With mpstat we can see that this cpu (it's the first cpu) is pretty busy with cpu cross-calls (column xcal -> 3000 xcals for cpu0):

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
  0 3 0 3016 621 200 356 7 45 109 0 579 12 6 0 82
  1 1 0 21 270 0 377 6 119 23 0 503 4 1 0 95
  2 0 0 8 184 0 225 5 82 17 0 113 5 1 0 94
  3 0 0 5 149 0 226 0 78 4 0 143 1 1 0 98
  4 0 0 12 119 0 265 0 62 12 0 417 9 1 0 90
  5 0 0 21 113 0 238 6 61 11 0 163 19 1 0 80
  6 0 0 6 97 0 199 0 55 13 0 142 1 0 0 99
  7 0 0 24 113 0 253 0 53 5 0 177 1 1 0 98
  8 0 0 4 131 0 286 1 62 19 0 105 1 0 0 99
  9 0 0 8 101 0 217 1 59 13 0 109 4 1 0 95
 10 0 0 6 111 0 249 0 68 6 0 111 1 0 0 99
 11 0 0 16 123 0 256 3 58 8 0 530 4 1 0 95
 12 2 0 30 149 0 326 4 74 16 0 192 0 1 0 99
 13 0 0 12 111 0 257 3 67 18 0 180 4 1 0 95
 14 0 0 23 126 0 298 5 76 9 0 210 10 1 0 89
 15 0 0 2 99 0 213 0 57 7 0 122 1 0 0 99
 16 0 0 3 104 0 245 0 54 16 0 83 1 1 0 98
 17 0 0 23 91 0 193 1 38 7 0 174 1 0 0 99
 18 1 0 21 75 0 172 4 46 12 0 91 24 0 0 76
 19 0 0 18 85 0 217 3 61 5 0 104 12 1 0 87
 20 0 0 39 181 0 380 2 61 15 0 121 1 1 0 98
 21 0 0 75 250 8 513 4 63 7 0 424 1 1 0 98
 22 0 0 201 423 328 205 3 58 8 0 211 21 1 0 78
 23 1 0 12 104 3 223 0 54 15 0 147 1 0 0 99
 24 0 0 20 114 23 179 6 40 18 0 125 21 0 0 79
 25 0 0 22 128 20 243 3 50 12 0 100 4 0 0 96
 26 0 0 9 91 0 198 2 48 7 0 98 2 0 0 98
 27 0 0 17 83 0 175 4 47 4 0 111 13 0 0 87
 28 0 0 36 85 0 174 9 45 18 0 1284 17 1 0 82
 29 0 0 34 100 0 219 8 45 7 0 221 33 1 0 66
 30 0 0 11 81 0 189 0 54 10 0 108 1 0 0 99
 31 2 0 5 83 0 201 1 59 9 0 150 2 1 0 97

Torsten Juergeleit - [19/Jun/08 04:38 AM ]
>Based on the DTrace output, how can you be sure that the time is being spent spinning and not actually doing work in the run method?

Because the script is collect only threads being on-cpu for a very short amount of time (60µsec -> only a few cpu cycles). In the script the predicate "timestamp - t[cpu] < 60000" is used.

Our jobs (the busiest one is triggered every 5 seconds) are running definitely longer than 5µsecs :-)


>If you have a job running every 5 seconds, the scheduler thread is going to be very frequently in there starting the job.

But does running a job every 5 seconds entitle QuartzSchedulerThread.run() to call Thread.sleep() up to 26 times per second?

Torsten Juergeleit - [19/Jun/08 04:42 AM ]
By changing the spinning interval in QuartzSchedulerThread from 10msec to 100msec cuts the number of cpu cross-calls on cpu0 nearly by half (mpstat 1):

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
  0 3 0 3016 621 200 356 7 45 109 0 579 12 6 0 82
  1 1 0 21 270 0 377 6 119 23 0 503 4 1 0 95
  2 0 0 8 184 0 225 5 82 17 0 113 5 1 0 94
  3 0 0 5 149 0 226 0 78 4 0 143 1 1 0 98
  4 0 0 12 119 0 265 0 62 12 0 417 9 1 0 90
  5 0 0 21 113 0 238 6 61 11 0 163 19 1 0 80
  6 0 0 6 97 0 199 0 55 13 0 142 1 0 0 99
  7 0 0 24 113 0 253 0 53 5 0 177 1 1 0 98
  8 0 0 4 131 0 286 1 62 19 0 105 1 0 0 99
  9 0 0 8 101 0 217 1 59 13 0 109 4 1 0 95
 10 0 0 6 111 0 249 0 68 6 0 111 1 0 0 99
 11 0 0 16 123 0 256 3 58 8 0 530 4 1 0 95
 12 2 0 30 149 0 326 4 74 16 0 192 0 1 0 99
 13 0 0 12 111 0 257 3 67 18 0 180 4 1 0 95
 14 0 0 23 126 0 298 5 76 9 0 210 10 1 0 89
 15 0 0 2 99 0 213 0 57 7 0 122 1 0 0 99
 16 0 0 3 104 0 245 0 54 16 0 83 1 1 0 98
 17 0 0 23 91 0 193 1 38 7 0 174 1 0 0 99
 18 1 0 21 75 0 172 4 46 12 0 91 24 0 0 76
 19 0 0 18 85 0 217 3 61 5 0 104 12 1 0 87
 20 0 0 39 181 0 380 2 61 15 0 121 1 1 0 98
 21 0 0 75 250 8 513 4 63 7 0 424 1 1 0 98
 22 0 0 201 423 328 205 3 58 8 0 211 21 1 0 78
 23 1 0 12 104 3 223 0 54 15 0 147 1 0 0 99
 24 0 0 20 114 23 179 6 40 18 0 125 21 0 0 79
 25 0 0 22 128 20 243 3 50 12 0 100 4 0 0 96
 26 0 0 9 91 0 198 2 48 7 0 98 2 0 0 98
 27 0 0 17 83 0 175 4 47 4 0 111 13 0 0 87
 28 0 0 36 85 0 174 9 45 18 0 1284 17 1 0 82
 29 0 0 34 100 0 219 8 45 7 0 221 33 1 0 66
 30 0 0 11 81 0 189 0 54 10 0 108 1 0 0 99
 31 2 0 5 83 0 201 1 59 9 0 150 2 1 0 97



And with the aformentioned Dtrace script QuartzSchedulerThread.run() doesn't show up anymore:

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4383 20 15

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x230
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep
              java/lang/Thread.sleep
              org/quartz/core/QuartzSchedulerThread.run
              StubRoutines (1)
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
     4383 115 16

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4384 20 16

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x1d8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x284
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              weblogic/nodemanager/server/ServerMonitor.runMonitor*
              weblogic/nodemanager/server/ServerMonitor.run
              java/lang/Thread.run
              0xf9000218
    29760 466 16

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4381 20 17

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4381 20 17

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x1d8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x284
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              weblogic/nodemanager/server/ServerMonitor.runMonitor*
              weblogic/nodemanager/server/ServerMonitor.run
              java/lang/Thread.run
              0xf9000218
    29760 454 17

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x1d8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x284
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              weblogic/nodemanager/server/ServerMonitor.runMonitor*
              weblogic/nodemanager/server/ServerMonitor.run
              java/lang/Thread.run
              0xf9000218
    29760 464 17

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4386 20 18

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0x1d8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x284
              libjvm.so`JVM_Sleep+0x260
              java/lang/Thread.sleep*
              weblogic/nodemanager/server/ServerMonitor.runMonitor*
              weblogic/nodemanager/server/ServerMonitor.run
              java/lang/Thread.run
              0xf9000218
    29760 460 18

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4385 20 19

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4385 20 20

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4386 20 20

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4382 19 21

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4382 19 22

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4384 20 25

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4383 20 30

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xec
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     4122 20 46

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xe8
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x25c
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x208
              libc.so.1`_lwp_start
     1002 42 51

              libc.so.1`__pollsys+0x4
              libc.so.1`poll+0x7c
              libjvm.so`__1cIos_sleep6Fxi_i_+0xdc
              libjvm.so`__1cCosFsleep6FpnGThread_xi_i_+0x284
              libjvm.so`__1cNWatcherThreadDrun6M_v_+0xe8
              libjvm.so`__1cG_start6Fpv_0_+0x200
              libc.so.1`_lwp_start
    29760 8 56