 |
History
| Log In
|
|
|
|
Dan, I still cannot reproduce this. I've tried many, many times over the weeks since you reported it. I've also not had any other reports of this. The interesting thing is that the thread dump you've posted here shows a scenario that is impossible (at least so far as designed and my brain can verify in the implementation). The scenario shown in these stacks of the thread dump is: * QuartzSchedulerThread is holding signalLock and waiting for a free thread in the pool in SimpleThreadPool.runInThread(). * All threads in the pool are waiting for signalLock This indeed would be a dead lock. The thing that makes this impossible (even though you obviously got it some how) is that runInThread() is called by QuartzSchedulerThread inside of this block: int availTreadCount = qsRsrcs.getThreadPool().blockForAvailableThreads(); if(availTreadCount > 0) { ... ....qsRsrcs.getThreadPool().runInThread(shell)... ... } The contract of blockForAvailableThreads() is that it only returns values larger than zero (never zero or less) - and that the returned value is the number of threads available in the pool. So... QuartzSchedulerThread should not be able to get to the call to runInThread() when the threads are exhausted, and it should therefore never block inside that method (because the wait() shown in QuartzSchedulerThread's stack above is only called if the pool is exhausted), and thus the deadlock shouldn't occur. So how did this happen? I'd be interested in seeing the rest of your thread dump -- for example I see that you have a call to pauseAll() in progress as well. Does your app somehow gain hold of and use the SimpleThreadPool itself (such that the avail count can change between QuartzSchedulerThread's calls to blockForAvailableThreads() and runInThread())? Help!?!? I'm not sure that my problem (described here: http://forums.opensymphony.com/thread.jspa?threadID=402480&tstart=0) is the similar issue to this one.
Anyway, I have 5 worker threads and only 2 jobs in my JobStore. And still after some period of time worker threads become blocked one by one. After that MisfireHandler becomes blocked too, trying to hadle misfired trigger. <... thread dump ...> Thread Group: jboss : max priority:10, demon:false Thread: MyQuartzScheduler_Worker-1 : priority:5, demon:false, threadId:47, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.core.QuartzScheduler.notifySchedulerThread(QuartzScheduler.java:1736) org.quartz.core.JobRunShell.run(JobRunShell.java:287) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:531) Thread: MyQuartzScheduler_Worker-2 : priority:5, demon:false, threadId:48, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.core.QuartzScheduler.notifySchedulerThread(QuartzScheduler.java:1736) org.quartz.core.JobRunShell.run(JobRunShell.java:287) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:531) Thread: MyQuartzScheduler_Worker-3 : priority:5, demon:false, threadId:49, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.core.QuartzScheduler.notifySchedulerThread(QuartzScheduler.java:1736) org.quartz.core.JobRunShell.run(JobRunShell.java:287) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:531) Thread: MyQuartzScheduler_Worker-4 : priority:5, demon:false, threadId:50, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.core.QuartzScheduler.notifySchedulerThread(QuartzScheduler.java:1736) org.quartz.core.JobRunShell.run(JobRunShell.java:287) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:531) Thread: MyQuartzScheduler_Worker-5 : priority:5, demon:false, threadId:51, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.core.QuartzScheduler.notifySchedulerThread(QuartzScheduler.java:1736) org.quartz.core.JobRunShell.run(JobRunShell.java:287) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:531) Thread: QuartzScheduler_MyQuartzScheduler-NON_CLUSTERED_MisfireHandler : priority:5, demon:false, threadId:53, threadState:BLOCKED, lockName:java.lang.Object@1802282 org.quartz.core.QuartzSchedulerThread.signalSchedulingChange(QuartzSchedulerThread.java:202) org.quartz.core.SchedulerSignalerImpl.signalSchedulingChange(SchedulerSignalerImpl.java:90) org.quartz.impl.jdbcjobstore.JobStoreSupport.signalSchedulingChange(JobStoreSupport.java:3065) org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3823) Thread Group: QuartzScheduler:MyQuartzScheduler : max priority:10, demon:false Thread: MyQuartzScheduler_QuartzSchedulerThread : priority:5, demon:false, threadId:52, threadState:TIMED_WAITING, lockName:java.lang.Object@960f0e java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool.blockForAvailableThreads(SimpleThreadPool.java:416) org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:251) <... thread dump ...> Full thread dump is too large to post it here. May I send it by email? Or I can attach it to this issue. yes, please attach the full thread dump to the issue. What you've posted is interesting, but I would really like to see the full dump. I believe I've reasoned out how this is happening, even though I can't reproduce it.
I'll work toward a fix. Thank you both for the info - it is the same issue, just manifested in two slightly different ways. Actually, strike my last comment.... I still can't figure out how this is possible. :(
(and still can't reproduce) The thread dumps do not show a deadlock, just a wait, and as I described a few comments earlier, that wait should never occur, because a thread should have been guaranteed to be available. :( Can you provide more platform info about what hardware and OS you are running on, Yuri? Here is server info:
AvailableProcessors 2 OSArch x86 OSVersion 5.2 JavaVendor Sun Microsystems Inc. JavaVMName Java HotSpot(TM) Client VM FreeMemory 100870608 ActiveThreadGroupCount 7 TotalMemory 133365760 JavaVMVersion 1.5.0_10-b03 ActiveThreadCount 47 JavaVMVendor Sun Microsystems Inc. OSName Windows 2003 JavaVersion 1.5.0_10 MaxMemory 532742144 The issue have been reproduced on JBoss 4.2.2GA and JBoss 4.2.0GA I'm using JobInitializationPluggin 2 jobs triggered periodically by cron triggers JobStoreCMT on MSSQL Issue was caused by some of the threading re-work in 1.6.1 RC1, have now re-worked it some more. ;-)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Resolved
Critical
Threading issue around SimpleThreadPool.runInThread...
Spring 2.5.4
Rough steps to reproduce the issue:
1. Set up a thread pool with 5 worker threads
2. Configure 6 jobs that run for around a minute each
3. Run the following code:
for (String jobGroup : getScheduler().getJobGroupNames()) {
for (String jobName : getScheduler().getJobNames(jobGroup)) {
getScheduler().triggerJob(jobName, jobGroup);
}
}