Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QuickBuild

Builds are not stopping #2794

#1
vanderhu ·
I noticed lately the problem that builds are not stopping properly they keep on running continously while the agents are no longer executing the build anymore. Is there a way to still stop them?

I see this in Audit log for example:
Build B.876.CL-1555366-TEST(#2006013) was stopped.
Build request was cancelled from Queue.

For same configuration as job is not stopping at all.
  • replies 8
  • views 3767
  • stars 0
#2
robinshen ADMIN ·
While the build is stuck, can you please collect below info to help us investigating the issue?

1. Run jstack to get stack trace of QB server JVM:
/path/to/jdk/bin/jstack <QB server JVM process id>

2. Run jmap to get memory dump of QB server JVM:
/path/to/jdk/bin/jmap -dump:format=b,file=/path/to/server_dump.out <QB server JVM process ID>

Then ftp the memory dump and jstack output to sea.pmease.com (user/pass will be sent to you via email).

Please note that QB spaws two processes: one is wrapper process and another is JVM process, all above actions have to be done agains the JVM process.
#3
robinshen ADMIN ·
After that please restart QB server to bring QB back in track.
#4
robinshen ADMIN ·
Looks like my email of ftp user/pass bounced back. If necessary please send an email to [robin AT pmease DOT com] with an altenate address, and I will reply with the ftp account.
#5
vanderhu ·
When reporting this problem it was already hanging for couple of hours not able to stop, but after another +- 3 hours it finally stopped. Next time it happens I will do the steps as described by you.
#6
waldemar ·
I have just experienced a similar case. I had a build that was running much longer than expected, about 30min instead of expected 10min and I attempted to stop it. It did not work. In this case, too, audit log reports:

Build qb-482(#880) was stopped.

But it was not, it kept running. Here's the response to the steps Robin suggested:

build@bigbertha$ jstack 15397
15397: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
build@bigbertha$ jstack -F 15397
Attaching to process ID 15397, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypes" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:585)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:150)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:569)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Debugger attached successfully.
jstack requires a java VM process/core!

Same response to jmap
Sorry if it's not very helpful. Perhaps we could do something to catch a trace next time that happens.
Waldemar
#7
robinshen ADMIN ·
QB actually spawns two processes, one is the wrapper process and another is the JVM process, the jmap and jstack has to be run agains the JVM process. We found several cases of this issues occuring at different sites, and I have the impression that most of them are caused by server overloading or network connection issues (I found some connection refused/reset error messages in server/agent log when build is not stopping ). Right now we are waiting for more data to analyze the problem.
When this issue happens at your side, does QB show that all steps are finished, but the build is still running? If so, can you please check the configuration overview page to see in what stage the build request is (SENDING_BUILD_NOTIFICATIONS, COLLECTING_BUILD_METRICS, etc.)?
#8
waldemar ·
It happened again. This time, I've got logs.

I this case, two builds of the same configuration are running. The first build was stopped but it continues to run. The second build is stuck on a resource held by the first build. The build in question has a step that starts five child steps which in turn invoke their single sub-child steps that run on other agents, one step on each agent.

The situation is as follows
- audit log reports Build qb-648(#1077) was stopped (but it is still running)
- overview reports: RUNNING_BUILD
- the parent step reports: running (agent 12)
- child1: failed (agent 12), sub-child: cancelled (agent w06)
- child2: cancelled (agent 12), sub-child: running (agent w06) -- same agent as above but after sub-child 1 completed
- child3: failed (agent 12), sub-child: failed (agent 8)
- child4: running (agent 12), sub-child: running (agent 2)
- child5: failed (agent 12), sub-child: failed (agent 7)

The other build, that was blocked, eventually timed out waiting on resources.

jstack and jmap logs are available
#9
robinshen ADMIN ·
Please send me below info when this happens:
- screenshot of the step graph
- build id of the problematic build
- jmap and jstack result of QB server
- jmap and jstack result of agents which running the non-stopping child steps, in this case, it is agent w06, agent 12 agent 2

Please note that this these information has to be taken while the build is in stuck state, and you may upload the all the information to ftp://sea.pmease.com, and I will send you account of our ftp server via email.
Log in to reply