Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QuickBuild

spontaneous failure to access source control #4602

support request
2 #1
drdt ·

At a time when the system is largely idle, certain automated jobs on our production QB server started to experience the following errors:

2024-11-12 13:32:27,751 [pool-2-thread-571222] ERROR com.pmease.quickbuild.DefaultBuildEngine -
Error processing build request 4df57b8a-7d65-4990-8acb-6f001eba691e
java.lang.RuntimeException: Error executing check condition job.
Caused by: com.pmease.quickbuild.QuickbuildException:
Cannot run program "p4" (in directory "/opt/qbserver-8810/bin"): error=0, Failed to exec spawn helper: pid: 1964899, exit value: 1

A job worked at 6:17 am, and then at 6:24 it would not start (no jobs would start) due to the above message (showing up in a red bar on the Configuration overview page). Attempts to trigger the job by hand resulted in the same message, even after clearing the error.

We have no problems running Perforce from the command line as the same user. The system does not appear to be overloaded. I tried moving, renaming, and replacing the offending executable to no avail.

We are running QuickBiuld v14.0.19 on Ubuntu 20.04.3 LTS, with OpenJDK 21.0.4+7-Ubuntu-1ubuntu220.04. We cannot easily reboot the server due to the heavy load. Fortunately only a subset of our builds are affected - those using QB built-in Perforce integration. But we are very much in trouble right now.

Please help.

  • replies 3
  • views 117
  • stars 0
#2
robinshen ADMIN ·

Please see my response over email.

1 Reply
#3
drdt ·

I just wanted to follow up in case anyone else encounters this. The situation was that the Ubuntu server did an automated patch of the JDK installation at 6:23am, from OpenJDK 21.0.4+5 to OpenJDK 21.0.4+7. This kind of patch had happened before with no issues, but I guess it happened while one of the libraries was in use by QB due to a build running.

This apparently caused a conflict between the library on disk and the one in memory, so QB was no longer able to launch subshells.

We were able to resolve the issues by restarting the QB server during off-hours, and we also modified the server config to disable unattended automated updates.

#4
robinshen ADMIN ·

Thanks for sharing!

Log in to reply