At a time when the system is largely idle, certain automated jobs on our production QB server started to experience the following errors:

2024-11-12 13:32:27,751 [pool-2-thread-571222] ERROR com.pmease.quickbuild.DefaultBuildEngine -
Error processing build request 4df57b8a-7d65-4990-8acb-6f001eba691e
java.lang.RuntimeException: Error executing check condition job.
Caused by: com.pmease.quickbuild.QuickbuildException:
Cannot run program "p4" (in directory "/opt/qbserver-8810/bin"): error=0, Failed to exec spawn helper: pid: 1964899, exit value: 1

A job worked at 6:17 am, and then at 6:24 it would not start (no jobs would start) due to the above message (showing up in a red bar on the Configuration overview page). Attempts to trigger the job by hand resulted in the same message, even after clearing the error.

We have no problems running Perforce from the command line as the same user. The system does not appear to be overloaded. I tried moving, renaming, and replacing the offending executable to no avail.

We are running QuickBiuld v14.0.19 on Ubuntu 20.04.3 LTS, with OpenJDK 21.0.4+7-Ubuntu-1ubuntu220.04. We cannot easily reboot the server due to the heavy load. Fortunately only a subset of our builds are affected - those using QB built-in Perforce integration. But we are very much in trouble right now.

Please help.