Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

An improvement relate to Memory Management (JVM) in parrallel steps. #4569

ngocanhnu ·

Hi@robinshen
Recently we have some error build when using parrallel steps.
Here is the log that I found on server:

Error notifying task node of job finishing (job class: $job_class, job id: com.pmease.quickbuild.resource.ResourceAllocateJob, task node: $qbagent_name:$qbagent_port)
 com.pmease.quickbuild.RemotingException: Java heap space

We use the above config in our agent:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=8192

Because of our privacy policy, I cannot share the actual content of step. But I think you can reproduce the error like above.

  1. Create a java program to simulate java.lang.OutOfMemoryError in Java
import java.util.Vector;

public class MemoryEater
{
          public static void main(String[] args)
                    {
                                Vector v = new Vector();
                                long endTime = System.currentTimeMillis() + 10000;
                                    while (System.currentTimeMillis() <endTime)
                                                {
                                                              byte b[] = new byte[14856];
                                                                    v.add(b);
                                                                          Runtime rt = Runtime.getRuntime();
                                                                                System.out.println( "free memory: " + rt.freeMemory() );
                                                                                    }
                                      }
}
  1. run script to make 200 java program like in step 1
#!/bin/bash
last=200
for i in $(seq 1 $last);
do
        cp MemoryEater.java MemoryEater$i.java
        sed -i "s/MemoryEater/MemoryEater$i/g" MemoryEater$i.java
        javac MemoryEater$i.java
done
  1. Run the 200 parrallel steps with 200 file abve with command:
java -Xmx2048m MemoryEater${params.get("NUMBER")}

The config in repeat parameters:

Parameter Name : NUMBER
Parameter Values:
groovy:
def List numbers=[]
for(int i in 1..200) numbers.add(i.toString())
return numbers

  1. Run the build with max i in different cases (200/145/140/100) and the result is different.
    The suspicious thing is when running in parallel with under 145 files the build can success and when increase the number build start failing.
    Can you try to figure out why the agent only can run this specific numbers?

We change the Parallel step to Sequential step, the issue is resolve but the time to run it is terrible slow compare with Parallel step.
I change the config wrapper.java.maxmemory=8192MB to 12GB. But it is pass the build randomly so we donot think this is greate solution.

  • replies 5
  • views 543
  • stars 2
robinshen ADMIN ·

Are the parallel composition step and script step running on same node? Also please show me the full error reported relating to "com.pmease.quickbuild.RemotingException: Java heap space" (if the build log does not have the detail, please check console log of related agent).

ngocanhnu ·

Yes, the parallel composition step and script step running on the same node.
The log on build:

Step 'master>parallel_composition_steps' is failed.
java.lang.RuntimeException: Error executing step process job.
        at com.pmease.quickbuild.stepsupport.StepProcessTask.reduce(StepProcessTask.java:126)
        at com.pmease.quickbuild.stepsupport.StepProcessTask.reduce(StepProcessTask.java:19)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:168)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:172)
        at com.pmease.quickbuild.stepsupport.ParallelStep.triggerChildren(ParallelStep.java:83)
        at com.pmease.quickbuild.stepsupport.CompositeStep.run(CompositeStep.java:133)
        at com.pmease.quickbuild.stepsupport.Step.doExecute(Step.java:661)
        at com.pmease.quickbuild.stepsupport.Step.execute(Step.java:575)
        at com.pmease.quickbuild.stepsupport.StepExecutionJob.executeStepAwareJob(StepExecutionJob.java:31)
        at com.pmease.quickbuild.stepsupport.StepAwareJob.executeBuildAwareJob(StepAwareJob.java:56)
        at com.pmease.quickbuild.BuildAwareJob.execute(BuildAwareJob.java:79)
        at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
    Caused by: java.lang.OutOfMemoryError: Java heap space

The log on agent:

ERROR com.pmease.quickbuild.grid.GridTaskFuture - Unable to find job (job class: com.pmease.quickbuild.resource.ResourceAllocateJob, job id: xxx, build id: N/A, job node: XXX)```
robinshen ADMIN ·

What is your agent OS and QB version?

ngocanhnu ·

QB version: 10.0.x
Agent OS: Ubuntu 18.04

robinshen ADMIN ·

The java program you created each runs OOM and parallel execution many of them will cause OS running short of memory, and it is not odd that QB server itself being impacted when requests memory from OS. And the error may occur randomly depending on when OS rejects memory request when the overall memory is low.