Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

OOM error generating buildstats #4482

drdt ·

I have as job that I run once in a while to cleanup and regenerate the build statistics reports (using the "Rebuild Statistics" step).
However, for my job which has a lot of builds in it history, I got an error "GC overhead limit exceeded".

Is this a practical limit or can it be increased somehow? Would adding more memory to the box help? How much memory should I need to calculate the statistics on 35,000 build records? (I know this is a lot, but my team likes to keep things forever.)

Could perhaps an improvement be made to limit the statistics collection to a certain range?

16:11:52,506 INFO  - Cleaning up statistics (name:Build Stats, configuration:root/legacy/Builds)...
16:16:09,255 INFO  - Executing post-execute action...
16:16:09,792 ERROR - Step 'master>rebuild legacy build stats' is failed.
  java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.pmease.quickbuild.bootstrap.BootstrapUtils.wrapAsUnchecked(BootstrapUtils.java:56)
    at com.pmease.quickbuild.util.ExceptionUtils.wrapAsUnchecked(ExceptionUtils.java:82)
    at com.pmease.quickbuild.stepsupport.Step.doExecute(Step.java:689)
    at com.pmease.quickbuild.stepsupport.Step.execute(Step.java:577)
    at com.pmease.quickbuild.stepsupport.StepExecutionJob.executeStepAwareJob(StepExecutionJob.java:31)
    at com.pmease.quickbuild.stepsupport.StepAwareJob.executeBuildAwareJob(StepAwareJob.java:56)
    at com.pmease.quickbuild.BuildAwareJob.execute(BuildAwareJob.java:77)
    at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:131)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
  • replies 5
  • views 660
  • stars 0
robinshen ADMIN ·

Is this the complete stack trace? How much server memory have you configured in conf/wrapper.conf via wrapper.java.maxmemory?

drdt ·

Thank you for the quick response. Yes, that is the entire stack dump.

wrapper.java.initmemory=512
wrapper.java.maxmemory=4096

The box has 4 cores, 8192gb RAM.

robinshen ADMIN ·

Please increase maxmemory to 6G to see if it can handle this many builds.

drdt ·

I finally got to a safe place to increase the Java memory to 6G. This time, it ran for about an hour before crashing. And it brought down the entire server (much to my chagrin; not sure if it did this the last time).

Can we look at a way to limit this activity before it reaches this point?

robinshen ADMIN ·

Can you show me the full stack trace?