Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Is there any way to avoid stopping buildagent service? #3598

jerry-lee ·
Dear,

We have an critical issue to use QuickBuild to make our own build system.
The issue is buildagent service is missing(actually buildagent status has been changed as stopped, right away) when click the "STOP" in specified a step.
In addition, it has has been stopped even nobody click "STOP" button.
We tried to figure out what happend in buildagent but the log is not available if buildagent is missing, so we couldn't figure it.

If you could, please let me know the way to avoid stopping buildagent even user click the "STOP" or unexpected stopping.

Here is the our QuickBuild installation info.
Version : 6.1.15

Plus, this is the agent side log when buildagent has been detached from start the build to the end.
2016-08-11 18:01:08,889 [qtp499127685-23] WARN  org.eclipse.jetty.servlet.ServletHandler - /download/171562//artifacts/CoreFileError.txt
java.lang.RuntimeException: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171562/artifacts/CoreFileError.txt (No such file or directory)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:144)
at com.pmease.quickbuild.web.DownloadServlet.doGet(DownloadServlet.java:213)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1061)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171562/artifacts/CoreFileError.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:136)
... 31 more
2016-08-11 18:13:32,112 [qtp499127685-22] WARN org.eclipse.jetty.servlet.ServletHandler - /download/171355//artifacts/xxxxx/xxxxxxx.rpm
java.lang.RuntimeException: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171355/artifacts/xxxxx/xxxxxxx.rpm (No such file or directory)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:144)
at com.pmease.quickbuild.web.DownloadServlet.doGet(DownloadServlet.java:213)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1061)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171355/artifacts/xxxxx/xxxxxxx.rpm (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:136)
... 31 more
2016-08-11 18:16:52,740 [qtp499127685-26] WARN org.eclipse.jetty.servlet.ServletHandler - /another_xxxxx/another_xxxxxxx.rpm
java.lang.RuntimeException: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171355/artifacts/another_xxxxx/another_xxxxxxx.rpm (No such file or directory)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:144)
at com.pmease.quickbuild.web.DownloadServlet.doGet(DownloadServlet.java:213)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at com.pmease.quickbuild.Quickbuild$DisableTraceFilter.doFilter(Quickbuild.java:1061)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /home/quickbuild/buildagent/storage/builds/171355/artifacts//another_xxxxx/another_xxxxxxx.rpm (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.pmease.quickbuild.util.ServletUtils.serve(ServletUtils.java:136)
... 31 more
2016-08-11 18:58:10,852 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14914...
2016-08-11 18:58:10,854 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 6447...
2016-08-11 18:58:10,854 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 39530...
2016-08-11 18:58:10,854 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 39508...
2016-08-11 18:58:10,854 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 35922...
2016-08-11 18:58:10,855 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 35813...
2016-08-11 18:58:10,855 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14871...
2016-08-11 18:58:10,855 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14870...
2016-08-11 18:58:10,856 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14863...
2016-08-11 18:58:10,856 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 36252...
2016-08-11 18:58:10,856 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 36251...
2016-08-11 18:58:10,856 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 36248...
2016-08-11 18:58:10,857 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 36247...
2016-08-11 18:58:10,857 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14322...
2016-08-11 18:58:10,878 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14914...
2016-08-11 18:58:10,879 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 6447...
2016-08-11 18:58:10,879 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 39530...
2016-08-11 18:58:10,880 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 39508...
2016-08-11 18:58:10,881 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 35922...
2016-08-11 18:58:10,881 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 35813...
2016-08-11 18:58:10,881 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14871...
2016-08-11 18:58:10,881 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14870...
2016-08-11 18:58:10,882 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 14863...
2016-08-11 18:58:10,882 [pool-1-thread-33] INFO com.pmease.quickbuild.execution.killtree.ProcessTree - Killing process 36252...
2016-08-11 18:58:10,896 [WrapperListener_stop_runner] INFO com.pmease.quickbuild.Quickbuild - Stopping plugin manager...
2016-08-11 18:58:10,898 [WrapperListener_stop_runner] INFO com.pmease.quickbuild.Quickbuild - Stopping QuickBuild agent...


Thanks to review.
  • replies 7
  • views 2215
  • stars 0
robinshen ADMIN ·
Is the agent running linux OS? And is this always happens on particular agent, or happens with different agents?
jerry-lee ·
[quote="robinshen"]Is the agent running linux OS? And is this always happens on particular agent, or happens with different agents?[/quote]

Sorry for late.
First of all, yes, the agent runnling linux which is Ubuntu 14.04.
Obviously, it happened all build agents not on particular agent.
It looks like specified a binary process kills QuickBuild buildagent process.
robinshen ADMIN ·
When QB stops the build, it traverses all the process tree forked in your build script, and I doubt for some reason some forked process mangles with build agent process to cause build agent itself being killed. So I guess it should be particular to some special process being forked in your build script. To verify, can you please do below test?
1. Add a simple test configuration, and add a command build step running on one of your Ubuntu build agent, and specify the build command simply as "sleep 60"
2. Run the test configuration, and then kills the build, to see if the agent is killed.
3. If build agent is not killed in step 2, then modify the build command to run some particular process forked from the normal build script, and kill the build again to see if build agent dies
4. Do this until we find the process in question.
jarig ·
We see the same issue, here is what we've managed to capture via Auditd logging.


6427 - wrapper
\_ 32475 - java
\_ (bunch of processes)
\_ 33463 - bash
\_ 34553 - make
\_ 34556 - make
\_ 34606 - make


Then in auditd log we see that at some point 34606 has ppid=1 and kills our wrapper. I assume that when build stop initiated java killed parent processes before child and thus some child detached and went to under ppid=1.
However, java continued to traverse processes and kill them, so it tackles that detached one (with ppid=1) and starts killing all childs of ppid=1 which includes wrapper itself.
OR
Make tears down itself, but then I don't get why it could kill wrapper, even though their parent are mutual(pid=1).


type=SYSCALL msg=audit(1485793873.338:47516246): arch=c000003e syscall=62 success=yes exit=0 a0=0 a1=f a2=0 a3=7fff116917a0 items=0 ppid=1 pid=34606 auid=7777 uid=43211 gid=43211 euid=43211 suid=43211 fsuid=43211 egid=43211 sgid=43211 fsgid=43211 tty=(none) ses=23617 comm="make" exe="/usr/bin/make" key="kill_signals"
type=EOE msg=audit(1485793873.338:47516245):
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=6429 oauid=7777 ouid=43211 oses=23617 ocomm="java"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=6427 oauid=7777 ouid=43211 oses=23617 ocomm="wrapper-linux-x"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34605 oauid=7777 ouid=43211 oses=23617 ocomm="cmake"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34603 oauid=7777 ouid=43211 oses=23617 ocomm="cmake"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34601 oauid=7777 ouid=43211 oses=23617 ocomm="make"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34592 oauid=7777 ouid=43211 oses=23617 ocomm="make"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34589 oauid=7777 ouid=43211 oses=23617 ocomm="cmake"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34587 oauid=7777 ouid=43211 oses=23617 ocomm="java"



A bit further in log there are calls that shows Java (buildagent) is killing that guilty make above.


type=SYSCALL msg=audit(1485793873.338:47516251): arch=c000003e syscall=62 success=no exit=-3 a0=7f23 a1=f a2=7f23 a3=7f8efa8221c4 items=0 ppid=6427 pid=32475 auid=7777 uid=43211 gid=43211 euid=43211 suid=43211 fsuid=43211 egid=43211 sgid=43211 fsgid=43211 tty=(none) ses=23617 comm="java" exe="/usr/lib/jvm/jdk1.8.0_111/bin/java" key="kill_signals"
type=EOE msg=audit(1485793873.338:47516251):
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=33437 oauid=7777 ouid=43211 oses=23617 ocomm="bash"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=33436 oauid=7777 ouid=43211 oses=23617 ocomm="bash"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=33435 oauid=7777 ouid=43211 oses=23617 ocomm="bash"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=33402 oauid=7777 ouid=43211 oses=23617 ocomm="java"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=32550 oauid=7777 ouid=43211 oses=23617 ocomm="orient"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=32548 oauid=7777 ouid=43211 oses=23617 ocomm="sh"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=32547 oauid=7777 ouid=43211 oses=23617 ocomm="sh"
type=OBJ_PID msg=audit(1485793873.338:47516246): opid=34606 oauid=7777 ouid=43211 oses=23617 ocomm="make"
robinshen ADMIN ·
Can you please check in the agent log to find below messages for this particular auditd event?
Killing process <process id>...

This will tell what processes got killed by QB agent with the order.
jarig ·
I have no logs from previous case, but here is from the recent one (happened just now). ppid of guilty process now is not 1, however following parent chain leads to ppid=1 eventually.


ppid=1
\_ 22647 - make
\_ 22650 - make
\_ 22747 - make
\_ 23104 - make <-- this one killed wrapper


The topmost make with ppid=1, couldn't find original ppid.
I assume it was sh at some point in time; our auditd log tracks only kill processes, so I think sh could just crash/quit leaving makes orphant (and thus make them leak to under ppid=1).


type=SYSCALL msg=audit(1487135999.222:2463672): arch=c000003e syscall=62 success=yes exit=0 a0=587a a1=f a2=0 a3=7ffe38d6eaa0 items=0 ppid=1 pid=22647 auid=4294967295 uid=43211 gid=43211 euid=43211 suid=43211 fsuid=43211 egid=43211 sgid=43211 fsgid=43211 tty=(none) ses=4294967295 comm="make" exe="/usr/bin/make" key="kill_signals"




type=SYSCALL msg=audit(1487135999.222:2463681): arch=c000003e syscall=62 success=yes exit=0 a0=0 a1=f a2=0 a3=7ffcd1cf42a0 items=0 ppid=22747 pid=23104 auid=4294967295 uid=43211 gid=43211 euid=43211 suid=43211 fsuid=43211 egid=43211 sgid=43211 fsgid=43211 tty=(none) ses=4294967295 comm="make" exe="/usr/bin/make" key="kill_signals"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=21572 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=1106 oauid=-1 ouid=43211 oses=-1 ocomm="java"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=1104 oauid=-1 ouid=43211 oses=-1 ocomm="wrapper-linux-x"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23050 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23048 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23044 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23003 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22786 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22755 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22747 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22650 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22647 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22482 oauid=-1 ouid=43211 oses=-1 ocomm="bash"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22455 oauid=-1 ouid=43211 oses=-1 ocomm="bash"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22454 oauid=-1 ouid=43211 oses=-1 ocomm="bash"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22453 oauid=-1 ouid=43211 oses=-1 ocomm="bash"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=22425 oauid=-1 ouid=43211 oses=-1 ocomm="java"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=21575 oauid=-1 ouid=43211 oses=-1 ocomm="orient"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=21573 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23106 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23105 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23104 oauid=-1 ouid=43211 oses=-1 ocomm="make"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23096 oauid=-1 ouid=43211 oses=-1 ocomm="as"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23095 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23093 oauid=-1 ouid=43211 oses=-1 ocomm="arm-linux-andro"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23092 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23079 oauid=-1 ouid=43211 oses=-1 ocomm="as"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23078 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23077 oauid=-1 ouid=43211 oses=-1 ocomm="arm-linux-andro"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23076 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23057 oauid=-1 ouid=43211 oses=-1 ocomm="as"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23056 oauid=-1 ouid=43211 oses=-1 ocomm="cc1plus"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23055 oauid=-1 ouid=43211 oses=-1 ocomm="arm-linux-andro"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23054 oauid=-1 ouid=43211 oses=-1 ocomm="sh"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23051 oauid=-1 ouid=43211 oses=-1 ocomm="as"
type=OBJ_PID msg=audit(1487135999.222:2463681): opid=23107 oauid=-1 ouid=43211 oses=-1 ocomm="make"




wrapper | TERM trapped. Shutting down.
jvm 1 | 2017-02-15 05:19:59,216 INFO - Killing process 23101...
jvm 1 | 2017-02-15 05:19:59,220 INFO - Killing process 23100...
jvm 1 | 2017-02-15 05:19:59,220 INFO - Killing process 23099...
jvm 1 | 2017-02-15 05:19:59,220 INFO - Killing process 23003...
jvm 1 | 2017-02-15 05:19:59,221 INFO - Killing process 23009...
jvm 1 | 2017-02-15 05:19:59,221 INFO - Killing process 23001...
jvm 1 | 2017-02-15 05:19:59,221 INFO - Killing process 23000...
jvm 1 | 2017-02-15 05:19:59,221 INFO - Killing process 22723...
jvm 1 | 2017-02-15 05:19:59,222 INFO - Killing process 23044...
jvm 1 | 2017-02-15 05:19:59,222 INFO - Killing process 23045...
jvm 1 | 2017-02-15 05:19:59,223 INFO - Killing process 23043...
jvm 1 | 2017-02-15 05:19:59,223 INFO - Killing process 23042...
jvm 1 | 2017-02-15 05:19:59,223 INFO - Killing process 23088...
jvm 1 | 2017-02-15 05:19:59,223 INFO - Killing process 23089...
jvm 1 | 2017-02-15 05:19:59,224 INFO - Killing process 23087...
jvm 1 | 2017-02-15 05:19:59,224 INFO - Killing process 23086...
jvm 1 | 2017-02-15 05:19:59,224 INFO - Killing process 22747... <--- parent make of make that killed wrapper
jvm 1 | 2017-02-15 05:19:59,224 INFO - Killing process 23050...
jvm 1 | 2017-02-15 05:19:59,224 INFO - Killing process 23051...
jvm 1 | 2017-02-15 05:19:59,225 INFO - Killing process 23049...
jvm 1 | 2017-02-15 05:19:59,225 INFO - Killing process 23048...
jvm 1 | 2017-02-15 05:19:59,225 INFO - Killing process 23056...
jvm 1 | 2017-02-15 05:19:59,230 INFO - Killing process 23057...
jvm 1 | 2017-02-15 05:19:59,230 INFO - Killing process 23055...
jvm 1 | 2017-02-15 05:19:59,231 INFO - Killing process 23054...
jvm 1 | 2017-02-15 05:19:59,231 INFO - Killing process 22755...
jvm 1 | 2017-02-15 05:19:59,231 INFO - Killing process 23078...
jvm 1 | 2017-02-15 05:19:59,231 INFO - Killing process 23079...
jvm 1 | 2017-02-15 05:19:59,231 INFO - Killing process 23077...
jvm 1 | 2017-02-15 05:19:59,232 INFO - Killing process 23076...
jvm 1 | 2017-02-15 05:19:59,232 INFO - Killing process 23095...
jvm 1 | 2017-02-15 05:19:59,232 INFO - Killing process 23096...
jvm 1 | 2017-02-15 05:19:59,232 INFO - Killing process 23093...
jvm 1 | 2017-02-15 05:19:59,232 INFO - Killing process 23092...
jvm 1 | 2017-02-15 05:19:59,233 INFO - Killing process 22786...
jvm 1 | 2017-02-15 05:19:59,233 INFO - Killing process 22650...
jvm 1 | 2017-02-15 05:19:59,233 INFO - Killing process 22647... <-- make that had ppid=1 in auditd log
jvm 1 | 2017-02-15 05:19:59,233 INFO - Killing process 22482...
jvm 1 | 2017-02-15 05:19:59,234 INFO - Killing process 22455...
jvm 1 | 2017-02-15 05:19:59,234 INFO - Killing process 22454...
jvm 1 | 2017-02-15 05:19:59,234 INFO - Killing process 22453...
jvm 1 | 2017-02-15 05:19:59,234 INFO - Killing process 22425...
jvm 1 | 2017-02-15 05:19:59,235 INFO - Killing process 21575...
jvm 1 | 2017-02-15 05:19:59,235 INFO - Killing process 21573...
jvm 1 | 2017-02-15 05:19:59,235 INFO - Killing process 21572...
jvm 1 | 2017-02-15 05:19:59,241 INFO - Killing process 23101...
jvm 1 | 2017-02-15 05:19:59,241 INFO - Killing process 23100...
jvm 1 | 2017-02-15 05:19:59,241 INFO - Killing process 23099...
jvm 1 | 2017-02-15 05:19:59,241 INFO - Killing process 23003...
jvm 1 | 2017-02-15 05:19:59,242 INFO - Killing process 23009...
jvm 1 | 2017-02-15 05:19:59,242 INFO - Killing process 23001...
jvm 1 | 2017-02-15 05:19:59,242 INFO - Killing process 23000...
jvm 1 | 2017-02-15 05:19:59,242 INFO - Killing process 22723...
jvm 1 | 2017-02-15 05:19:59,243 INFO - Killing process 23044...
jvm 1 | 2017-02-15 05:19:59,243 INFO - Killing process 23045...
jvm 1 | 2017-02-15 05:19:59,243 INFO - Killing process 23043...
jvm 1 | 2017-02-15 05:19:59,243 INFO - Killing process 23042...
jvm 1 | 2017-02-15 05:19:59,243 INFO - Killing process 23088...
jvm 1 | 2017-02-15 05:19:59,244 INFO - Killing process 23089...
jvm 1 | 2017-02-15 05:19:59,244 INFO - Killing process 23087...
jvm 1 | 2017-02-15 05:19:59,244 INFO - Killing process 23086...
jvm 1 | 2017-02-15 05:19:59,244 INFO - Killing process 22747...
jvm 1 | 2017-02-15 05:19:59,244 INFO - Killing process 23050...
jvm 1 | 2017-02-15 05:19:59,245 INFO - Killing process 23051...
jvm 1 | 2017-02-15 05:19:59,245 INFO - Killing process 23049...
jvm 1 | 2017-02-15 05:19:59,245 INFO - Killing process 23048...
jvm 1 | 2017-02-15 05:19:59,245 INFO - Killing process 23056...
jvm 1 | 2017-02-15 05:19:59,246 INFO - Killing process 23057...
jvm 1 | 2017-02-15 05:19:59,246 INFO - Killing process 23055...
jvm 1 | 2017-02-15 05:19:59,246 INFO - Killing process 23054...
jvm 1 | 2017-02-15 05:19:59,246 INFO - Killing process 22755...
jvm 1 | 2017-02-15 05:19:59,254 INFO - Killing process 23078...
jvm 1 | 2017-02-15 05:19:59,254 INFO - Killing process 23079...
jvm 1 | 2017-02-15 05:19:59,255 INFO - Killing process 23077...
jvm 1 | 2017-02-15 05:19:59,255 INFO - Killing process 23076...
jvm 1 | 2017-02-15 05:19:59,255 INFO - Killing process 23095...
jvm 1 | 2017-02-15 05:19:59,255 INFO - Killing process 23096...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 23093...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 23092...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 22786...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 22650...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 22647...
jvm 1 | 2017-02-15 05:19:59,256 INFO - Killing process 22482...
jvm 1 | 2017-02-15 05:19:59,257 INFO - Killing process 22455...
jvm 1 | 2017-02-15 05:19:59,257 INFO - Killing process 22454...
jvm 1 | 2017-02-15 05:19:59,257 INFO - Killing process 22453...
jvm 1 | 2017-02-15 05:19:59,257 INFO - Killing process 22425...
jvm 1 | 2017-02-15 05:19:59,282 INFO - Stopping plugin manager...
jvm 1 | 2017-02-15 05:19:59,288 WARN - Plugin is now stopped
robinshen ADMIN ·
So do you mean that it is your make process actually killing wrapper?