Your browser was unable to load all of the resources. They may
have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.
QB server and agent has to establish bi-directional connections. I guess the agent has a firewall rejecting connections from server. To workaround the issue, please make sure below command runs successfully on QB server:
telnet 172.16.4.2 22
Looked like the agent was indeed dead.
now after bringing up the agent we are still facing the issue where none of the requests are going through. We do have many other agents which are active.
Now we see the below error more consistently and nothing happens to the request which is queued:
ERROR com.pmease.quickbuild.rest.providers.AccessDeniedExceptionMapper - Access denied when accessing restful service.
com.pmease.quickbuild.AccessDeniedException: Access denied.
at com.pmease.quickbuild.rest.resources.BuildRequestResource.save(BuildRequestResource.java:56)
at sun.reflect.GeneratedMethodAccessor450.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:149)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:259)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:990)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:941)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:932)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:384)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:451)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:632)
at com.pmease.quickbuild.rest.RestServlet.service(RestServlet.java:42)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at com.pmease.quickbuild.web.HttpServiceProvider$1$1.service(HttpServiceProvider.java:86)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:318)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:380)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:748)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
If the agent still has this issue, please make sure the JVM is installed properly. Or please switch to a different JVM for test. As to the access denied problem, it seems that you are triggering build programmatically via RESTful API. If so, please make sure the user requesting build has the permission of "RUN_BUILD" in specified configuration.
Thanks Robin for the input.
What we are actually facing is that the request and the steps itself are taking a lot of time to complete but they are not failing.
I see that the request and the steps are going through but it just takes way lot of time (~4 - 5 times the actual time) to complete each step.
I looked at the access to the p4 and it seems to be working fine. The builds themselves on the agents are working fine but the steps are taking more time to update.
I see some combination of errors like the ones i pasted before and also the below one
ERROR com.pmease.quickbuild.RemotingSerializerFactory - Error invoking hessian service.
com.caucho.hessian.client.HessianConnectionException: 500: java.net.SocketException: Unexpected end of file from server
at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:195)
at $Proxy53.ping(Unknown Source)
at com.pmease.quickbuild.grid.DefaultAgentManager.buildAgentActive(DefaultAgentManager.java:163)
at com.pmease.quickbuild.grid.ConnectServlet.connect(ConnectServlet.java:113)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:192)
at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:110)
at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:416)
at com.pmease.quickbuild.grid.ConnectServlet.service(ConnectServlet.java:59)
at com.pmease.quickbuild.web.HttpServiceProvider$1$1.service(HttpServiceProvider.java:86)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:59)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:318)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:380)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:835)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.reflect.GeneratedConstructorAccessor292.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1368)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1362)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1016)
at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:177)
... 29 more
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:766)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:166)
... 29 more
I checked the memory consumption on the server info and its about ~35 %
Anything i could check to figure out why the steps are consuming so much time?
Looks like some agent experiences connection problems to server. Please identify those agents and remove them out of grid to see if things get better.
Thanks Robin!!
I looked at the agents and I find that most of them are in good state.
I see the following error on couple of them :
BuildAgent Scripts'...
2011-07-08 03:51:05,140 ERROR - Error finishing grid job.
com.pmease.quickbuild.RemotingException: java.io.IOException: Bad file descriptor
at com.pmease.quickbuild.bootstrap.BootstrapUtils.wrapAsUnchecked(BootstrapUtils.java:50)
at com.pmease.quickbuild.util.ExceptionUtils.wrapAsUnchecked(ExceptionUtils.java:83)
at com.pmease.quickbuild.log.BuildLogger.append(BuildLogger.java:84)
at com.pmease.quickbuild.model.Build.appendLog(Build.java:450)
at com.pmease.quickbuild.grid.GridTaskFuture.jobFinished(GridTaskFuture.java:102)
at com.pmease.quickbuild.grid.NodeServiceImpl.jobFinished(NodeServiceImpl.java:326)
at sun.reflect.GeneratedMethodAccessor385.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:192)
at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:110)
at com.caucho.hessian.server.HessianServlet.service(HessianServlet.java:416)
at com.pmease.quickbuild.grid.GridRemotingServlet.service(GridRemotingServlet.java:32)
at com.pmease.quickbuild.web.HttpServiceProvider$1$1.service(HttpServiceProvider.java:86)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:59)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:318)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:380)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:835)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
But i see that the agent is running fine and did recover from this error.
Can this error cause the degrade in performance? Why would this error occur?
It should not, but to narrow down the problem, please stop these agents to see if performance is fine. Also you may edit "bin/config.(bat|sh)" for the problematic agents to specify a fixed agent IP to see if it solves the issue.
Also please check if there are some anti-virus software installed on those agents. We ever had a customer experiencing build slowdowns and it turns out to be anti-virus issue.
Slowness was actually caused by one of the faulty database drives.
