Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QB3.0.9: Error deleting directory #1148

scastria ·
Of the 2800 builds I perform per week, I am getting maybe 1 or 2 of "Error deleting directory" errors. Any ideas what might be causing it?

Caused by: com.pmease.quickbuild.QuickbuildException: com.pmease.quickbuild.QuickbuildException: Error deleting directory C:\QuickBuild\workspace3\root\lgcbuilds\GGT\petrophysics\DS_5000_6_0_0
at com.pmease.quickbuild.util.FileUtils.deleteDir(FileUtils.java:738)
at com.pmease.quickbuild.util.FileUtils.cleanDir(FileUtils.java:113)
at com.pmease.quickbuild.util.Util.cleanDir(Util.java:161)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:95)
at org.codehaus.groovy.runtime.MetaClassHelper.doMethodInvoke(MetaClassHelper.java:599)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:904)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:740)
at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:761)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:749)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:167)
at script1286306914939.run(script1286306914939.groovy:3)
at com.pmease.quickbuild.plugin.basis.BasisPlugin$13.evaluate(BasisPlugin.java:212)
at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:78)
at com.pmease.quickbuild.setting.step.executeaction.ScriptExecuteAction.execute(ScriptExecuteAction.java:36)
at com.pmease.quickbuild.setting.step.executeaction.ScriptExecuteAction$$EnhancerByCGLIB$$2c5114aa.CGLIB$execute$0(<generated>)
at com.pmease.quickbuild.setting.step.executeaction.ScriptExecuteAction$$EnhancerByCGLIB$$2c5114aa$$FastClassByCGLIB$$5bf5b786.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
at com.pmease.quickbuild.DefaultScriptEngine$Interpolator.intercept(DefaultScriptEngine.java:271)
at com.pmease.quickbuild.setting.step.executeaction.ScriptExecuteAction$$EnhancerByCGLIB$$2c5114aa.execute(<generated>)
at com.pmease.quickbuild.stepsupport.Step.execute(Step.java:459)
at com.pmease.quickbuild.stepsupport.StepJob.execute(StepJob.java:34)
at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:120)
at java.lang.Thread.run(Unknown Source)
Caused by: com.pmease.quickbuild.QuickbuildException: Can not delete directory: C:\QuickBuild\workspace3\root\lgcbuilds\GGT\petrophysics\DS_5000_6_0_0\petrophysics\dependencies\boost\include\boost
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:766)
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:760)
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:760)
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:760)
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:760)
at com.pmease.quickbuild.util.FileUtils.internalDeleteDir(FileUtils.java:760)
at com.pmease.quickbuild.util.FileUtils.deleteDir(FileUtils.java:734)
... 26 more

at com.pmease.quickbuild.plugin.basis.BasisPlugin$13.evaluate(BasisPlugin.java:219)
at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:78)
... 10 more
  • replies 17
  • views 7780
  • stars 0
emack ·
This is happening to us with about 1 out of 30 builds, both with scripting (util.deleteDir) and when calling an external utility (rm.exe). I haven't figured out why, yet.
robinshen ADMIN ·
This happens because the directory is still in use by some other processes. Please make sure that:
1. Your build script does not fork daemon processes accessing any of the workspace files.
2. The workspace is not shared by one or more configurations. If this is necessary, please make sure configurations sharing workspaces never build in the same time.
3. Be careful when access files outside of current workspace in the build script as this may result in multiple builds accessing the same set of files in the same time.
scastria ·
I am not doing any of that and this is the same workflow from QB2 that worked fine for months. I remember in QB2 you made a few changes to the delete directory code. Maybe there is still a bug. I can live with this bug as long as it doesn't fail my build because of it. Any way to prevent a step's post-execute action from causing the step to fail?
robinshen ADMIN ·
The delete directory logic in QB2 is modified to solve this rare JVM issue:
http://forums.sun.com/thread.jspa?trang ... 4&tstart=0

It is possible that directory deleting still experience this as the fix is intended to reduce this issue to the minimum but can not avoid it completely. If it is fine to ignore this error, you may add try...catch block to util.deleteDir(...) and ignore the catched exception. Or you may call Thread.sleep() and System.gc() before calling util.delete() to sleep for enough time.
roger ·
So I wasn't seeing this in QB 2.x but since switching to QB 3.x yesterday have seen it a number of times. This is a big problem for us as our nightly builds clean out the configuration workspace to ensure the build doesn't have any unwanted files still around, however if this fails part way through the delete we're left with Perforce thinking that a lot of the files are still on the HDD when it next comes to check stuff out, which then results in builds failing due to this discrepency.
roger ·
I've added QB-686 for this as it's a critical problem for us and has been seen to fail on a folder that was checked out ~24hrs ago by the previous nightly clean build, however the contents of the folder take no part in the steps that run for this configuration so we shouldn't be causing any files to get locked in that folder. It's possible Windows or a Virus Scanner have stumbled across the folder at just the wrong time, but it wasn't an issue in QB 2.x for us.

-r
robinshen ADMIN ·
I would like to ask all users experiencing this issue: is this only happening on Windows?
emack ·
Hi Robin,
I am running only on Windows, but I will try to set something up on Linux.

Ed
robinshen ADMIN ·
I compared the code between 2.1.x and 3.0.x, and the directory deletion logic is exactly the same. I also tried some load test on Windows pulling many files from SCM, calling external build scripts, and then finally cleaning up the workspace. And hundreds of builds are running without a single failure. However once some process stepped into some directories/files, the deletion fails, for example when I opened a windows command window and change to some directory underneath the workspace.
robinshen ADMIN ·
Please follow below steps to further investigate the issue:
1. download "handle.exe" from here: http://technet.microsoft.com/en-us/sysi ... 96655.aspx
2. copy "handle.exe" into system path
3. run "handle.exe" once and accept the license
4. modify the step cleaning up workspace to execute below script:
groovy:
try {
util.cleanDir(configuration.workspaceDir);
} catch (Exception e) {
util.execute("handle.exe " + "\"" + configuration.workspaceDir.absolutePath.replace("\", "\\") + "\"");
throw e;
}

5. modify the configuration setting to use debug logging mode.
6. when QuickBuild fails to delete workspace again, search the build log for string "Sysinternals". The process locking any files in the directory will be printed right below this line.
scastria ·
I have not tried these steps yet but wanted to add some more information. A few months ago, I noticed that on my own machine using ANT similar behavior ONLY after switching to Win 7. My ANT script would run my clean target would delete my build directory tree. With WinXP, it would work fine every time. However, on Win7, if I used windows explorer to navigate underneath the build directory to view a file and then even close windows explorer, my ANT script would fail to delete the build directory. It wouldn't FAIL, but just quietly not delete everything leaving behind the directory in which I navigated to using windows explorer. If I ran my ANT script clean target over and over, eventually everything would be deleted.

For QB3, we have switched to Win2008 Server R2 which is the same kernel as Win7. For QB2, we would use WinXP. So not only is QB version different, but the OS is different too. So perhaps it is a win OS issue and not a QB3 issue.

Another post of mine in this forum revealed another Win7 specific bug in the way services are launched which required a patch to the Java Service Wrapper.
roger ·
So I've not changed O/S on my machines and I've got a group of XP x64 and Win7 x64. I've not seen this before on QB 2.x, I think the first two nights after the upgrade I had failure, don't think I saw it last night, so I'll wait and see and insert the logic above if we do still get it happening.

-r
emack ·
Robin,
Using your handle.exe suggestion, I discovered the culprit was Diskeeper:

10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Handle v3.42
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Copyright (C) 1997-2008 Mark Russinovich
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Sysinternals - http://www.sysinternals.com
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG -
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - DkService.exe pid: 5820 A6C: E:\dev\qb\continuous-compile\gonzales\tableau-data\build\tdetest\Win32\Release\atan2_test.obj

I have modified Diskeeper's auto-defrag schedule to off-hours.

Thank you!
Ed
roger ·
I've not had this problem again after the initial couple of failures on the first two nights after upgrading to 3.x. Most odd.

-r
robinshen ADMIN ·
[quote="emack"]Robin,
Using your handle.exe suggestion, I discovered the culprit was Diskeeper:

10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Handle v3.42
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Copyright (C) 1997-2008 Mark Russinovich
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - Sysinternals - http://www.sysinternals.com
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG -
10:36:20,221 [master>clean-build-dirs@emack:8811] DEBUG - DkService.exe pid: 5820 A6C: E:\dev\qb\continuous-compile\gonzales\tableau-data\build\tdetest\Win32\Release\atan2_test.obj

I have modified Diskeeper's auto-defrag schedule to off-hours.

Thank you!
Ed[/quote]
Thanks for sharing this info with us!
goydex ·
I am seeing this now on our QB 6.1.17 instance.

We have followed the directions, and despite no handles being indicated as held we still cannot delete the workspace folder.

10:15:44,240 DEBUG - Executing command: reg.exe ADD HKCU\Software\Sysinternals\Handle /v EulaAccepted /t REG_DWORD /d 1 /f
10:15:44,240 DEBUG - Command working directory: Q:\qb\buildagent\bin
10:15:44,307 DEBUG - The operation completed successfully.
10:15:44,308 DEBUG - Executing command: C:\Windows\System32\handle.exe Q:\THE_FOLDER
10:15:44,308 DEBUG - Command working directory: Q:\qb\buildagent\bin
10:15:44,572 DEBUG -
10:15:44,572 DEBUG - Nthandle v4.1 - Handle viewer
10:15:44,572 DEBUG - Copyright (C) 1997-2016 Mark Russinovich
10:15:44,572 DEBUG - Sysinternals - www.sysinternals.com
10:15:44,572 DEBUG -
10:15:44,608 DEBUG - No matching handles found.
10:16:07,521 DEBUG - Executing command: C:\Windows\System32\handle.exe Q:\THE_FOLDER
10:16:07,521 DEBUG - Command working directory: Q:\qb\buildagent\bin
10:16:07,548 DEBUG -
10:16:07,548 DEBUG - Nthandle v4.1 - Handle viewer
10:16:07,548 DEBUG - Copyright (C) 1997-2016 Mark Russinovich
10:16:07,548 DEBUG - Sysinternals - www.sysinternals.com
10:16:07,548 DEBUG -
10:16:07,582 DEBUG - No matching handles found.
10:16:07,595 INFO - Executing post-execute action...
10:16:07,595 ERROR - Step 'master>Clean Workspace with Exception Checking' is failed: Failed to delete file 'Q:\THE_FOLDER'.
10:16:07,721 INFO - Executing post-execute action...
10:16:07,730 ERROR - Step 'master' is failed: Composite step 'master' failed due to unsatisfied success condition.


Whan can you suggest?

The Q:\ drive in this case is not a virtual drive, its a physical disk. It happens on all servers, be they virtual or physical.
robinshen ADMIN ·
How about sleep a while before calling script to delete the directory?