Skip to content

Commit

Permalink
SlaveDirectoryUploadCommand: Make more robust
Browse files Browse the repository at this point in the history
On Windows, we've been repeatedly receiving exceptions from the
slave-side, because the buildslave process wasn't able to remove the
temporary file after the upload has finished.

This buildslave is running three builds in parallel at a time. We don't
see this issues when running just one build at a time, surprisingly.

This patch reduces the number of file objects created for an OS-level handle and seems to fully fix the issue.

Multiple reports about this issue:
- https://lists.buildbot.net/pipermail/devel/2009-December/005513.html
- http://trac.buildbot.net/ticket/3052

Fixes http://trac.buildbot.net/ticket/3052

Exception captured on the build slave on a Windows 10 machine:
```
<buildslave.commands.transfer.SlaveDirectoryUploadCommand instance at 0x0000000003947248>
2017-03-23 05:22:44-0700 [Broker,client] Unhandled Error
	Traceback (most recent call last):
	  File "c:\Python27\lib\site-packages\twisted\spread\pb.py", line 565, in expressionReceived
	    method(*sexp[1:])
	  File "c:\Python27\lib\site-packages\twisted\spread\pb.py", line 935, in proto_answer
	    d.callback(self.unserialize(netResult))
	  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 383, in callback
	    self._startRunCallbacks(result)
	  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 491, in _startRunCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 578, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "c:\Python27\lib\site-packages\buildslave\commands\transfer.py", line 248, in finished
	    os.remove(self.tarname)
	exceptions.WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\foo\\appdata\\local\\temp\\tmp4altyc'
```
  • Loading branch information
krf committed Mar 27, 2017
1 parent 369ea01 commit 2f191d6
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 4 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
:py:class: `~buildbot_worker.commands.transfer.SlaveDirectoryUploadCommand` no longer throws exceptions because the file "is used by another process" under Windows
10 changes: 6 additions & 4 deletions worker/buildbot_worker/commands/transfer.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,20 +214,22 @@ def start(self):

# Create temporary archive
fd, self.tarname = tempfile.mkstemp()
fileobj = os.fdopen(fd, 'wb')
self.fp = os.fdopen(fd, "rb+")

if self.compress == 'bz2':
mode = 'w|bz2'
elif self.compress == 'gz':
mode = 'w|gz'
else:
mode = 'w'
archive = tarfile.open(name=self.tarname, mode=mode, fileobj=fileobj)
# TODO: Use 'with' when depending on Python 2.7
# Not possible with older versions: exceptions.AttributeError: 'TarFile' object has no attribute '__exit__'
archive = tarfile.open(mode=mode, fileobj=self.fp)
archive.add(self.path, '')
archive.close()
fileobj.close()

# Transfer it
self.fp = open(self.tarname, 'rb')
self.fp.seek(0)

self.sendStatus({'header': "sending %s" % self.path})

Expand Down

0 comments on commit 2f191d6

Please sign in to comment.