`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux) #546

altendky · 2016-10-27T12:08:11Z

I am checking out repositories into a temporary directory, doing various things with them including calling tree(), and then deleting the directories and the temporary directory. This works fine in Linux but not in Windows. I have created this SSCCE to show the issue. I will take some time right now to try to attempt to fix it myself but I like to document my issues first (so others know and just in case someone else sees it and has insight).

Note that I am making an effort (del_rw()) to clear read-only items.

I am testing on:

Windows 10 Enterprise 64-bit (in VirtualBox)
Python 3.5.2 32-bit

t.py.txt

Output:

(venv) C:\Users\IEUser\Desktop\407>python t.py
Traceback (most recent call last):
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 40, in <module>
    shutil.rmtree(dir, onerror=del_rw)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "t.py", line 19, in del_rw
    os.remove(name)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 40, in <module>
    shutil.rmtree(dir, onerror=del_rw)
  File "C:\Users\IEUser\Desktop\407\venv\lib\tempfile.py", line 808, in __exit__
    self.cleanup()
  File "C:\Users\IEUser\Desktop\407\venv\lib\tempfile.py", line 812, in cleanup
    _shutil.rmtree(self.name)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "t.py", line 42, in <module>
    raise PermissionError('Failed with tree (GitPython {})'.format(gitpython_hash)) from e
PermissionError: Failed with tree (GitPython 5149c807ec5f396c1114851ffbd0f88d65d4c84f)

altendky · 2016-10-27T13:17:16Z

After the tree() call there are two entire Git processes that are left open... Pausing in the debugger and killing those process trees manually (via Process Explorer) does allow clean deletion of the directories.

The 'offending' instances of Git are launched per the below stack traces. I did snip the last few stack frames that were for the PyCharm debugger activities.

>>> import traceback; traceback.print_stack()
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 1580, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 964, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/IEUser/Desktop/407/t.py", line 40, in <module>
    tree = repo.tree('master')
  File "c:\users\ieuser\desktop\407\gitpython\git\repo\base.py", line 442, in tree
    return self.rev_parse(text_type(rev) + "^{tree}")
  File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 193, in rev_parse
    obj = name_to_object(repo, rev[:start])
  File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 130, in name_to_object
    return Object.new_from_sha(repo, hex_to_bin(hexsha))
  File "c:\users\ieuser\desktop\407\gitpython\git\objects\base.py", line 64, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "c:\users\ieuser\desktop\407\gitpython\git\db.py", line 37, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 930, in get_object_header
    cmd = self._get_persistent_cmd("cat_file_header", "cat_file", batch_check=True)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 913, in _get_persistent_cmd
    cmd = self._call_process(cmd_name, *args, **options)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 868, in _call_process
    return self.execute(call, **_kwargs)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 586, in execute
    proc = Popen(command,

and

>>> import traceback; traceback.print_stack()
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 1580, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 964, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/IEUser/Desktop/407/t.py", line 40, in <module>
    tree = repo.tree('master')
  File "c:\users\ieuser\desktop\407\gitpython\git\repo\base.py", line 442, in tree
    return self.rev_parse(text_type(rev) + "^{tree}")
  File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 216, in rev_parse
    obj = to_commit(obj).tree
  File "C:\Users\IEUser\Desktop\407\venv\lib\site-packages\gitdb\util.py", line 237, in __getattr__
    self._set_cache_(attr)
  File "c:\users\ieuser\desktop\407\gitpython\git\objects\commit.py", line 143, in _set_cache_
    binsha, typename, self.size, stream = self.repo.odb.stream(self.binsha)  # @UnusedVariable
  File "c:\users\ieuser\desktop\407\gitpython\git\db.py", line 42, in stream
    hexsha, typename, size, stream = self._git.stream_object_data(bin_to_hex(sha))
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 947, in stream_object_data
    cmd = self._get_persistent_cmd("cat_file_all", "cat_file", batch=True)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 913, in _get_persistent_cmd
    cmd = self._call_process(cmd_name, *args, **options)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 868, in _call_process
    return self.execute(call, **_kwargs)
  File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 586, in execute
    proc = Popen(command,

altendky · 2016-10-27T14:02:00Z

Does not seem to work (Still throws exception when deleting directory. I guess this just means it's not setup as a context manager):

with git.Repo.clone_from(url, dir) as repo:
    with repo.tree('master') as tree:
        pass

Does work (No exception when deleting directory):

repo = git.Repo.clone_from(url, dir)

tree = repo.tree('master')
del repo
del tree

So, perhaps it is not a bug and I simply need to make sure that my repo and tree get deleted?

altendky · 2016-10-27T14:27:32Z

Better 'workaround':

tree = repo.tree('master')
repo.git.clear_cache()


          Workaround in collectdevices.py for bug in gitpython

gitpython-developers/GitPython#546 Ref T407

Byron · 2016-12-08T12:04:08Z

@altendky Thanks for posting your progress here, as I do hope that others will find it useful! It's a known limitation of GitPython and its API, and best I can offer right now is to manually handle it by implementing some sort of workaround.


          tests: win: don't cleanup processes on Popen

Caused by two bugs: https://bugs.python.org/issue37380 and gitpython-developers/GitPython#546 (comment) Signed-off-by: Ruslan Kuprieiev <[email protected]>


          tests: win: don't cleanup processes on Popen

Caused by two bugs: https://bugs.python.org/issue37380 and gitpython-developers/GitPython#546 (comment) Signed-off-by: Ruslan Kuprieiev <[email protected]>


          tests: win: don't cleanup processes on Popen

Workaround for two bugs: https://bugs.python.org/issue37380 and gitpython-developers/GitPython#546 Signed-off-by: Ruslan Kuprieiev <[email protected]>


          tests: win: don't cleanup processes on Popen

Workaround for two bugs: https://bugs.python.org/issue37380 and gitpython-developers/GitPython#546 Signed-off-by: Ruslan Kuprieiev <[email protected]>


          tests: win: don't cleanup processes on Popen

Workaround for two bugs: https://bugs.python.org/issue37380 and gitpython-developers/GitPython#546 Signed-off-by: Ruslan Kuprieiev <[email protected]>

ankostis added tag.Windows tag.leaks labels Oct 27, 2016

ankostis mentioned this issue Oct 30, 2016

Possible leak in double-booking cursor gitpython-developers/smmap#31

Open

altendky added a commit to altendky/st that referenced this issue Oct 31, 2016

Workaround in collectdevices.py for bug in gitpython

6a12f70

gitpython-developers/GitPython#546 Ref T407

Byron added the acknowledged label Dec 8, 2016

slacAWallace mentioned this issue Oct 31, 2018

GipPython lock folder with repo on windows so it could not be removed #387

Open

efiop mentioned this issue Jun 23, 2019

tests: win: don't cleanup processes on Popen iterative/dvc#2174

Merged

2 of 2 tasks complete

This was referenced Jul 15, 2019

Fix test suite clean up (2019.2.1) saltstack/salt#53853

Closed

Fix test suite clean up (Neon) saltstack/salt#53854

Closed

ishepard mentioned this issue Jan 23, 2020

Added process metrics and refactored code of previous ones ishepard/pydriller#75

Merged

gitpython-developers / GitPython

`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux) #546

`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux) #546

altendky commented Oct 27, 2016

altendky commented Oct 27, 2016

altendky commented Oct 27, 2016 •

edited

altendky commented Oct 27, 2016

Byron commented Dec 8, 2016

gitpython-developers / GitPython

Join GitHub today

`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux) #546

`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux) #546

Comments

altendky commented Oct 27, 2016

altendky commented Oct 27, 2016

altendky commented Oct 27, 2016 • edited

altendky commented Oct 27, 2016

Byron commented Dec 8, 2016

altendky commented Oct 27, 2016 •

edited