Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upHang in git.Repo.clone_from() [Inter-process Communication Problem] #72
Comments
Fix hanging when clone or pull execute See http://www.popekim.com/2008/12/never-use-pipe-with-python-popen.html
Fix hanging when clone or pull execute See http://www.popekim.com/2008/12/never-use-pipe-with-python-popen.html Fix version parsing for msysgit
|
I am having the same problem. I tried the solution posted above and the clone command starts working. I am sure that this workaround cannot be used permanently since stderr would always be compromised. Is there, perhaps, a workaround that can be applied in a script that uses gitpython? I tried various things like closing or re-directing stderr in my script. Nothing worked. Environment: 64-bit, Windows 7, Service Pack 1, gitpython-0.3.2.rc1. |
|
I have recently written other process-handling code and used a select loop to read both channels in a single thread without blocking. This is what one should do here as well. |
|
Thanks for the suggestion, Byron. Unfortunately, this will not work on Windows. Well, at least the way I tried it: File "./a_clone_test.py", line 20, in main
rlist,wlist,xlist = select.select([], [sys.stderr], [])
select.error: (10038, 'An operation was attempted on something that is not a soc
ket')
my@myhost /c/cygwin/home/me/workspaces/infra/scripts (master)So, the documentation is accurate, you can only select sockets in Windows, i.e. something that came from winsock. |
|
The git process handling is a major point of improvement in 0.3.5, as it is causing plenty of issues and has not been fixed sufficiently well yet. |
|
Process handling was improved to the point where pipes are unable to run full and block. |
|
@Byron unfortunately we've still seen this issue with gitpython 1.0.1 on windows. i didn't follow all the codepaths, but using i'm not exactly familiar with the implementation, but https://github.com/gitpython-developers/GitPython/blob/master/git/cmd.py#L661 still seems to have a codepath, which doesn't use the |
We keep stdout closed, which seems to have the side-effect of stdout being connected to your TTY, in case you run a terminal. However, this shold also prevent deadlocks, as only stderr is used. The alternative would have been to try to fetch lines concurrently, and we have been there. For clone(), `communicate()` is used, and with some luck this will just do the right thing. Even though last time I checked, it didn't ... ? Lets see. Stab at #72
|
Indeed, by looking at the code linked by you, I'd think that a deadlock is possible here. However, it seems that this functionality is used just once in an unrelated command. When looking at the code-path taken by fetch/pull, it becomes apparent that it will enforce opening stdout, even though it never reads from it - this could be the deadlock you experience. The A few changes have been made, and you are welcome to test them with your setup. In case it works, please let me know. |
Task:
Clone from a remote repository having ssh://[email protected]/git/remote_repo format, which has submodules included from the outside of domain.com.
Problem:
current_repository = git.Repo.clone_from(r"ssh://[email protected]/git/remote_repo", r"d:\temp")
the remote_repo has 313 objects (files and folders). Above method call hangs after fetching 294 objects. Total size of remote_repo is 82 MB.
Observations:
Possible but incomplete solution:
there is some problem with stderr=PIPE parameter of subprocess.Popen call in execute method of Python27\Lib\site-packages\GitPython-0.3.2.RC1-py2.7.egg\git\cmd.py.
If I change the code to below:
The process does not hang.