Repository navigation
Python launcher on Windows does not detect active venv #83180
Description
| BPO | 38999 |
|---|---|
| Nosy | @pfmoore, @tjguk, @zware, @eryksun, @zooba, @RossBoylan |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-12-08.18:49:03.610>
labels = ['type-bug', '3.10', 'OS-windows']
title = 'Python launcher on Windows does not detect active venv'
updated_at = <Date 2021-03-27.04:15:08.040>
user = 'https://bugs.python.org/AlexandrosKarypidis'bugs.python.org fields:
activity = <Date 2021-03-27.04:15:08.040>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2019-12-08.18:49:03.610>
creator = 'Alexandros Karypidis'
dependencies = []
files = []
hgrepos = []
issue_num = 38999
keywords = []
message_count = 12.0
messages = ['358017', '358018', '358020', '358061', '358155', '358186', '358277', '358282', '358312', '358313', '358856', '389588']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'Alexandros Karypidis', 'rossb']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38999'
versions = ['Python 3.10']
Activity
AlexandrosKarypidis commentedon Dec 8, 2019
When you activate a venv on Windows and use a shebang with a major verion qualifier, the python launcer does not properly detect that a venv is active and uses the system installation instead.
The incorrect behavior is documented in this SO question where another user has confirmed and suggested it is a bug: https://stackoverflow.com/questions/59238326
Steps to reproduce (needs script.py attached below):
Install Python 3.7 on Windows 10 (64 bit)
Run script.py you should see:
PS C:\pytest> .\script.py
EXECUTABLE: C:\Program Files\Python37\python.exe
PREFIX: C:\Program Files\Python37
BASE PREFIX: C:\Program Files\Python37
Create and activate a virtual environment with:
PS C:\pytest> python -m venv .venv
PS C:\pytest> . .\.venv\Scripts\Activate.ps1
Run script.py you should see it ignore the active virtual environment:
(.venv) PS C:\pytest> .\script.py
EXECUTABLE: C:\Program Files\Python37\python.exe
PREFIX: C:\Program Files\Python37
BASE PREFIX: C:\Program Files\Python37
I am using Windows 10 64-bit, update 1903 and Python 3.7.5-64
AlexandrosKarypidis commentedon Dec 8, 2019
Forgot the simple script:
#!/usr/bin/env python3
AlexandrosKarypidis commentedon Dec 8, 2019
As confirmed by debug information when setting PYLAUNCH_DEBUG=1, the shebang seems to be ignored when using '#!//usr/bin/env python3' but works fine when using '#!//usr/bin/env python'.
This is with '#!//usr/bin/env python' (proper):
------------------------------------------------
(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 167 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python
searching PATH for python executable
Python on path: C:\pytest\.venv\Scripts\python.exe
...
This is with '#!//usr/bin/env python3' (wrong):
------------------------------------------------
(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 168 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
...
As you can see in the second case even though it regognises the python3 command, it makes no attempt to find it in the path.
Note that it appears that Windows installations only carry 'python.exe' and do not have a 'python3.exe' (neither in the native installation folder, nor in the virtual environment folder) so searching on the path would only 'work' if the user has copied python.exe to python3.exe in their <venv>\Scripts folder. (I actually tried that and it did not work).
A proper solution would probably need to search for 'python.exe' even for the '#!//usr/bin/env python3' shebang to detect if a virtual environment is present, possibly even confirming that the virtual environment is of the appropriate version.
pfmoore commentedon Dec 9, 2019
See PEP-486 (https://www.python.org/dev/peps/pep-0486/). This is intended behaviour, as it is assumed that explicitly specifying a Python version means that the user had a specific Python interpreter in mind.
There are also technical issues - if the shebang says #!/usr/bin/python2, and a virtual environment is active, how do we know if it's a Python 2 or 3 environment (without launching the interpreter, which is a significant extra cost for the launcher)? It would definitely be wrong to launch a Python 3 interpreter in that case.
AlexandrosKarypidis commentedon Dec 9, 2019
I think this situation is not ideal and the limitation seems artificial.
If I understand correctly, there are two parameters that govern the behaviour of the python launcher:
There are 4 combinations available:
a) env with version qualifier
b) env without version qualifier
c) non-env with version qualifier
d) non-env without version qualifier
Of the above, the python launcher supports (b) and (d).
For case (c) the launcher is configurable with py.ini
For case (a) it was decided to treat it as case (b) and search for "python.exe" in the path. To quote:
In the case of (1) I read:
"The launcher also looks for the specific shebang line #!/usr/bin/env python. On Unix, the env program searches for a command on $PATH and runs the command so located. Similarly, with this shebang line, the launcher will look for a copy of python.exe on the user's current %PATH% and will run that copy.
As activating a virtualenv means that it is added to PATH, no special handling is needed to run scripts with the active virtualenv - they just need to use the #!/usr/bin/env python shebang line, exactly as on Unix. (If there is no activated virtualenv, and no python.exe on PATH, the launcher will look for a default Python exactly as if the shebang line had said #!python)."
Why does the launcher search for "python.exe" specifically when using a version qualifier with env? This is very different to what happens on UNIX and causes the issues I reported. I see no reason why using #!/usr/bin/evn pythonX where X is a major version should not check the PATH for that specific version (e.g. #!/usr/bin/evn python3 should check for python3.exe in the PATH) and only fall back to plain "python.exe" if no match is found. This would be much closer to UNIX behaviour and allow for "common" shebangs across Windows/Linux/Darwin etc.
I would also note that if I create a python2 virtual environment and use a #!/usr/bin/env python3 shebang, a search on the path for python.exe would yield the version 2 executable from the virtual environment folder. So the argument about there being a technical issue is anyway true (the same wrong thing can happen in the current implementation).
Do you have any suggestion on how to work around this limitation using the current implementation of python launcher? Is there any way to use #!/usr/bin/env python3 across both Windows and non-Windows?
eryksun commentedon Dec 10, 2019
I don't think the "/usr/bin/env" case needs to limit qualified names like "python3[.x]" to registered installations. This is a choice made to simplify the implementation. If it finds a generic "python.exe" executable in PATH, for 3.5+ it is possible to query version information directly from it. Older versions lack this metadata, but it's also possible to inspect the PE import table (i.e. IMAGE_DIRECTORY_ENTRY_IMPORT) for a dependency on "python2x.dll" or "python3x.dll".
AlexandrosKarypidis commentedon Dec 11, 2019
My suggestion to gradually fall back from most specific to least specific version, was an attempt to reconcile the input from various viewpoints.
I think that ideally the behaviour should be as close to UNIX as possible. This means that it should just search the path for that binary (which is the whole point of using /usr/bin/env) and use it (even if it is the wrong version).
While I understand that one can do more (verify the version in whatever executable is located, etc) there is a lot in favor of consistency across platforms (versus even better behavior).
So my vote if the community decides to address this would be to just "search the path for <whatever>.exe when encountering #!/usr/bin/env <whatever>" and then launch it and hope for the best.
This makes it simple for users to intervene via the PATH (which is the whole point of /usr/bin/env) and launch whatever they want.
Currently, creatign a venv on windows only places "python.exe" in the Scripts folder (another difference to Linux). I would argue this is "correct" though because I think this was a decision of DISTROS (to use python for version 2 and python3 for version 3, while undertransition). Most distros still do that to this day and it will likely stick around. One day there will be only "python" and it will mean "version 3" but until then, it's better for the py launcher to do the same as on Linux and say "could not find python3 in the path" where the user can intervene (e.g. copy python.exe as python3.exe) and "fix" it manually.
Of course, doing "smarter" things on Windows is always an option but I would say that at least the "minimal" should be done to support consistency. (As I said, I would also suggest that no more than this minimal be done, but not in order to keep implementation simple, but to keep it consistent with other platforms).
Then again, that's merely one point of view from someone who is a user and does not even contribute to the project...
zooba commentedon Dec 11, 2019
That's one point of view, but the other is that the whole point of the shebang line is for users to be able to run the script without having to use a terminal at all. And once you're there, you no longer have a customisable PATH to work with.
"Simple for users to intervene via the PATH" is not true on Windows. Modifying PATH globally can break applications or (some) system components, and modifying it temporarily requires becoming a terminal user, which is not the majority.
Personally, I think supporting the shebang line in py.exe is a misfeature and would prefer we'd never done it (though it predates my involvement). But if someone wants to implement support for detecting a venv and matching it to the shebang, I won't actively block it going in.
eryksun commentedon Dec 12, 2019
Do you mean all shebangs, including those with a native file path?
Or do you mean just virtual shebangs, and in particular those that search PATH via "/usr/bin/env"? It's not without controversy even in Unix. For example, see [1] and the rebuttal to it in the comments.
In the case of this issue, "env" is used to run a script in an active virtual environment. This doesn't rely on the user or system PATH, assuming the command can be found in the active scripts directory.
As an alternative to modifying the launcher, pip (via distlib) could be updated to transform "#!/usr/bin/env pythonX[.Y]" shebangs into "#!/usr/bin/env python" when installing scripts in Windows. pip and distlib are third-party tools, however, so that suggestion is beyond the scope of this issue tracker.
[1] https://jmmv.dev/2016/09/env-considered-harmful.html
3 remaining items
Baljak commentedon Mar 10, 2023
This issue is not specific to virtual environments, because according to the Windows Launcher documentation, the first Python executable found in PATH should be used by the launcher instead of the default one:
And in fact, this is perfectly valid when putting the shebang line
#!/usr/bin/env python, but not with#!/usr/bin/env python3, while it clearly should work too.As a side note, I work on a cross-platform code that heavily relies on Python scripts for its execution and we bundle a specific version of Python with it. Because of this, we also tweak the
PATHenvironment variable to ensure that this version is used. Now, the current issue is quite a problem since our scripts are hosted on a Git repository with the#!/usr/bin/env python3shebang line, to make them work on Linux, but this requires any Windows user to patch these scripts manually to make them work on their OS, by replacing the shebang line with#!/usr/bin/env python. And this is not great, actually.zooba commentedon Mar 13, 2023
It's probably easier to copy/rename the
python.exetopython3.exethan to modify the scripts.What that section of documentation means is that it'll find an executable matching the name (
pythonorpython3in your examples) and treat it as a Python executable. If it doesn't find anything, it falls back to the well-known templates, wherepython3gets treated aspy -3which overrides theVIRTUAL_ENVenvironment variable.So that either makes this a request for
python3to match files namepython.exewhen searching PATH (and the*w.exeand*_d.exevariations we also may have installed), or a request forpy -3to preferVIRTUAL_ENVrather than its regular search. Either of these would be technically a breaking change, so we have to make the case for it, but it's possible that we could.Baljak commentedon Jun 2, 2023
Renaming
python.exetopython3.exedoes not work with the Launcher provided with Python 3.10.pythongh-83180: Made launcher treat shebang 'python' tags as low prioâĶ
Merge remote-tracking branch 'cpython/main' into pythongh-83180
Merge remote-tracking branch 'cpython/main' into pythongh-83180
Merge remote-tracking branch 'cpython/main' into pythongh-83180
gh-83180: Made launcher treat shebang 'python' tags as low priority sâĶ
pythongh-83180: Made launcher treat shebang 'python' tags as low prioâĶ