sysconfig data inconsistencies caused by the disabling the site initialization on virtual environments

[FFY00]

Bug report

Bug description:

Currently, running the Python interpreter with -S (disabling the site module initialization) causes several inconsistencies sysconfig.get_paths() depends on whether sysconfig was imported before or after the site initialization

Currently, sysconfig calculates the paths at import time and caches them. This results in it resulting incorrect paths if the site initialization happens after sysconfig was imported.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

Expected output when the site module has been initialized:

$ python
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

And just to make sure this is not dependent on the interpreter being run with -S, and rather on the sysconfig import time.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

Linked PRs

• GH-126789: fix some sysconfig data on late site initializations #126812

[FFY00]

Things are a bit messy, as the virtual environment mechanism is technically part of the site customization. In reality, the pyvenv.cfg detection is implemented on the interpreter initialization, but the customization itself is done by site. I feel like this should probably be reconsidered.

Anyway, to match this model, if site has been initialized, sysconfig.get_paths() should always return the virtual environment paths. So, the correct result should be as follows:

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

[FFY00]

Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

[FFY00]

While fixing the main bug, I have noticed that the prefix and exec_prefix variables get overwritten by _init_posix, so on virtual environments sysconfig.get_config_vars('prefix') and sys.prefix will have different values, the same goes for exec_prefix.

IMO this is worth to fix, but we probably shouldn't backport it as it is a significant behavior change, and there's a possibility it would break existing user code.

[FFY00]

Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

Just some more background on this, sysconfig.get_paths() uses an incomplete copy of the config vars directory to expand the path placeholder, which is what threw me off initially. This makes it a little bit more tricky to fix the issue, as we need to make sure we update the config vars cache without creating a recursive loop, but wasn't too difficult to get the implementation right.