Sysconfig merge

[frenzymadness]

No description provided.

[frenzymadness]

Well, this seems to be almost complete and I'd like to merge it before I get to other parts of distutils we need to merge to sysconfig.

In points:

• A lot of functions or global variables are the same so an import from sysconfig to distutils.sysconfig is all we need.
• Sometimes, a function has a different name or slightly different API which is also solved without big issues.
• One function (_parse_makefile) has a small difference so I've implemented a compatibility switch.

My plan was to not touch tests at all to prove that the change of the deprecated module has no impact on its functionality but this plan failed. The main reason is that sysconfig uses _CONFIG_VARS global dictionary and distutils.sysconfig has _config_vars. The different names are not an issue here. The issue is that sysconfig._CONFIG_VARS is guarded by saved_test_environment but distutils.sysconfig._config_vars is not. If a test previously left some changes in distutils.sysconfig._config_vars, it had no effect on the result but now after the merge, it caused a lot of troubles. As a result, I had to implement or adjust setUp/tearDown methods for the tests which are (in)directly manipulating with the global dictionary. I am not always using the same names in those methods to make my modifications look similar to existing ones.

Another change in a test (test_venv) is related to a fact that pip uses distutils.sysconfig so we have to ignore deprecation warnings where we run ensurepip via subprocess. Later, I'll send a PR to pip to fix this.

What I think is left to be done:

• document somehow that distutils.sysconfig is deprecated
• add documentation for new functions in sysconfig module - copy/move it from distutils.sysconfig docs
• document new kw argument in _parse_makefile.

Cc: @vstinner @encukou @hroncok

[hroncok]

The description is awesome. Plese let the PEP discussion know that you are working on this (even if you are not yet ready to share the code).

[zooba]

Thanks so much for working on this!

[zooba]

Strictly, sys.platform == "win32" is the correct check for this case (sys.platform is "how it was built", while os.name is "which POSIX emulation are we using" - they just happen to line up right now ;) )

[frenzymadness]

The functions _fix_pcbuild is defined in sysconfig only if os.name == 'nt' so I can either keep the condition as is or change it in both modules. What do you think would be better?

[zooba]

Long term I'd rather it was changed in sysconfig. I don't mind what happens in distutils.sysconfig.

[frenzymadness]

os.name == "nt" is used in sysconfig in many places so I'd rather keep it as TODO.

[merwok]

It’s possible that re was imported in this function rather than at module scope because building CPython (and the _sre modules) needs sysconfig.

[frenzymadness]

Ok, I'll add try/except to handle situation when re module won't be available. Or do you have a better idea?

[zooba]

That may not work if the problem is circular imports. Better to just not use re until it's needed (and there's probably nothing gained from re.compile here anyway; the regular re methods will cache well enough).

[frenzymadness]

That would mean import re and define some of the regexes in three places:

• expand_makefile_vars function in sysconfig
• _parse_makefile function in sysconfig
• globally in distutils.sysconfig because they're imported from there in read_setup_file function in distutils.extension (not future proof)

I'd rather keep them in one place. I've checked and there seems to be no circular import. Do you think there might be a problem which the fresh builds in CI cannot spot?

[merwok]

FTR I found some tickets about differences between the two modules:

• https://bugs.python.org/issue39825
• https://bugs.python.org/issue24908

[frenzymadness]

FTR I found some tickets about differences between the two modules:

* https://bugs.python.org/issue39825

* https://bugs.python.org/issue24908

Thank you. Both seem to be better fixed in sysconfig and the first one already has a PR which does exactly that. I'll follow both of them.

[frenzymadness]

Doc status:

• distutils.sysconfig is marked as deprecated
• docs for get_python_inc and get_python_lib is copied to sysconfig docs
• there is still unresolved discussion about what to do with customize_compiler function so I'm keeping its doc where it is now.
• the function with changed API is private so there is no documentation for it.

[frenzymadness]

My plan is to get back to this now and open an upstream PR once I solve the problem on macOS.

[frenzymadness]

Upstream PR: python#23142