Usages of `urllib.request.urlopen` should be replaced by `requests`

[jiasli]

When verifying HTTPS connection, urlopen and requests by default use different CA bundles:

• urlopen uses the system CA bundle (https://docs.python.org/3/library/ssl.html#ssl.create_default_context)
• requests uses (https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification)
  1. REQUESTS_CA_BUNDLE
  2. CURL_CA_BUNDLE
  3. certifi.where()

Such inconsistency frequently causes issues. It is also difficult to handle such inconsistency correctly:

• [ARM] Fix #19420: az bicep install: Fix the CERTIFICATE_VERIFY_FAILED for generating SSL #20183
• [ARM] Fix #20842: az bicep: Fix to use requests environment variables for CA bundle #21807
• [Bicep] Accessing https://downloads.bicep.azure.com/ from Mariner 2.0 fails with ssl.SSLCertVerificationError #26007

According to the documentation of urllib.request:

https://docs.python.org/3/library/urllib.request.html

The Requests package is recommended for a higher-level HTTP client interface.

All Azure Python SDKs use requests to make HTTPS request, so does azure.cli.core.util.send_raw_request, so all usages of urllib.request.urlopen should be replaced by requests.

[yonzhan]

Thank you for opening this issue, we will look into it.

[jiasli]

acr:

azure-cli/src/azure-cli/azure/cli/command_modules/acr/helm.py

Line 383 in ba44e95

with urlopen(url, context=_ssl_context()) as response:

[jiasli]

acs:

azure-cli/src/azure-cli/azure/cli/command_modules/acs/custom.py

Line 1584 in b3ca1eb

version = urlopen(source_url + '/stable.txt', context=context).read()

[jiasli]

appservice:

azure-cli/src/azure-cli/azure/cli/command_modules/appservice/custom.py

Line 1430 in 17b9b55

response = urlopen(file_name, context=_ssl_context())

[jiasli]

backup:

azure-cli/src/azure-cli/azure/cli/command_modules/backup/custom.py

Line 1558 in 102599d

with urlopen(windows_script.url) as response, open(file_name, 'wb') as out_file:

[jiasli]

resource:

azure-cli/src/azure-cli/azure/cli/command_modules/resource/custom.py

Line 292 in 36786c5

req = urlopen(url, context=_ssl_context())

azure-cli/src/azure-cli/azure/cli/command_modules/resource/_bicep.py

Line 135 in 36786c5

request = urlopen(_get_bicep_download_url(system, release_tag, target_platform=target_platform))

[jiasli]

storage:

azure-cli/src/azure-cli/azure/cli/command_modules/storage/azcopy/util.py

Line 203 in 4b86ba6

req = urlopen(url)

[jiasli]

vm:

azure-cli/src/azure-cli/azure/cli/command_modules/vm/custom.py

Line 22 in 9bde773

from urllib.request import urlopen # noqa, pylint: disable=import-error,unused-import,ungrouped-imports

[jiasli]

core:

azure-cli/src/azure-cli-core/azure/cli/core/util.py

Line 1127 in 5757778

req = urlopen(url, context=_ssl_context())

[jiasli]

telemetry:

azure-cli/src/azure-cli-telemetry/azure/cli/telemetry/components/telemetry_client.py

Line 109 in 3b6dba2

http_client_t.urlopen(request, timeout=10)

[jiasli]

These usages of urlopen don't trigger ssl.SSLCertVerificationError on Mariner (#26007). This indicates these lines are not covered by recording test.