client/updater design: mirror config redesign

[jku]

This is a future wishlist item for client API (#1135), based on working on pip integration. I'm not sure what exactly should happen here, just documenting the issues

Warehouse may be an unusal setup but here's how it works (using server names of the pypi instance as examples):

• pypi.org serves TUF metadata, no other server serves metadata
• pypi.org also serves target files (content is html files like https://pypi.org/simple/django/, file location is undecided but potentially e.g. https://pypi.org/simple/django/index.html)
• files.pythonhosted.org also serves target files (in numerous directories like https://files.pythonhosted.org/packages/8f/1f/74aa91b56dea5847b62e11ce6737db82c6446561bddc20ca80fa5df025cc/Django-1.1.3.tar.gz#sha256=0e5034cf8046ba77c62e95a45d776d2c59998b26f181ceaf5cec516115e3f85a)

Issues encountered while implementing a client for this:

• There's no way in the client mirror config to say "this server has no metadata: don't request from here" (Updater: mirrors configuration tweaks #1079)
• confined_target_dirs seems useless: it is not possible to use e.g. "packages/" or "simple/" as a confining directory in the above target file example because only files directly under those directories would then be accepted

This could be fixed in TUF... but reality is that pip does not really need a mirror config: it always knows which server it wants to use for a particular download. With current API this means I will end up storing two mirror configurations and choosing the correct one every time before a download happens.

[jku]

This could be fixed in TUF

After further meditation: this particular issue cannot be fixed in TUF client. Even if confined_target_dirs allowed subdirectory matching, pip could not use them to separate index files and distribution files: The directory names "simple" and "packages" are just implementation details of pypi.org, not part of the Warehouse API. Some other instance might use the same directory name on both servers.

This is essentially a result of not namespacing the target names of index files and distribution files in the Warehouse TUF implementation: it is impossible to tell the two targets apart without some heuristics.

[jku]

Mirrors have been annihilated in new client:

• Metadata is only downloaded from one host
• For targets, caller can give Updater a default download url prefix, and can override it case-by-case when calling download_target