Bugs with Proget handling of simple repository APIs

itops_6398

Hello,

I believe that I've identified a number of bugs with the handling of Python package indexes, specifically simple repository APIs. The specification docs which I will be referring to in my post can be found here;
https://peps.python.org/pep-0503/

Your PyPI connector only supports repositories with a base URL of /simple - this is the default endpoint for simple repository APIs, but is not mandated as a requirement as per PEP503
Your PyPI connector does not support additional HTML present in the body of a repository. This issue is evident when attempting to setup a connector to https://bcms.bloomberg.com/pip/simple/ which has additional HTML to prettify the experience when navigating via a browser. Your HTML parser for the healthcheck will throw an error due to the presence of a <hr> tag in the body. Once again, PEP503 does not mandate that the body is clear of additional HTML, as long as the below are satisfied;

a. file must be valid HTML5
b. There must be one <a> per project
c. the text inside each <a> must be the project name
d. the href of each <a> must match the the project location

In both instances, I am able to successfully install packages from these repositories via pip without issue, but I am not able to setup a proget connector due to the aforementioned problems. The first issues looks to be due to the fact that your are implicitly appending /simple to any URL setup under a PyPI connector, the second issue could possibly be due to a custom HTML parser you've written which is not built to handle this scenario.

Thanks

gdivis

Hi @itops_6398,

Thank you for the bug report. I've logged this as PG-2507, and we will likely have it fixed for the next release of ProGet 2023.20, scheduled for October 13. If the fix or testing turns out to be more than expected, it may get deferred to the following release, but I don't expect that to be the case here.

itops_6398

Thanks for your response - great to hear!

Would it also be possible to add support for PyPI connectors to package indexes that are not based on /simple please?

An example of this would be the package index hosted on https://www.haver.com/Python - where the service is hosting a PEP503 compliant simple repository, but the endpoint is /Python

Thanks!

gdivis

Sure. I've logged that change as PG-2508. It should be easy enough to make that URL configurable in ProGet.