ProGet & new Python development environment

[stuart.houston_1512]

Hi,

Looking for some advice around using ProGet as a SCA Tool for Data Scientists and Analysts using Python.

At present there is a very legacy setup for Python in my company which I'm looking to update or replace to make it easier for DS's and DA's to utilise Python. Our current set up is an internally hosted Python repo (it's a mirror of conda I believe from a few years ago) but it has not been looked after properly over the last few years with reorgs etc. and users have an older version of Anaconda available to them.

What I'm looking to do is update the whole development environment and I'm trying to decide on the best approach to take for this using ProGet as an SCA Tool to monitor the Python libraries being used within the business. The first option is keep our internal repo and sit ProGet in front of it to proxy any new libraries or updates to the repo and roll out a fresh install of Anaconda to users. The second option is do away with our internal repo and just use ProGet as a proxy/cache instead with the fresh install of Anaconda.

Ideally I'm trying to reduce the IT admin/overhead of having a Python development environment whilst making it easier for DS's and DA's to use Python whilst maintaining control from an IT Security perspective over what libraires we're allowing into our corporate network.

Any advice would be much appreciated.

[apxltd]

Hi @stuart-houston_1512 ,

I would go with the second option, i.e. migrating packages from your internal repository to ProGet. It should be relatively easy to do this with a bulk file-system import.

The concern I would have with trying to configure ProGet up as a "proxy" to an old, internal conda repository you configured is that ProGet doesn't really operate as a "proxy" (i.e. blindly forwarding requests), but instead aggregates results from multiple sources using an API.

The Conda API isn't very well documented, doesn't provide much metadata about packages, and an old internal server will most certainly have bugs/quirks that ProGet would never be aware of. So your end users will end up with a buggy experience.

Connecting to the official Anaconda repositories is fine, and if there are any issues/bugs (like they change the API or something), we can easily reproduce and fix it.

Alex

[stuart.houston_1512]

Thanks @apxltd,

I'm assuming then the ideal scenario is effectively restart from scratch with a fresh Anaconda install and then configure ProGet to be connected to the Anaconda and PyPi repos with the new install pointed at ProGet for it's default channels so that any requests for new libraries or updates has to first be approved using the tool.

In terms of existing packages, anything that is still required we can then make use of the bulk file-system import.

Thanks

Stuart

[apxltd]

Hi @stuart-houston_1512 ,

If you're using almost entirely libraries that are available on PyPI.org and Anacaond.org, that makes the most sense. Users probably won't notice much of a difference.

Once libraries start being pulled through ProGet, you can start setting up policies/compliance rules to restrict packages. Or at least get warned about them.

At some point you can set up a package approval workflow, but I usually don't recommend that from "day one" - it's a bit too restrictive for end users, who are used to any package, any time.

If it's easy for you to identify first-party packages (maybe they are prefixed with MyCorp or something), then you can bring those in with bulk import. If no one uses (downloads) them after while, you can delete them with retention policies.

Cheers,
Alex

[stuart.houston_1512]

Thanks Alex,

With the Enterprise version of the tool where it includes SCA capabilities it mentions that the PGVC is bundled as an offline database. What I'd be keen to understand is how often we'd need to update this DB or if the tool works in real-time fashion to keep vulnerabilites up-to-date?

Thanks

Stuart

[dean-houston]

Hi @stuart-houston_1512,

ProGet will download nightly updates of the vulnerability database from an inedo.com server.

If you're running in a totally air-gapped environment, then you obviously won't get these updates. However, each version of ProGet includes an up-to-date database, so upgrading will get you the updates.

--Dean

[stuart.houston_1512]

Great thanks for confirming Dean