ProGet Connector Filters: not enough doc

jerome.jolidon_1453

Hello,

I'm trying to get the connector filters to work for an npm feed connected to npmjs.org. To try out the feature I've tried putting the following filters:
!@microsoft/*
!microsoft/*
!microsoft*
!@microsoft*

But in the end the packages from Microsoft can still be downloaded.

There's not enough documentation on that feature, I think. There should really be examples of what works and doesn't.

NanciCalo

Hello,

We're definitely keen on improving the documentation!

What version of ProGet are you using? In ProGet 5.3, it should involve...

navigate to the "Manage Feed" page.
select the "Connectors & Replication" tab.
select the desired connector name.
click "Add Filter," enter @microsoft/ and select "Block."

note this requires a paid Proget license.

Cheers,
Nanci

jerome.jolidon_1453

Hello Nanci,

Yep, I have two paid licenses, and I'm testing on two different instances, one running 5.3.4 (I won't blame you if you tell me I need to upgrade, I wish I could) and one running the 5.3.17 docker image.

The proposed @microsoft/ does not work, package @microsoft/tsdoc is still unfiltered.

The field mask mentions e.g. Microsoft.* or JQuery which made me thing that wildcards were allowed, all I tried until now had wildcards. I have the impression that it's not completely clear in any case, and the documentation is pretty scarce.

Sincerely,
Jérôme

NanciCalo

hey @jerome-jolidon_1453

Thanks for confirming about the versions. There was a change in ProGet 5.3.9 that might have fixed this:

PG-1789 - FIX: Connector filter rules not working on NuGet feeds

But if it's not working on 5.3.17, then it must not have helped. So from here, we'll try to reproduce this and let you know the results.

Please stay tuned!

Cheers,
Nanci

jerome.jolidon_1453

Hello,

Yes, I believe that for nugets, the filters work as of 5.3.17, but they don't seem to work for npm packages. I will also be looking into docker filters, so I might have more feedback soon.

Thank you for investigating this.

Cheers,
Jérôme

rhessinger

Hi @jerome-jolidon_1453,

Can you share what npm connector filters you are using? I can verify that @microsoft/* works on NPM filters. I tested using a block of * and allow @microsoft/*. That allowed only @microsoft packages to be downloaded. I will add that if an npm package has already been cached, the download is still allowed. These filters only work if the package is remote and has not been cached.

I will note that I have found a UI bug that will not show the download button blocked for NPM packages, but if you attempt to download the package you will get a 404 error.

Thanks,
Rich

jerome.jolidon_1453

Hello Rich,

Indeed, I did not go that far, I trusted the UI. The blocking patterns actually work for NPM, including on our old 5.3.4.

Maybe a few remarks regarding the behavior of the UI for blocked packages:

As you mentioned, the Blocked label doesn't appear
In addition, the version list is empty, but it could be by design
On the other hand, if the version list should appear empty, maybe it should not show the "Latest version" link on the search results page
The download link indeed goes to a 404 page, which is good, but perhaps the button should be disabled altogether
When logged as admin, the "Pull to ProGet" workflow is also enabled, but it seems it is also blocked down the road - still, it fails without feedback, that could be an improvement
I don't know if the "Delete" button should be enabled for packages that have not been pulled or added locally.
I think the Promote workflow should also be hidden, but that could be discussed - please note that I haven't tried it.

Again, thanks a lot for the investigation!

Sincerely,
Jérôme

rhessinger

Hi @jerome-jolidon_1453,

Thanks for all of the information, this is very helpful. I will make sure we can get these addressed. Can you tell me what your filtering level is on your connector?

Thanks,
Rich

jerome.jolidon_1453

I'm not sure to know what you mean by "filtering level". Currently I only block packages that match internal naming rules, another layer of protection against Dependency Confusion. I used Microsoft's for testing purposes to confirm that the way I defined the rules actually worked (the title of the thread does mention doc being a bit sparse ;-)).

Sincerely,
Jérôme

rhessinger

Hi @jerome-jolidon_1453 ,

If you navigate to the connector's overview page, you should see a link called configure filtering to the right of the Package Filters heading. That will show you your filtering level. I agree with your comments on our filtering documentation. I have also contacted our products team to do a rewrite of this documentation.

Thanks,
Rich

jerome.jolidon_1453

Ah, thanks, I didn't know it was called filtering level. I kept the recommended value, "Block download only".

rhessinger

Hi @jerome-jolidon_1453,

Thanks for the extra information, we will make sure to get the UI fixed to properly show that the package is blocked and we will also get the documentation updated for this as well.

Thanks,
Rich

jerome.jolidon_1453

Hello,

All good, thank you for the support, the investigation and the oncoming fixes.

Cheers,
Jérôme