Proget PGScan - How does this work with Dotnet SPA and NPM?

arozanski_1087

I've got a few questions about how the pgscan utility works with regards to NPM packages.

The code for the npm scanner indicates that it requires a package-lock.json file to be present in that directory. Is this intended to be used against the root solution, or a csproj? I have a project where this is used, but I consistently get a "pgscan : Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary." error regardless of where that file is.
If we're using Yarn, which produces a yarn.lock file instead, would I still be able to use this with the pgscan utility? Yarn uses the same packages as NPM, with different commands + a slightly different process for how it behaves.
If I have a dotnet SPA, would I be required to report my packages twice: once to a nuget feed and once to a npm feed in order to get my package data uploaded? I think the answer is yes, but mostly because I don't understand what the Identify command is for, or why i should use it instead of publish.

Dan_Woolf

I'll do my best to answer your questions:

When scanning an NPM project, the input parameter should be the path and name of the package.json (ex: \wwwroot\package.json). This will then parse the package-lock.json for dependencies in that same folder. If this is what you are already doing, then can you please send over the command you are using and the full error you are receiving?
We currently do not support yarn, but we are definitely interested in expanding our support to include it. Would you have any interest in working with us on implementing this feature?
Currently you will need to run the command twice; once for NuGet and once for NPM. The inspect command will work in an additive fashion and just append the new packages that were found into the ProGet project. This is something we are also looking to improve to add support to auto-scan both NuGet and NPM in one scan. As for the difference between inspect and publish
- inspect is the new command for ProGet 2022 and higher. This will add the dependencies to a Project in ProGet's SCA feature and is an overall better way to see your dependencies and it will link them to packages, vulnerabilities, and licenses in all your feeds.
- publish is the ProGet v6 and v7 command and it will only add the dependencies to a single feed at a time directly on the package itself. This command also requires the feed to be passed to record the results.
- You can also view the implementation differences furthur in our pgscan GitHub repository, https://github.com/Inedo/pgscan, if you would like to see more details.

I hope this answers all of your questions, but please let us know if you have more or need clarification on anything. Also, if you are interested in working to help us add improved support for the new features I mentioned, let us know and we can work together to get these features implemented.

Thanks,
Dan

arozanski_1087

Heyo @Dan_Woolf

Thanks for the clarification on the package.json parameter.
I'd be interested in helping work on it. What would be needed of me?
does Inspect require me to manually create a project for SCA or is there some manner of creating them programmatically?

Dan_Woolf

Hi @arozanski_1087,

No problem!
I will discuss this with the team further, but basically, we may ask some questions about how you use yarn and we'll send over some pre-release versions of pgscan to test with your setup. I'll have more details on what will be needed later this week once I talk with the products team.
The pgscan inspect command will handle creating the project and/or release if it doesn't exist. If the project and release already exist, it will also update them with any changes it has.

Thanks,
Dan

arozanski_1087

@Dan_Woolf

Testing:
that test plan should work perfectly fine for me. We have 2 main scenarios for using this on yarn:

there are frontend-only builds where it's a project running on node.js and yarn
there are dotnet builds with SPA .csproj in them that downloads yarn in the background. if it was npm those calls would be done the same there.

Inspect:
I tried that command on a project that I haven't created an SCA for. I received a peculiar error testing it out that I'm not sure how i'd go about troubleshooting. It didn't create the project, and when I made one myself it still errored out with this message.

I have version 1.4.1 installed as a dotnet tool. Below is what I'm calling:

pgscan identify `
            --input=$pwd\PROJECT.sln `
            --proget-url=https://myprogeturl.mydomain `
            --consumer-package-version=0.1.0 `
            --project-name=PROJECT `
            --api-key=myApiKey `
            --report

Dan_Woolf

Hi @arozanski_1087,

Thanks for the additional information. I'll need to research this a little further because nothing is jumping out at me looking through the code. Please stay tuned!

Thanks,
Dan

arozanski_1087

@Dan_Woolf Thank you!

if it helps, my version is 2022.18 (Build 7)

Dan_Woolf

Hi @arozanski_1087,

I'm sorry I missed this earlier, but it looks like there was a typo on the GitHub page for the identify documentation, it previously used --consumer-product-version. That parameter should have been --version. If you run the pgscan help identify command, you will see it is appropriately listed there. My colleague updated the documentation to include the proper parameter. That should get you passed the error above.

I also noticed another issue. If you run the command twice, the dependencies that were found on the second run will not be added to the project. I'm working on getting this fixed, but as a workaround, you can do the following:

Navigate to your Project & Release in ProGet
Navigate to the "Imported SBOMs" tab
Download the latest SBOM (please note that viewing the XML in the browser removes the XML namespace which will prevent future uploads)
Click the "Upload SBOM" button
Copy and paste the contents from your downloaded SBOM into the "SBOM file" field
Check the "Overwrite release" option.
Click Import

That will then add the missing packages to your release. To fix this, we will need to make a change to both pgscan and ProGet. I will do my bet to get this fix into tomorrow's release.

Thanks,
Dan

Dan_Woolf

Hi @arozanski_1087,

We were able to get the fix in for the appending dependencies, PG-2294. This will be released tomorrow in ProGet 2022.23 and we will also be releasing a new pgscan, v1.4.2, that includes a new parameter, --append-dependencies, that will allow you to append dependencies from multiple scans.

Thanks,
Dan

arozanski_1087

@Dan_Woolf

Thanks for the update!

for the --append-dependencies flag do i only use that on the 2nd attempt?

Dan_Woolf

Hi @arozanski_1087,

We talked it over in our team meeting yesterday and decided to change ProGet to always create a new release when one doesn't exist and add new dependencies when the release does exist. This way that parameter will not be needed. So you will just need to upgrade ProGet to v2022.23 when it is released later today.

As for the Yarn support and scanning multiple package types in the initial scan, we plan to work on that early next week, so we should have something for you soon.

Thanks,
Dan

rhessinger

Hi @arozanski_1087,

I just wanted to let you know that we just released a new version of pgscan, 1.4.2. Going forward it will automatically scan npm packages anytime it is scanning for NuGet packages using a .NET solution or project, there will not be any new parameter needed to enable the search.

We are still working on adding yarn support, but that one will take us a bit of time to add. We will update you when that one is ready to be used.

Thanks,
Rich