[ProGet] Incorrect package publish date affecting policies

amy.j

Hi,

In our setup, we have an 'external' feed and a 'curated' feed for each package ecosystem. Each 'external' feed has relevant connectors to the registries. When we would like to open up use of packages internally, the packages are promoted (from the 'external' feed) to our 'curated' feed.

We have noticed that, for our 'curated' feeds, the package publish date is the date the package has effectively been promoted (not the original package publish date). On our 'curated' feeds, we currently have the 'Aged package (3 years)' and 'Recently published (14 days)' policies enabled and set to 'warn'. Currently warnings are not showing up for aged packages and incorrectly showing on the other hand even though they are not recently published. This also causes issues with the Yarn 'npmMinimalAgeGate' check.

We have ensured, the below, 'Feeds.UseConnectorPublishDate' setting is enabled.

Just wondering if this behaviour is occurring due to our feed setup, external -> curated, and therefore the above setting doesn't work because the packages are just promoted to our 'curated' feeds?

Thanks!

stevedennis

Hi @amy.j ,

This behavior is expected, as the publish date is not carried over via a promotion.

In retrospect, it should have - but this is the sort of behavior we need to be careful about changing in a maintenance release. We'll discuss this internally and decide - please stay tuned, wehope to update by end of next week.

Thanks,
Steve

amy.j

Hi @stevedennis,

I understand, an update by the end of the week would be great - thank you!

amy.j

Hi,

Just wondering if there are any updates?

Thank you

stevedennis

Hi @amy.j,

Good news! We've decided to correct this, along with some other errant behavior with carrying over certain server-side metadata from connectors.

We are planning to ship this in the upcoming maintenance release (scheduled for August 7) via PG-3342 (FIX: Ensure correct server-side metadata (publish date, listed, deprecated) is set when adding package via pull, download, or promote from remote connectors).

This is currently in testing / code review now, asit's a bit of a riskier change. But assuming there's no issues with it, it will be available after the release.

Thanks,
Steve

amy.j

Thank you @stevedennis, really appreciated!

amy.j

Hi, it seems like PG-3342 is open and scheduled for the next release? Did something change whilst in testing?

Thanks

dean-houston

Hi @amy.j ,

Looks like the code was actually merged in and shipped, but that status wasn't reflected on the YouTrack issue, so the note didn't get added to the release.

Anyway I've modified the status and manually updated the release notes now.

-- Dean

amy.j

Hi @dean-houston,

Thanks for the update! After testing, the original issue appears to be resolved.

I did notice that aged package warnings are not appearing on the feeds I tested. Packages are showing as compliant, even where I would have expected an aged package warning. The package metadata is now more visible, which is helpful, but I was just wondering if you could confirm whether this functionality is working as expected, or if there is something I'm missing?

Thank you

atripp

Hi @amy.j ,

If the package is already cached in ProGet, then the publish date is already set. So, i would make sure to delete the package and try it again.

Otherwise, can you share more specifics? That way we can create a reproduction case.

Thanks,
Alana

amy.j

Apologies for opening this up again, would really appreciate any guidance on the below, thanks!

In our external Maven feed, org.springframework:spring-webflux 6.2.19 shows as non-compliant due to the recently published policy.
The module, sources.jar, and javadoc.jar show as published back in June (which is correct and therefore shouldn't be blocked). However, a publish date is not set for the pom file.

Once the package is actually pulled into ProGet, the pom file publish date is set to the promoted date and is therefore blocked in the external feed. Also noticed a similar issue with some NuGet packages, displaying the publish date as 01/01/1900 00:00:00. Not sure if there's a bug if the publish date metadata cannot be retrieved? For some packages, the pom file just doesn't show when you look at the external feed. If you promote, it promotes without the pom file. If you explicitly download the pom file, it then appears in the external feed and can be promoted.

atripp

Hi @amy.j ,

I'm afraid publish dates won't work reliably on Maven feeds.

A Maven repository does not contain server-side metadata like Download Count, Publish Date, Deprecated, etc., or even a search capability It's little more than web-based file system with maven-metadata.xml files that are generally maintained by the mvn client (e.g. it will always upload a new maven-metadata.xml file as part of a publish).

The Maven Index file (i.e. nexus-maven-repository-index.gz) attempted to solve the lack of metadata, but it's quite problematic these days (it's like 20-30GB at this point) and is generally disabled by default in most other products (e.g. Artifactory, Sona). On of the problems is that the file does not contain entries for all artifacts, including the .pom.

Publish date is part of the NuGet API, but old packages or some servers may not maintain. It's server-side metadata (like download count), and some basic repositories will just use default dates and 0.

Thanks,
Alana

amy.j

Thanks Alana for the clarification regarding Maven metadata and publish dates.

Our challenge is that feed policies are currently used as part of an automated promotion process. If a .pom file has no upstream publish date and ProGet falls back to the cache/promote date, older packages can be incorrectly treated as newly published and become non-compliant.

Given that publish date metadata is not reliably available for Maven repositories, is there a recommended workaround or best practice for Maven feeds?

What would Inedo recommend as an alternative approach for implementing newly published package controls where publish date information is unavailable or inconsistent? Would you recommend changing our approach for Maven packages altogether?

Thank you!

amy.j

Also there seems to be an issue with pulling in certain jar files, eg, when org.springframework:spring-webflux 6.2.19 is promoted no jar file is listed - so cannot be consumed. This has happened for a few packages. I'm not sure if this is a behaviour due to the policy that is set, however other required files are pulled through?