[ProGet] Incorrect package publish date affecting policies

stevedennis

This behavior is expected, as the publish date is not carried over via a promotion.

In retrospect, it should have - but this is the sort of behavior we need to be careful about changing in a maintenance release. We'll discuss this internally and decide - please stay tuned, wehope to update by end of next week.

Thanks,
Steve

amy.j

Hi @stevedennis,

I understand, an update by the end of the week would be great - thank you!

amy.j

Hi,

Just wondering if there are any updates?

Thank you

stevedennis

Hi @amy.j,

Good news! We've decided to correct this, along with some other errant behavior with carrying over certain server-side metadata from connectors.

We are planning to ship this in the upcoming maintenance release (scheduled for August 7) via PG-3342 (FIX: Ensure correct server-side metadata (publish date, listed, deprecated) is set when adding package via pull, download, or promote from remote connectors).

This is currently in testing / code review now, asit's a bit of a riskier change. But assuming there's no issues with it, it will be available after the release.

Thanks,
Steve

amy.j

Thank you @stevedennis, really appreciated!

amy.j

Hi, it seems like PG-3342 is open and scheduled for the next release? Did something change whilst in testing?

Thanks

dean-houston

Hi @amy.j ,

Looks like the code was actually merged in and shipped, but that status wasn't reflected on the YouTrack issue, so the note didn't get added to the release.

Anyway I've modified the status and manually updated the release notes now.

-- Dean

amy.j

Hi @dean-houston,

Thanks for the update! After testing, the original issue appears to be resolved.

I did notice that aged package warnings are not appearing on the feeds I tested. Packages are showing as compliant, even where I would have expected an aged package warning. The package metadata is now more visible, which is helpful, but I was just wondering if you could confirm whether this functionality is working as expected, or if there is something I'm missing?

Thank you

atripp

Hi @amy.j ,

If the package is already cached in ProGet, then the publish date is already set. So, i would make sure to delete the package and try it again.

Otherwise, can you share more specifics? That way we can create a reproduction case.

Thanks,
Alana

amy.j

Apologies for opening this up again, would really appreciate any guidance on the below, thanks!

In our external Maven feed, org.springframework:spring-webflux 6.2.19 shows as non-compliant due to the recently published policy.
The module, sources.jar, and javadoc.jar show as published back in June (which is correct and therefore shouldn't be blocked). However, a publish date is not set for the pom file.

Once the package is actually pulled into ProGet, the pom file publish date is set to the promoted date and is therefore blocked in the external feed. Also noticed a similar issue with some NuGet packages, displaying the publish date as 01/01/1900 00:00:00. Not sure if there's a bug if the publish date metadata cannot be retrieved? For some packages, the pom file just doesn't show when you look at the external feed. If you promote, it promotes without the pom file. If you explicitly download the pom file, it then appears in the external feed and can be promoted.

atripp

Hi @amy.j ,

I'm afraid publish dates won't work reliably on Maven feeds.

A Maven repository does not contain server-side metadata like Download Count, Publish Date, Deprecated, etc., or even a search capability It's little more than web-based file system with maven-metadata.xml files that are generally maintained by the mvn client (e.g. it will always upload a new maven-metadata.xml file as part of a publish).

The Maven Index file (i.e. nexus-maven-repository-index.gz) attempted to solve the lack of metadata, but it's quite problematic these days (it's like 20-30GB at this point) and is generally disabled by default in most other products (e.g. Artifactory, Sona). On of the problems is that the file does not contain entries for all artifacts, including the .pom.

Publish date is part of the NuGet API, but old packages or some servers may not maintain. It's server-side metadata (like download count), and some basic repositories will just use default dates and 0.

Thanks,
Alana

amy.j

Thanks Alana for the clarification regarding Maven metadata and publish dates.

Our challenge is that feed policies are currently used as part of an automated promotion process. If a .pom file has no upstream publish date and ProGet falls back to the cache/promote date, older packages can be incorrectly treated as newly published and become non-compliant.

Given that publish date metadata is not reliably available for Maven repositories, is there a recommended workaround or best practice for Maven feeds?

What would Inedo recommend as an alternative approach for implementing newly published package controls where publish date information is unavailable or inconsistent? Would you recommend changing our approach for Maven packages altogether?

Thank you!

amy.j

Also there seems to be an issue with pulling in certain jar files, eg, when org.springframework:spring-webflux 6.2.19 is promoted no jar file is listed - so cannot be consumed. This has happened for a few packages. I'm not sure if this is a behaviour due to the policy that is set, however other required files are pulled through?

atripp

Hi @amy.j ,

Without a reliable publish date, there's no reasonable way to rely on an automated compliance rule that relies on the publish date.

I would recommend developing a Maven-specific approach, since this feed type is so limited. You've probably noticed it already, but license detection is also not very reliable. But, I wouldn't worry too much about it. Keep in mind:

the Java ecosystem seems to publish much less frequently than JavaScript
there are fewer transient dependencies in Java
version ranges tend to be less popular and much more conservative

Also there seems to be an issue with pulling in certain jar files, eg, when org.springframework:spring-webflux 6.2.19 is promoted no jar file is listed - so cannot be consumed. This has happened for a few packages.

This is probably related to that unreliable Maven Index file.

The Maven API does not support "file listing", so there's no automated way to know what artifacts are contained within a particular release. You just "kinda know" by reading the project's release notes, etc.

Early on, we just "assumed" there would always a .jar file, but I guess that's not the case because there are .war and .ear files too. We also considered parsing the web-based file listing page (e.g. spring-webflux/6.2.19, but a lot of repositories don't provide a listing.

We're open to ideas, but it's just an unfortunately limited ecosystem.

Thanks,
Alana

stevedennis

Hi @amy.j ,

Good news! Thanks to another user, we were able to figure out how to get a Publish Date for Maven artifacts. It seems to work on MavenCentral, and possibly other servers -- it realy depends on if they implement Last-Modified headers on HTTP requests.

This will be implemented in today's release via PG-3411

Thanks,
STeve