PHP Composer feed connect to vcs type repository or how to upload

dubrsl_1715

Hi,

I'm trying to migrate our php composer repositories to ProGet. We currently use satis for private repositories. And I'm having problems.

Composer can receive packages from VCS repositories https://getcomposer.org/doc/05-repositories.md#vcs. And this is convenient for private repositories. In ProGet I did not find such type of connector for composer. :(

This is due to the fact that ProGet receives metadata from composer.json, which should be in a zip archive https://docs.inedo.com/docs/proget/feeds/composer. And if all other metadata can be kept there, then specifying the version in composer.json and committing to git is not convenient. And usually no one does this. There are many branches, tags and composer works fine without a version in composer.json. Repackaging a zip archive, as suggested in the documentation, is also not possible, because its hash will differ from the one in git. And changing the hash leads to recreating composer.lock.
Specifying the version in the archive name is also not an option. The version format is much wider than the file name format. I have branches and tags that have / in their name.

Is it possible to upload metadata separately from the archive?
Or maybe you can suggest another solution?

Thank you!

stevedennis

Hi @dubrsl_1715 ,

Thanks for the clear explanation; I'm afraid your best bet here is to "take the plunge" and adopt package management best practices. A key tenet being a "self-contained archive file with a manifest" that is read-only and "cryptographically sealed" (i.e. hashed).

A "pointer to a Git commit" seems convenient at first, but there's a good reason that many ecosystems (including Composer) have moved away from it -- ultimately it's just not scalable and doesn't handle team workflows very effectively.

This will likely involve updating your composer.lock files and also the way you publish packages. In general, your process for creating a new version of a PHP packages should look something like:

Checkout code
Modify composer.json file with a version number
Package the file
Publish to ProGet

As you get more mature, you can get into pre-release versioning and repackaging.

Hope that helps,

Steve

ayatsenko_3635

@stevedennis, if I understand correctly, ProGet supports some package management conventions that are unfamiliar to PHP developers, while disregarding those that have been used within the PHP and Composer ecosystem for over 13 years. These practices have been adopted by nearly all major projects, including Symfony, Laravel, and even Composer itself. The current Composer-ProGet adapter, in its present form, is unusable in real-world scenarios. And to use ProGet with private PHP packages in a way similar to Private Packagist, we would have to completely change our current workflow to one that goes against established PHP ecosystem best practices and even repackage modified archives.

dubrsl_1715

@stevedennis said in PHP Composer feed connect to vcs type repository or how to upload:

A "pointer to a Git commit" seems convenient at first, but there's a good reason that many ecosystems (including Composer) have moved away from it

I haven't heard anything about this. Can you show me proof?

-- ultimately it's just not scalable and doesn't handle team workflows very effectively.

So you're saying that keeping everything in zip archives is better than keeping it in git? Did I understand you correctly? (The question is related to Composer)

I don't agree with the workflow you propose regarding composer. We won't be able to use ProGet without additional git integration. And if I additionally have to maintain and develop software that will provide seamless git integration, then the question arises: why should I pay for ProGet?
There may be other benefits of ProGet. I am continuing the test implementation.

stevedennis

Hi @ayatsenko_3635, @dubrsl_1715,

Thanks for the feedback and continued discussion.

So, our Composer feed is relatively new, so we are open to exploring new ideas. One option might be to do an "import" of packages into ProGet, similar to how we handle connectors in Terraform feeds. But that's something that could also be done with a script, so we'd want to see what that looks like as a prototype.

That said, we definitely can't implement "content pointers" to Git repositories. The Git-based "package" model is not only outdated but it has several "fatal" Software Supply Chain problems.

The biggest problem, by far, is that package content is hosted by a third party (i.e. GitHub) and managed by another third party (i.e. the repository owner). At any time, a package author or the host can simply delete the repository and cause a major downstream impact - like the infamous left-pad incident in npm.

This is why other package ecosystems have adopted a "read only" package repositories and disallow deletes (except for rare cases like abuse). Once you upload a package to npmjs, nuget, rubygems, etc. -- it's permanently there, and users can always rely on that being the case.

That's simply not possible with Git-based "packages". The "content pointer" must be updated periodically updated, such as if the author decides to move the Git repo to GitLab, Gitea, etc. Now you no longer have a read-only package repository, but one that must be editable. Who edits? Are they tracked? What edits are allowed? Etc.

There are several of other issues, especially with private/organizational usage, like the fact that there's no reasonable way to QA/test packages (i.e. compared to using a pre-release/repackaging workflow) and that committing/publishing are coupled (i.e. you tag a commit to publish it). This makes governance impractical.

And that's not to mention the fact that there's no real way to "cache" or "privatize" third-party remote Git repositories. Unless, of course, you mirror them into your own Git server... which is technically challenging, especially if the author changes hosts.

We first investigated feeds years ago, but they didn't have package files at the time -- only Git repository pointers. Like Rust/Cargo (which used to be Git-based, and still technically supports Git-based packages), we anticipate this same maturity coming to Packagist/Composer as well.

So while we certainly understand that this is a workflow shift, it's a natural evolution/maturation of package management. Likewise, it took decades for the development world to shift from "file shares" and legacy source control to "Git", but that's what everyone has standardized on.

That's the world ProGet operates in, so it might be a bit "too far ahead", or we might be misreading the tea leaves - but a "package mindset" is a requirement in using ProGet. But hoepfully we can make that easier, possibly with exploring "package imports" or something like that.

Cheers,
Steve