PHP Composer feed connect to vcs type repository or how to upload

dubrsl_1715

Hi,

I'm trying to migrate our php composer repositories to ProGet. We currently use satis for private repositories. And I'm having problems.

Composer can receive packages from VCS repositories https://getcomposer.org/doc/05-repositories.md#vcs. And this is convenient for private repositories. In ProGet I did not find such type of connector for composer. :(

This is due to the fact that ProGet receives metadata from composer.json, which should be in a zip archive https://docs.inedo.com/docs/proget/feeds/composer. And if all other metadata can be kept there, then specifying the version in composer.json and committing to git is not convenient. And usually no one does this. There are many branches, tags and composer works fine without a version in composer.json. Repackaging a zip archive, as suggested in the documentation, is also not possible, because its hash will differ from the one in git. And changing the hash leads to recreating composer.lock.
Specifying the version in the archive name is also not an option. The version format is much wider than the file name format. I have branches and tags that have / in their name.

Is it possible to upload metadata separately from the archive?
Or maybe you can suggest another solution?

Thank you!

stevedennis

Hi @dubrsl_1715 ,

Thanks for the clear explanation; I'm afraid your best bet here is to "take the plunge" and adopt package management best practices. A key tenet being a "self-contained archive file with a manifest" that is read-only and "cryptographically sealed" (i.e. hashed).

A "pointer to a Git commit" seems convenient at first, but there's a good reason that many ecosystems (including Composer) have moved away from it -- ultimately it's just not scalable and doesn't handle team workflows very effectively.

This will likely involve updating your composer.lock files and also the way you publish packages. In general, your process for creating a new version of a PHP packages should look something like:

Checkout code
Modify composer.json file with a version number
Package the file
Publish to ProGet

As you get more mature, you can get into pre-release versioning and repackaging.

Hope that helps,

Steve

ayatsenko_3635

@stevedennis, if I understand correctly, ProGet supports some package management conventions that are unfamiliar to PHP developers, while disregarding those that have been used within the PHP and Composer ecosystem for over 13 years. These practices have been adopted by nearly all major projects, including Symfony, Laravel, and even Composer itself. The current Composer-ProGet adapter, in its present form, is unusable in real-world scenarios. And to use ProGet with private PHP packages in a way similar to Private Packagist, we would have to completely change our current workflow to one that goes against established PHP ecosystem best practices and even repackage modified archives.

dubrsl_1715

@stevedennis said in PHP Composer feed connect to vcs type repository or how to upload:

A "pointer to a Git commit" seems convenient at first, but there's a good reason that many ecosystems (including Composer) have moved away from it

I haven't heard anything about this. Can you show me proof?

-- ultimately it's just not scalable and doesn't handle team workflows very effectively.

So you're saying that keeping everything in zip archives is better than keeping it in git? Did I understand you correctly? (The question is related to Composer)

I don't agree with the workflow you propose regarding composer. We won't be able to use ProGet without additional git integration. And if I additionally have to maintain and develop software that will provide seamless git integration, then the question arises: why should I pay for ProGet?
There may be other benefits of ProGet. I am continuing the test implementation.

stevedennis

Hi @ayatsenko_3635, @dubrsl_1715,

Thanks for the feedback and continued discussion.

So, our Composer feed is relatively new, so we are open to exploring new ideas. One option might be to do an "import" of packages into ProGet, similar to how we handle connectors in Terraform feeds. But that's something that could also be done with a script, so we'd want to see what that looks like as a prototype.

That said, we definitely can't implement "content pointers" to Git repositories. The Git-based "package" model is not only outdated but it has several "fatal" Software Supply Chain problems.

The biggest problem, by far, is that package content is hosted by a third party (i.e. GitHub) and managed by another third party (i.e. the repository owner). At any time, a package author or the host can simply delete the repository and cause a major downstream impact - like the infamous left-pad incident in npm.

This is why other package ecosystems have adopted a "read only" package repositories and disallow deletes (except for rare cases like abuse). Once you upload a package to npmjs, nuget, rubygems, etc. -- it's permanently there, and users can always rely on that being the case.

That's simply not possible with Git-based "packages". The "content pointer" must be updated periodically updated, such as if the author decides to move the Git repo to GitLab, Gitea, etc. Now you no longer have a read-only package repository, but one that must be editable. Who edits? Are they tracked? What edits are allowed? Etc.

There are several of other issues, especially with private/organizational usage, like the fact that there's no reasonable way to QA/test packages (i.e. compared to using a pre-release/repackaging workflow) and that committing/publishing are coupled (i.e. you tag a commit to publish it). This makes governance impractical.

And that's not to mention the fact that there's no real way to "cache" or "privatize" third-party remote Git repositories. Unless, of course, you mirror them into your own Git server... which is technically challenging, especially if the author changes hosts.

We first investigated feeds years ago, but they didn't have package files at the time -- only Git repository pointers. Like Rust/Cargo (which used to be Git-based, and still technically supports Git-based packages), we anticipate this same maturity coming to Packagist/Composer as well.

So while we certainly understand that this is a workflow shift, it's a natural evolution/maturation of package management. Likewise, it took decades for the development world to shift from "file shares" and legacy source control to "Git", but that's what everyone has standardized on.

That's the world ProGet operates in, so it might be a bit "too far ahead", or we might be misreading the tea leaves - but a "package mindset" is a requirement in using ProGet. But hoepfully we can make that easier, possibly with exploring "package imports" or something like that.

Cheers,
Steve

dubrsl_1715

@stevedennis said in PHP Composer feed connect to vcs type repository or how to upload:

Hi.

Thank you for answering.

So, our Composer feed is relatively new, so we are open to exploring new ideas. One option might be to do an "import" of packages into ProGet, similar to how we handle connectors in Terraform feeds.

Where do you import from? We are interested in importing from git. Because git is the original source of the code. And after publishing a tag or branch, we need to create a version of the corresponding tag. Otherwise, we will need some kind of layer that will extract metadata and packages. Aka packagist or satis.

That said, we definitely can't implement "content pointers" to Git repositories. The Git-based "package" model is not only outdated but it has several "fatal" Software Supply Chain problems.

I have a feeling that I've been transported back 35 years, when people carried 5-inch floppy disks with files to each other.

The biggest problem, by far, is that package content is hosted by a third party (i.e. GitHub) and managed by another third party (i.e. the repository owner). At any time, a package author or the host can simply delete the repository and cause a major downstream impact - like the infamous left-pad incident in npm.

Why does this bother you? If someone wants to set up a connector for a repository, they have the right to do so. If the repository disappears, it's no problem to find a new fork. We live in a global world. Once information becomes public, it doesn't disappear.

This is why other package ecosystems have adopted a "read only" package repositories and disallow deletes (except for rare cases like abuse). Once you upload a package to npmjs, nuget, rubygems, etc. -- it's permanently there, and users can always rely on that being the case.

These are all problems with public repositories. I have a private repository.

That's simply not possible with Git-based "packages". The "content pointer" must be updated periodically updated, such as if the author decides to move the Git repo to GitLab, Gitea, etc. Now you no longer have a read-only package repository, but one that must be editable. Who edits? Are they tracked? What edits are allowed? Etc.

This is not a problem at all and is not related to the topic of the question.

There are several of other issues, especially with private/organizational usage, like the fact that there's no reasonable way to QA/test packages (i.e. compared to using a pre-release/repackaging workflow) and that committing/publishing are coupled (i.e. you tag a commit to publish it). This makes governance impractical.

I wonder how all the developers and QA work? This answer wasn't generated by AI? :)

And that's not to mention the fact that there's no real way to "cache" or "privatize" third-party remote Git repositories. Unless, of course, you mirror them into your own Git server... which is technically challenging, especially if the author changes hosts.

We first investigated feeds years ago, but they didn't have package files at the time -- only Git repository pointers. Like Rust/Cargo (which used to be Git-based, and still technically supports Git-based packages), we anticipate this same maturity coming to Packagist/Composer as well.

I hope this never happens.

So while we certainly understand that this is a workflow shift, it's a natural evolution/maturation of package management. Likewise, it took decades for the development world to shift from "file shares" and legacy source control to "Git", but that's what everyone has standardized on.

That's the world ProGet operates in, so it might be a bit "too far ahead", or we might be misreading the tea leaves - but a "package mindset" is a requirement in using ProGet. But hoepfully we can make that easier, possibly with exploring "package imports" or something like that.

I realized that we have slightly different views. And in the case of binary data (docker image for example, or rpm, deb) packages make sense. But in the case of code, going back to files is a step backwards. SCM (git) has a lot of advantages over zip. Composer can download as zip, which is a container for transportation. And the original code is always stored in SCM (git).
It was expected that ProGet would be able to replace packagist.org for private repositories and we would be able to use a single system for our composer packages, rpm, docker images, helm charts, terraform modules and statates. But with the composer, there is still a problem and it is a blocker to migrate to ProGet.

Thank you for trying to help.

--
WBR,
Viacheslav Dubrovskyi

apxltd

Hi @dubrsl_1715 ,

I don't think ProGet is the right solution for you guys. Supply Chain Security and OSS Governance are clearly not concerns, and you're simply not going to get any real value out of a product designed to help organizations solve those problems at the cost of convenience.

I think you will be much better served with the free/OSS repositories for your various repositories; we call that Level 2 in Package Maturity, and I can't imagine any of the "pain points" articulated in that article will resonate.

There's nothing wrong with that. As I've always said, sometimes a Source Control Shingle is the right tool for the job.

Best,
Alex

dubrsl_1715

@apxltd said in PHP Composer feed connect to vcs type repository or how to upload:

Hi @dubrsl_1715 ,

Hi Alex,
Thank you for answer.

I don't think ProGet is the right solution for you guys. Supply Chain Security and OSS Governance are clearly not concerns, and you're simply not going to get any real value out of a product designed to help organizations solve those problems at the cost of convenience.

It's strange to hear this. I was sure that you must be interested in creating a product that would solve real tasks. Here we are a client with a real requirement: we want to be able to have an analogue of a private packagist with all the advantages of ProGet. But I can't configure work with composer feed in any way.

The local feed is partially working. I can't upload part of existing zip packages. Because zip packages don't have a version in composer.json. And passing the version in the file name doesn't work because ProGet can't extract the version from the file name. Example: vendor-package-name-main.zip can't be uplouded. Or vendor-package-name-0.1.dev.zip.

Is there any way to solve this without repacking the zip? Passing the version via an additional argument would be fine.

The proxy feed works in a very primitive way. Added 1 Feed + 2 connectors to the packagist and private repository. Does not switch between connectors. After not finding a package in the private repository, it does not see packages from packagist. There are no priorities for connectors. It is not possible to make aggregate feeds.

I think you will be much better served with the free/OSS repositories for your various repositories; we call that Level 2 in Package Maturity, and I can't imagine any of the "pain points" articulated in that article will resonate.

We have an opensource product. And we actively use packagist.org
But we also have enterprise versions that require more control. Namely - control of packages that we depend on from packagist, control of private packages.

There's nothing wrong with that. As I've always said, sometimes a Source Control Shingle is the right tool for the job.

Git solved these problems a long time ago.

--
WBR,
Viacheslav Dubrovskyi

apxltd

Hi @dubrsl_1715

I was sure that you must be interested in creating a product that would solve real tasks.

Not really :)

As a products company, we solve problems that a sufficient number of users would be willing to pay for and that aligns with our overall strategic vision.

We also do User-Driven Development, which means we prioritize helping our existing users/customers solve problems instead of adapting the product for nonusers/evaluators like you guys.

We don't consider packagist a competitor nor are we currently marketing users to switich; our solution is similar but not an analog. It requires a change to your workflow.

The PHP/Composer market is already hyper-niche and I'm not convinced there's enough demand to develop a "versionless" package format (i.e. where a version number cannot be discovered in the manifest file or the package file name) format just for this particular use case (i.e. alternate workflow / simpler migration from packagist).

None of the other feed types have this requirement, and it'd be nontrivial to modify our model to support this unique requirement. It would have downstream impacts to features like replication, disk-importing, reindexing, etc.

Thanks,

Alex