Support for R and CRAN

UPDATE: CRAN (R) feeds are here:
https://blog.inedo.com/inedo/introducing-cran-feeds-in-proget/

--

Hi Indedo support team!

I'd like to know if there are any plans of supporting R packages and in particular also an upstream connector to the CRAN project / package index.

See https://cran.r-project.org/web/packages/index.html.

We're behind a corporate firewall with no direct internet access from our clients. Some of our data scientists use R and are looking for a product to "proxy" the CRAN repository.

Please let us know if there are plans to support that and if there is a timeline / roadmap for that.

Thanks, Manuel!

Product: ProGet
Version: 5.1.15

This is the first request for such a feed type, but ideally we'd love to get support for all of the package types :)

We like to partner with our customers to help develop new feed types, because we don't use these programming languages (let alone these packages).

Do you know anything about the internal of CRAN or R feed protocols? Typically these are undocumented and require reverse engineering. Do you know if package formats are documented? Are there other private repositories available? How does one publish packages?

Basically if we could get some sort of insight into what the behind-the-scenes stuff looks like we'll be in better shape to at least guesss at how difficult this might be to do.

I'd second this request - it would be very useful for us too. And happy to engage directly with someone on the technical detail if it helps you reach an opinion more quickly.

Drop me a note if you'd like to do so.

Thanks

apxltd

hi all, thanks for the interest/comments; I decided to write-up a page that details this on the docs.

http://inedo.com/support/documentation/proget/feeds/other-types

I'm hoping we can use this public thread to maintain the discussion on technical detail; otherwise it'll get stuck in my email, or somewhere else, and we can get everyone to chime in this way.

That said, @M-W if you've got any insight into how R/CRAN works please do share :)

valeon

Hi

I have a bit of experience with R, and I like ProGet, so I thought I would add my 5p to this thread.

CRAN mainly hosts the package sources. The actual repository is this folder:
https://cran.rstudio.com/src/contrib/

This is just what it looks like, a list of tar.gz files which are essentially package sources. The only "special" files are the PACKAGES* ones. The one without extension is a yaml file with some metadata of all the packages in the folder. The .gz one is the same thing compressed, the .rds one is the same thing in a special R data format. I believe you can get away with just the PACKAGES and the PACKAGES.gz files.

Every package (tar.gz) will have a DESCRIPTION file in its root, which is another yaml file with all the metadata for the package. To build the PACKAGES file, the fields of the example below are copied:

Package: A3
Version: 1.0.0
Depends: R (>= 2.15.0), xtable, pbapply
Suggests: randomForest, e1071
License: GPL (>= 2)
NeedsCompilation: no

and they are pasted all together separated by empty lines.

For windows clients, CRAN also generates binary versions of the packages for the different versions of R. These are listed here:
https://cran.rstudio.com/bin/windows/contrib/

Each one of the numeric folders, look very much like the sources folder with the same PACKAGES files. The only difference is that in the R subfolder of the zip packages, instead of the source of the R functions there is now a compiled binary. The DESCRIPTION file is still there though with all the metadata.

The folder structure is important, since essentially, in the R client the command will be something like:

install.packages("lubridate", repos = "http://my-private-host")

This command knows to look exactly in the right folders depending on the environment of the caller (operating system, version of R, etc). For windows, it will work even without the binaries at all, but it would make a big difference in terms of speed.

As far as hosting in-house packages, I wouldn't expect from ProGet to compile them and provide binaries, leaving them as source only should be fine. For the packages coming from CRAN though, it would be good to relay the windows binary versions as well as the source one.

DISCLAIMER: This is part experience, part guesswork so any of those might not work. I have only used CRAN as a package consumer.

apxltd

@valeon fantastic, thanks so much! This will really help us explore; it doesn't look "too bad", and is "somewhat similar" to how Debian manages it's packages.

We'll try to start hacking around with a POC in the coming weeks, hopefully; i'll update when I can!

miles.waller_2091

Hi,

Was just wondering if the PoC made any progress at all?

Thanks!

atripp

Hello; unfortunately we hit a few snags in getting the environment and PoC code running (a bit more variety in R/CRAN packages), and then we ran out of budgeted time :(

But it's still definitely on our roadmap and we're going to take another stab at it in NOvember.

olivier

Hi,

I'm just another user registering my interest in this thread. This would definitively be a big plus for our organization as well :)

Hope this will be added on the roadmap (and best of luck for your ongoing PoC) !

Cheers,

apxltd

Hi all, a quick update!

We've just started a new internal project ( @gdivis will be leading the efforts), and you can expect to hear something in the next week or so with a rough plan / timeline.

Thanks for all your continued interest.

gdivis

Thanks everyone for your interest in this. As @apxltd said, I'll be leading the effort on this. As the package format and repository system both seem relatively simple, I don't foresee too many technical challenges with adding this to ProGet, but there are a few things I'd like to make sure I understand right now:

ProGet typically needs to understand versioning rules for packages (to show which is the latest, to do retention policies, etc). From what I've gathered, CRAN uses 2 or more integers separated by dots. Does anyone know if that's required universally, or if it's just a convention?
CRAN doesn't appear to have a server-side repository API, instead just exposing a list of packages using HTML. Is my understanding correct here?
Submitting a package to CRAN looks like it's only done manually using a form here. Is that right, or is there some other automated mechanism that is used?

Also- we don't have any in-house R experts, so any (tentative) commitments to try out a prerelease version of ProGet with CRAN support would be very helpful in getting this feature out sooner :)

Thanks!

It-purchasing_9924

Hello @gdivis, this is a feature some engineers at my company would benefit from greatly. It would be super nice to have our own 'official place' to install R packages. As soon as you have a prerelease version we'd gladly be willing to test.

Thanks!

atripp

Thanks, noted :)

There was more demand for RPM/Yum packages, so we recently added those. Now, we are focusing on ProGet 5.3, so perhaps after that we can reconsider this --- if we get more community interest that will go a long way.... so if anyone elee is reading these and interested, let us know.

olivier

Would you have any feedback about your latest POC ?

Thanks :)

apxltd

@olivier we're mostly still looking for insight/answers into the questions @gdivis had asked, so if you can help on this we could at least move the ball forward

@gdivis said in Support for R and CRAN:

ProGet typically needs to understand versioning rules for packages (to show which is the latest, to do retention policies, etc). From what I've gathered, CRAN uses 2 or more integers separated by dots. Does anyone know if that's required universally, or if it's just a convention?

CRAN doesn't appear to have a server-side repository API, instead just exposing a list of packages using HTML. Is my understanding correct here?

Submitting a package to CRAN looks like it's only done manually using a form here. Is that right, or is there some other automated mechanism that is used?

miles.waller_2091

Hi

1 - Here's some info on the version numbers: http://r-pkgs.had.co.nz/description.html#version. 2 integers separated by dots or dashes is the minimum length, but can be up to 4. There's also the

Yes, it doesn't have a queryable API, just uses static assets. It doesn't need to build any html though (thats just for humans): it builds out a static package list by concatenating the individual files within each package, and that could be done dynamically.
That's right. There is devtools::release() but it looks like that only works for cran, so I think there's no consensus about a generic upload method. OTOH, once built its just a zip file so crafting the R to send it off over http is not hard.

atripp

Thanks @miles-waller_2091 ; we're heads-down in 5.3 now, so after that we'll be able to resume investigating this. Perhaps May/June? In the meantime, if more people can volunteer to help test, we'll be able to get this going rather quickly :)

entro_4370

Hi! Any news on R-support in ProGet? There seems to be quite a demand! As a Data Engineer in a large government that bases its package management on ProGet i feel that there is a clear gap here...

atripp

@entro_4370 said in Support for R and CRAN:

Hi! Any news on R-support in ProGet? There seems to be quite a demand! As a Data Engineer in a large government that bases its package management on ProGet i feel that there is a clear gap here...

I'm afraid it's still not on our roadmap, based on our research. Here's how we evaluate this sort of things.

At first, what's the market demand outside of existing users? We didn't find much opportunity to attract data engineers (or their bosses, who would buy software) who aren't already using ProGet (or a competing product) to use a private repository instead of CRAN directly. It's just not big discussion of R/CRAN community, unlike other package types.

And then, we surveyed/asked users. But we phrase it like this: how much more would you pay for ProGet it if it had this feature, because this is an honest assessment of "how much more value would this feature bring to you" (which is what we want to decide).

Unfortunately everyone we asked said, "it's a nice to have, but we would actually pay X% more if ProGet had feature X instead."

Anyways still something we want to do, but we want to make features that bring most value to most people at first...

of course if you have insights on this please let us know! Cheers.

entro_4370

@atripp Any news on CRAN-support? I can see that it is displayed as a coming feature under feeds/Developer libraries?!

atripp

Hi @entro_4370,

I'm afraid not... seems this thread has been a little quiet.

We haven't had many enterprise customers or sales leads requesting it as a feature, and we didn't see a good way to market the feature. It seems very few people search for CRAN-related topics or wanting private repository

Best,
Alana

entro_4370

@atripp Maybe it's because resources such as https://packagemanager.rstudio.com/client/#/repos/2/packages exist then.. Or could it be that almost no one runs [R] in on-prem production? Difficult knowing exact reasons..

atripp

@entro_4370 we are definitely seeing "package" interest from "data science" teams inside of large organizations, but it seems they're more Python users instead of R users ‍️

mrbill

Hi!
Status Quo here @atripp?
I'm working in a governmental authority where R is frequently used and we use ProGet for package repos. For us, a CRAN feed-type in ProGet would be VERY welcome. Currenlty we have no way to handle R-packages internally in a good way.
If R/CRAN support is still somewhere on your backlog we would be glad to help you guys with whatever we can =)

apxltd

Hey @mrbill and all,

We've since had a handful of "mentions" for it on the presales side, but it was more like a "wishlist" than anything else. There's probably some opportunity however - so I'm leaning towards re-evaluating for v2023.

Big thing will be partners who can show us how to use R -- I added this to our internal list for the v2023 review :)

Cheers,

Alex

mrbill

@apxltd Thanks for the update!

Ok we keep our fingers crossed that you can look into this during 2023.

Please let us know if we can be of any help!

Regards
Magnus

apxltd

@mrbill @entro_4370 CRAN (R) feeds have arrived

https://blog.inedo.com/inedo/introducing-cran-feeds-in-proget/

apxltd

@valeon @miles-waller_2091 @olivier @It-purchasing_9924 @entro_4370

Took a bit, but CRAN (R) feeds have arrived

https://blog.inedo.com/inedo/introducing-cran-feeds-in-proget/