Posts made by atripp

Hi @p-boeren_9744 ,

I added support to have npm packages treat SEE LICENSE IN as a embedded file licenses via PG-2085.

It now looks like this, and blocks/allows package:

This will be released in this week's upcoming maintenance release.
Cheers,
Alana

Hi @dustin-davis_2758 ,

I'm not really sure - I'm not familiar enough with ADO Docker Compose to help :/

The error is coming because the container repository (image) name is incorrect; it should be like proget.initech.com/feedName/initech/repositoryName

Generally you put this in your docker-compose.yml file, like this:

https://docs.inedo.com/docs/docker-compose-installation-guide#example-docker-compose-configuration-file

So that would be the first place I would look. If you have the proper image name in there, tehn I guess, ADO might be doing something different?

Let us know what you find!

Cheers,
Alana

hello @nmorissette_3673 ,

I can't think of anything in ProGet that could yield this behavior (most especially for a particular package), and I can't reproduce it with the package. So this is tricky to debug.

Please try reproducing with a fresh new feed.

1. Create NuGet Feed (nuuget), add connector to NuGet.org
2. Navigate to /feeds/nuuget/Puma.Security.Rules/2.4.7
3. Try to download package file /nuget/nuuget/package/Puma.Security.Rules/2.4.7

IF that works, then there's some difference between the two feeds.

If it doesn't work, it's likely something between ProGet (which would be weird, but maybe a content filter/proxy).

Let us know what you find!

cheers,
Aana

Hi @nmorissette_3673 ,

That's odd, but I wonder if the package file is deleted from disk, and it's a cached package?

If that's the case, you should see a very specific message about it, like "Could not find a part of the path 'c:\LocalDev\ProGet\PackageStore.nugetv2\F1\Puma.Security.Rules\Puma.Security.Rules.2.4.7.0.nupkg'.".

Otherwise, here's what I did to reproduce:

1. Create NuGet Feed (nuuget), add connector to NuGet.org
2. Navigate to /feeds/nuuget/Puma.Security.Rules/2.4.7
3. Try to download package file /nuget/nuuget/package/Puma.Security.Rules/2.4.7

Of course, it's no problem. If i delete package on disk, then i'll get a 404 error.

If I "delete cached package" from the Web UI, and then download again it's fine.

Hope this helps...

Cheers,
Alana

Well, that's an interesting way to specify an embedded license file. I don't know if that's a convention or specification, but that seems to be a new way of handling it. It's kind of documented now, which is good: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#license

Anyways, we already handle this for NuGet packages using a URL convention like this:

• No license - packageid://Aspose.Words/21.9.0
• File license -package://Aspose.Words/21.9.0/License\Aspose_End-User-License-Agreement.txt

Those will be defaulted in the fields if the package specifies no license or a file license.

Not sure if it works now for npm packages, but it'd be relatively easy to adoption that convention, and then suggest it when the "license" field starts with "SEE LICENSE IN"...

Anyways we'll investigate this and update in a day or two.

Cheers,
Alana

@araxnid_6067 thanks, gload it worked! I'll work to update the documentation about this topic :)

Hi @cronventis ,

Great find - that seems to explain what we're seeing: containerd reports on containers differently than dockerd. So, we'll just search for container images based on configurationblob_digest OR image_digest

This change was trivial, and will be in the next maintenance release (or available as a prerelease upon request) as PG-2081 - scheduled release date is Feb 11.

Cheers,
Alana

Hello,

Are you doing an API call using PowerShell or something to delete packages? Did this happen after a recent upgrade to ProGet v6?

This post may help: https://forums.inedo.com/topic/3418/upgrading-from-5-to-6-causes-api-key-to-stop-working/2

We can definitely consider adding the API-key authentication back - we didn't realize it worked in the first place :)

Please let us know if this is the issue.

Cheers,
Alana

@araxnid_6067 is it in the [DockerBlobs] table? If not, then ProGet doesn't know about it, and then it's safe to delete

Otherwise, it might still be referenced by a manifest, but ProGet doesn't have that relation in the database.

You'd have to parse [ManifestJson_Bytes] to find out. If you're comfortable with SQL, you could do a "hack" query to convert that column a VARCHAR, then use OPENJSON or a LIKE query to search all manifests for that digest.

Howver, that's what ProGet does during feed cleanup.

@Stephen-Schaff I'm afraid I don't... it's a bit tricky to use, since you need to request a bearer token first and then send that in a header value.

https://docs.docker.com/registry/spec/auth/token/#how-to-authenticate

Thanks @mcascone , I also added this to our "Promotion / Repackaging Visibility & Permissions Rethinking" task - sounds like something we can consider :)

Hi @mcascone ,

I admit this can be confusing and is unintuitive because these were added separately over time, and the weren't originally designed for how they're used today. We need to rethink/redesign this based on the use cases.

I'm going to add this thread under the "promotion/repackaging workflows" topic for our next major version of ProGet. Once we know what we want to do, we can may be able to implement some changes as a preview feature in v6.

FYI, this is exactly how the big API Key changes and feed/package usage instructions came about!

https://forums.inedo.com/topic/3204/proget-feature-request-api-key-admin-per-user

So stay tuned :)

Hi @Stephen-Schaff ,

The API Keys changes in ProGet v6 involved changing some of the authentication code, so seeing bugs/regressions where a connected systems (build/CI server) reports authentication errors is not unexpected.

Based on the error message you're sending, it looks like you were using an X-ApiKey header to authenticate to the Docker registry API. That actually wasn't supposed to be supported before (Docker API requires token-based authentication), and must have only worked because of a bug / unclear specification in our old authentication code...

So the options from here:

1. Allow anonymous access to view the feed
2. Modify your script to use Docker's token-based authentication
3. Rollback to v5

We can consider adding/documenting support for using X-ApiKey header in the Docker API, but as it's not possible at the moment....

Hi @cronventis, just wanted to let you know that this is complicated and it's not something we cannot quickly debug/diagnose.

Based on our analysis, the data being returned from your Kubernetes API is different than our instance, and the instances we've seen in the field. Our instance's API is returning the configuration digest, but it looks like your instance is returning the manifest digest.

Which one is correct? Why is your instance doing that? Why is ours doing this? It's a mess

Code-wise, it would be a trivial fix in ProGet to make. Basically we just change this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.ContainerConfigBlob_Digest);

... to this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.Image_Digest);

... except that would break our instance and the others that return configuration digests.

We're tempted to "munge" the data results (basically just concatenate both database resultsets), but it would be really nice to know (1) which is correct and (2) why one instance does one thing.

Anyways that's our latest thought. Do you have any insight into this? This is just so bizarre.

Well, we'll kepe thinking about it on our end as we have time. Just wanted to give you a sitrep.

Cheers,
Alana

Hi @araxnid_6067 ,

This behavior is expected, and it's handled via Garbage Collection for Docker Registries:

Unlike packages, a Docker image is not self-contained: it is a reference to a manifest blob, which in turn references a number of layer blobs. These layer blobs may be referenced by other manifests in the registry, which means that you can't simply delete referenced layer blobs when deleting a manifest blob.
This is where garbage collection comes in; it's the process of removing blobs from the package store when they are no longer referenced by a manifest. ProGet performs garbage collection on Docker registries through the "FeedCleanUp" scheduled job.

So basically, it will get deleted when the corresponding FeedCleanUp job runs. It's default to every night, and you can see the logs on the Admin > Manage Feed page.

Cheers,
Alana

Hi @colin_0011 , that certainly is an odd issue!

We've never seen it before, but it's coming from the library we're using (libgit2). I don't really know what it means, or what's causing it (is it a number of files in the repository, etc.), but I just have a couple ideas.

1. Restart server
2. Clear the GitWorkspaces (C:\ProgramData\BuildMaster\Temp\Service\GitWorkspaces)
3. Try using git.exe instead of the built-in library

You can do #3 by setting GitExePath parameter on the operation, or configuring a $DefaultGitExePath variable at the server or system level in BuildMaster; this will force all Git source control operations to use the CLI instead of the built-in library

It's possible the bug was already fixed in a newer version of the lbirary. What version of BuildMaster are you using?

Hi @robert_3065 ,

Glad it's working!

Good point about the error message; it's in a kind of general place, so I just replaced that unhelpful base64 decoding message with this (via PG-2069):

string userPassString;
try
{
    userPassString = context.Request.ContentEncoding.GetString(Convert.FromBase64String(authHeader.Substring("Basic ".Length)));
}
catch (FormatException)
{
    throw new HttpException(400, "Invalid Basic credential (expected base64 of username:password)");
}

Not the perfect solution, but better than now!

Cheers,
Alana

Hi @robert_3065,

Based on this, I think the _auth token in your .npmrc file isn't correct. That is the token that's sent to http://OURSERVER/npm/npm_internal/, which is supposed to be in a base64-formatted api:apikey or 'user:password format.

Here's some more information about it:
https://docs.inedo.com/docs/proget-feeds-npm#npm-token-authentication

Npm auth isn't so intuitive unfortunately :/

Alana

@Stephen-Schaff great to hear! And I guess another way to do it would be enabling/disabling the semver restrictions on the feed

Let us know if it keeps happening, and you can find a pattern - we'll see if we can identify what might be the cause of it

@mcascone

existing connection forcibly closed

I'm afraid this is more of the same; there's some sort of network policy that's blocking this connection. It could be the way your laptop is configured, but maybe it's also happening on the HTTPS/SSL level? Anyways, the remote server (not proget.inedo.com, but some intermediate) is disconnecting at some point.

The ProGet-5.3.43.upack would really only be useful for manual installation; but it also might be bad/corrupt/incomplete. You could try unzipping it to see.

Oh.... I probably should have said it before, but we have premade, single-exe offline installers for specific versions of ProGet: https://my.inedo.com/downloads/installers

Here is some more information about them; https://docs.inedo.com/docs/desktophub-offline

These 403 errors are all coming from your proxy server (firewall); unfortunately we/you have no visibility on that.

But it's clear that some requests are allowed, and others aren't. Maybe it doesn't like requests that are downloading a file called .upack. maybe it doesn't like the agent header. Maybe it tries to scan/verify contents with a virus check? It's a total guess

From here, best bet is to check with IT to see if they can inspect the firewall/proxy logs.

@Stephen-Schaff said "Any ideas on how I can get the latest tag to be auto applied?"

the "virtual" tags are recomputed when a tag is added, so if you can try tagging your image 1.0.2 (or something), and then deleting that tag, you should see1, 1.0, and latest all applied to that image

Hi @mcascone,

Our products our built with .NET 4.5.2, which uses the Windows certificate chain.

I suspect that ZScaler is replacing the certificate, and that's causing a trust problem. Maybe you can try installing ZScaler certificate directly in the store, and there are some registry tweaks / hacks that might make it work. Unfortunately I don't have any specifics on what you can try.

You should see same errors if you log-in as service account user, and try to visit the site in IE or Edge. PowerShell would also exhibit the same errors.

In any case, I would search for like "ZSCaler Certificate TLS error Windws" and what not, and hopefully find some specific things to try...

Best,
Alana

Hi @marc-ledent_9164 ,

We're currently investigating this.

FYI: Based on the URL, I think that there is not a valid license... which would make sense if you just set up the instance. No license will trigger an automatic redirect to /administration/licensing to address the license. However, that page is marked is "no license required"... so it shouldn't be redirecting.

Please stay tuned.

@hwittenborn thanks for letting me know -- that was going to be my next suggestion. It's very possible the key had an unexpected combination of properties/permissions and that didn't translate to the v6 internal model

But easy enough to recreate in a case like that :)

@hwittenborn this can sometimes be tricky to get working...

Did this work in ProGet v5? We made a major change to the api keys in v6, so knowing if it's a regression would help track this down

@shaun-d-scott_2657 Inedo products do not use log4j, so it's not an issue for ProGet :)

See more information here: https://blog.inedo.com/log4shell-high-severity-vulnerabilities

@darkbasics_6739 thanks, of course that's not the problem then

We're still investigating this, and it's quite odd why the extension isn't getting transfered. The workaround you're doing is fine to at least evaluate/test, but we'll most certainly get this fixed ASAP

@v-makkenze_6348 thanks for the additional information

ProGet does not cache npm package lists/indexes. It's always generated from the information in the database, so if you pushed it - then it's in the database. You should be able to see it in the ProGet UI right after publishing.

It's very possible that the npm client is doing some sort of caching as well, but isn't doing that cache with authenticated requests. I'm not familiar enough, but that's my guess.

You could use a tool like Fiddler to verify / see what requests npm is actually making.

Hi @mike_4027 ,

I would definitely recommend upgrading; 5.1 is a couple major versions behind.

The issue is most certainly related to the connector; when you disable it, do you get a near-immediate 404? Under Admin > Diagnostic Center, you may see connector errors/warnings as well.

Once you confirm it's the connector, then the next best thing to do would be to monitor the traffic between ProGet and the internet. This involves setting up a Proxy (Fiddler works nicely), and then having ProGet connect to that proxy (Admin > Proxy). You should be able to identify corresponding requests, and maybe we'll see something there!

Let us know what you find,

Alana

Hi @darkbasics_6739 ,

That's really weird, but I can see how that error might happen now. It's been addressed as OT-446, and will be fixed in the next maintenance release.

We're still not sure why the SCripting extension isn't coming over. We're thinking, it might be due to a time out of sorts, since the file is now pretty large. Obviously that should throw a different error.... is there a lot of bandwidth between servers?

Cheers,
Alana

@v-makkenze_6348 it looks like you've enabled caching, which means that ProGet's responses won't be newly generated. Please disable this :)

@sbindra_9387 you can enter the activation code in ProGet, when you click the activate button

Here is the instruction for manual activation:
https://docs.inedo.com/docs/myinedo-activating-a-license-key

Hi @patrick-groess_2616 ,

The /symbols/<feed-name> URL is only for Visual Studio's Symbol Location setting, and is used to download symbols. Do not try to use it with nuget.exe, it will give that error.

You need to use /nuget/<feed-name> to push symbol or nuget.exe packages.

Thanks,
Alana

@entro_4370 we are definitely seeing "package" interest from "data science" teams inside of large organizations, but it seems they're more Python users instead of R users ‍️

@jblaine_9526 sounds good, I've forwarded this to our presales / customer success team internally - so they'll probably reach-out in the coming days about that process to see how they can help

Hi @entro_4370,

I'm afraid not... seems this thread has been a little quiet.

We haven't had many enterprise customers or sales leads requesting it as a feature, and we didn't see a good way to market the feature. It seems very few people search for CRAN-related topics or wanting private repository

Best,
Alana

@lukel_3991 thanks.

So this works totally fine if you're using localhost instead of computername.domain.com then And then, computername.domain.com is a Windows server?

If that's the case, it means the agent (which is connected via PowerShell Remoting) isn't fully loading; basically, the Scripting extension isn't loading, or had an error.

Can you try using a different Temp path? Maybe there's a relation....

Thanks,
Alana

Hi @falk-winkler_2111,

ProGet free edition only supports self-connectors when they connect directly to ProGet. If you bypass the proxy, that should resolve the license violations for you.

To do this, just use http://localhost or http://<PROGET_IP_ADDRESS> in your connector without issue. Please give that a try and let me know if you run into any issues.

Cheers,
Alana

@jblaine_9526 good question; it's not exactly spec'd out, but that would make sense. If SAML is enabled (as one of the directories), then users should be able to bypass SAML behavior. Maybe two Login Buttons, or something.

Since SAML is a ProGet Enterprise feature, it's very possible we could implement this specific requirement much sooner as part of a different program (i.e. "customer success" vs "product roadmapping"), but neither is really my department

I can get you in touch with my colleague who's on that team, but he's not on the forums -- let me know if it's okay to share your email to him (I can see it from my end)

Hi @jblaine_9526 , there are!

I don't have a schedule just yet, but we are planning to simplify the "hybrid" user directory by just making it so that you can set an "active/inactive" flag on each user directory, and then ProGet will run through the active ones in a prioirty order.

Cheers,
Alana

Hi @mcascone,

It looks like this isn't being displayed in ProGet's UI, so we'll fix this via PG-2038

cheers,
Alana

Hello,

This is unfortunately due to bugs in the PowerShellGet & PowerShell Gallery API that aren't feasible to workaround. Basically, the API is reporting different versions than the package file, which makes working with local/cached packages and remote packages painful like this. However, it seems like it's going to be addressed by Microsoft, finally, in PowerShellget v3.

In any case, given your architecture (i.e. "Machines on our network cannot connect to PSGallery direct due to the firewall and are only allowed to retrieve via this one feed."), we recommend using a Package Approval Workflow instead.

• create an 'unapproved-powershell' and approved-powershell feed
• create a connector on 'unapproved-powershell' to PSGallery.org
• Promote packages from unapproved-powershell to approved-powershell

This will totally eliminate the "bad API" problem as well.

Cheers,
Alana

Generally speaking, if yo ucan try to identify the pattern -- like which pages are slow -- that will help.

Some different pages to investigate:

• / is somewhat resource intensive, and involves database calls
• /health uses minimal system resources
• /resources/images/layout/logo.svg is a static image, is unauthenticated, and is cached by the browser

Hi @dac_4595 ,

If ProGet is hosted in IIS, then one thing that might be happening is that Application Pool shuts down if it's not accessed in a set amount of time. That 5-10 seconds might be the initial start-up time. You can control the timeouts in IIS, and even set it to be AlwaysOn.

Otherwise, there's nothing in the browser's cache that would cause this. If anything, clearing the cache would cause things to go slower, as pages need to re-load.

Cheers,
Alana

@forcenet_4316 great, sounds good! Actually we didn't end up shipping 6.0.0 yet (we will soon), so I just changed PG-2011 to target 5.3.40, which is scheduled for release on Friday. This way it will be in both.

Thanks @drew-stinnett_4680 !

I've updated the documentation with a link to this discussion.

@forcenet_4316 thanks to the data you sent, I was finally able to figure this out!

The problem is with antlr4-runtime-4.5.2-1-sources.jar. When ProGet replicates or promotes this package (logic is the same), the artifact's fileType is assumed to be .jar instead of -sources.jar; this is what's causing the PK constraint. This seems to only impact a very small number of artifacts, since I think I mostly saw .sources.jar in other cases (which would work).

I've fixed this via PG-2011, which is scheduled for 6.0.1 (shipping later this week).

Are you planning to upgrade to v6 soon? If not, I should be able to backport the fix to v5.3.

It seems that PowerShell client doesn't provide much details about errors, so glad you could figure out what the issue was for powershellgallery.com. That message probably could mean a lot of things, and is probably unrelated to TLS/SSL..

So the first thing that's jumping out to me is this:

WARNING: Unable to resolve package source 'http://proget.domain.org/nuget/ps'

There is no https connection, so there'd be no TLS/SSL issues. But assuming you may have changed the url when posting this... here is a registry key you can use to force that setting (it makes that powershell command unnecessary)

https://inedo.com/support/kb/1161/tls-v12-configuration-and-connection-errors

If you're on HTTPS the other thing to look at would be certificates.

Otherwise, you may need to use something like Fiddler to see the actual request/responses that PowerShell client is issuing/reciveing.

hi @paul_6112 - just curiosu if this continues to happen? We had another report about this, but are still struggling to find out why / and a resolution. We're going to try to take another stab at figuring this out again.