Posts made by atripp

Hi @philipp-cender_3322 ,

The "inbound connection" is complex and a relatively new feature, and I don't have a ton of experiencing troubleshooting - so I'll do my best :)

So far, everything looks okay to me.

On the Otter Server, are you seeing any errors related to the server under Admin > Diagnostic Center? I see the server is in an "Error" state.

On the remote server, does the service stay running? If so, that's indicating it's able to establish a connection. But one thing you can try is to stop the service, and run in interactive mode (i.e. run InedoAgentService.exe run on the commandline). That will show you information about the connection.

Cheers,
Alana

@e-rotteveel_1850 thanks for explaining that, that's great to know!

Sometimes it's almost impossible to learn how these feed/package types are actually used, especially since we don't develop in those languages and really just focus mostly on API reverse-engineering ;)

FYI We are targeting late April for 2023.0 release

Hi @jim-borden_4965 !

I think that the "delete old versions" option of Retention Rules might be what you're looking for; that will let you keep the last "X" versions of each package. That, in combination with "unused versions" (i.e. not recently downloaded) typically cover nearly all desired retentions.

As far as a "feed independent" API, that's on our list as a "nice to have", and I don't know how much of an API we'll get in the first version of ProGet 2023. The "hard part" is usually specifications/docs, so if you have any ideas we'd be very open!

Currently, our idea is base it off of the upack api:
https://docs.inedo.com/docs/upack-feed-api-endpoints

Some things will be more difficult (or impossible?) than others. Especially for multi-platform package types like ruby, python. But TBD.

Cheers,
Alana

Hi @hashim-abu-gellban_3562 ,

I haven't set up Clair (v2) recently... but a little while back (as part of a ProGet regression test), I did set it up and it worked. I remember it wasn't very straight-forward and I had to redo a few things because I fat-fingered some of the Docker commands.

I'll see if I can some additional help on this, please stay tuned...

Cheers,
Alana

Hi @dhurley_5516

How to enable download statistics (disabled by default and we need enabled)

This is controlled by restrictPackageStatistics property.

How to support both ODATA (v2) and JSON-LD (v3) (default is only ODATA (v2))

There's no property for this; however it should be enabled by default (it is not). We can also add this as useApiV3

Which property in JSON controls the "Remove signature file" option (enabled by default but we need disabled)

There's also no property for this....; however it IS currently disabled by default . We can also add this as stripSignature

I made these changes as PG-2317, and they'll be in ProGet 2022.27 (which should ship next Friday).

Hi @hashim-abu-gellban_3562 ,

Currently ProGet works with Clair v2. Unfortunately, Clair v4 (there is no v3 by the way) is basically a "different product" and the API is completely different. The vulnerabilities that are scanned/reported are the same, it's really just the back-end. We are exploring updating to v4 (a major change) or just creating our own container scanner for PGVC; both are major undertakings.

That being said, it sounds to me like Clair v2 is currently running okay.

What's really through us off is the error message that you're getting...

Fetching updates for Clair_Index_Docker...
Persisted object is not a VulnerabilitySource.

That's an internal error to ProGet, and basically ProGet is failing to even try to query Clair. This must be a new regression (there are some new preview features for vulnerabilities), but we just can't figure out how you are getting that particular error message.

Essentially, it means the configuration in the ProGet database is incorrect; the Configuration_Xml column from select * from VulnerabilitySources should look something like this:

<Inedo.Extension.Clair.VulnerabilitySources.ClairVulnerabilitySource Assembly="Clair">
  <Properties ApiUrl="http://localhost:6060/" AuthenticationHeader="MySecretKey" />
</Inedo.Extension.Clair.VulnerabilitySources.ClairVulnerabilitySource>

Any insight or more information would be really helpful - especially if you can query the ProGet database to see what's in the table.

This is why Rich asked if you can "edit" the Vulnerablity Source in the ProGEt UI, because that should give the exact same error if the config is invalid.

Thanks

Hi @mhdos_4222 ,

I'm not sure if that's related (and it was, why changing network would have any impact last time).

When the current user is not authorized to perform the required task in a Docker feed, this is code that ProGet runs:

string[] scopeParts = new[] { "repository", fullRepositoryName, ex.SecuredTask == (int)ProGetSecuredTask.Feeds_ViewFeed ? "pull" : "push" };
context.Response.AppendHeader("WWW-Authenticate", $@"Bearer realm=""{authUrl}"",service=""{context.Request.Url.Host}"",scope=""{string.Join(":", scopeParts)}""");
WriteError(context, new DockerException(401, "UNAUTHORIZED", ex.Message), feed, w =>
{
    w.WriteStartObject();
    w.WritePropertyName("Type");
    w.WriteValue(scopeParts[0]);
    w.WritePropertyName("Name");
    w.WriteValue(scopeParts[1]);
    w.WritePropertyName("Action");
    w.WriteValue(scopeParts[2]);
    w.WriteEndObject();
});

I do know that the Docker authentication stuff is very sensitive/complex, and I don't think anyone can answer why ProGet writes pull instead of pull,push. But whatever we're doing now works...

This code has basically been the same for about five years now, and we don't want to just change it because that will probably cause something to break.

Do you have any documentation/evidence/specs or anything that points to pull,push being a correct response?

Thanks,
Alana

Hi @mhdos_4222 ,

I'm afraid I'm not sure how to help troubleshoot this very well; behind the scenes, the server (i.e. ProGet) side of things is fairly simple. Images are basically just manifest files and blobs, and ProGet will add those to the repository when receiving commands to a somewhat basic REST API.

What I would do is use a tool like Fiddler to capture the HTTP traffic between the client and server, and see if you can identify if there are any failed or missing requests. For example, maybe the client is never uploading 3cc66... for some reason. Or perhaps, the request is getting "eaten" but your ingress-controller for some reason.

I would also try to take your somewhat complicated configuration out of the equation, and just go with the most basic setup possible, like this: https://docs.inedo.com/docs/proget-how-to-install-on-aws-lightsail

You can then compare/contrast the HTTP traffic and find where there are issues.
Cheers,
Alana

Hi @e-rotteveel_1850 ,

Sorry but I had this mis-categorized internally , so I didn't see the reply.

This actually requires a fair amount of under-the-hood changes, because of the way we maintain an index of the conda packages in a SqlLite database. It's not terrible, but it's also not complex.

This was added as a "nice to have" in PG2023 :)

Cheers,
Alana

Hi @sebastian ,

FYI; for now, we'll plan to add support for OR when reading a SPDX from a manifest; we'll add this to the "nice to haves" in PG2023!

Cheers,
Alana

Hi @e-rotteveel_1850 , sorry I missed the notification on this b/c of how I categorized this.

Yes.. Q2 (late April for now) is the current plan, though this is something we would likely do after the main release.

Hi @falk-winkler_2111 ,

I believe in this case, you'll just enter the access token as the password, and select "Bearer" as the authentication type.

Cheers,
Alana

@ade8s_7742 so glad to hear that! It's really rare, but glad it wasn't database corruption!!

Hi @doejohn_7742 ,

The underlying error message "Invalid object name 'dbo.RpmPackages' is implying that there's something pretty wrong with the database; in this case, a missing table (RpmPackages).

If that's the case, it wouldn't be easy to fix or troubleshoot unfortunately, and would require SQL expertise, etc. We don't have any general advise for this, but we may be able to help some paid users depending on the issue (it's quite time consuming as you can imagine).

HOWEVER -- it could also be something really simple, like your username (i.e. set in the connection string when you upgrade) is in the wrong schema (needs to be dbo schema), so I would check that too. Our script is supposed to detct that, but sometimes it doesn't work.

Cheers,
Alana

Hi @darturow_6059 ,

I think the equivalent to a "Generic Repository" would be ProGet's asset directory:
https://docs.inedo.com/docs/what-is-an-asset-directory

However, there is no "connector" possible, except to another ProGet asset directory.

Cheers,
Alana

Thanks for clarifying @avaleriusdebeffort_3858

In this case, please do not monitor the console output. There is no useful information that you will gather from that, and it's entirely a diagnostic tool. It cannot be adjusted

If you wish to monitor the health of ProGet, please check the /health API endpoint. That will contain all of the information that you need.

@avaleriusdebeffort_3858 what do you want to log? You can enable more detailed logging on ngnix... but otherwise, in ProGet, we don't have any more detailed logging available for end-users to troubleshoot.

Hi @Justinvolved ,

We don't currently have that type of monitor/trigger, but it's something we'd like to support in the future... in addition to new triggers like for Docker images, etc. Our extensible ResourceMonitor Feature is capable of handling this, but it just needs to be coded.

If you're interested, it'd be relatively easy...

• Create a create a class called PackageVersion that will store latest version number of a package (like Revision Number on SvnRepositoryCommit.cs)
• Create a resource monitor that queries for package versions (like how SvnRepositoryMonitor.cs monitors paths)

Hope that helps !

Cheers,
Alana

Hi @mhdos_4222 ,

Sorry, I must be confused.

Is the image actually pushed? It's not very clear from the logs, but it looks like it's working after you login? You wrote:

The push command was executed after successfully logging into our self hosted Proget.

A 401 basically just means the credentials are incorrect or not sent, and I really can't tell from the logs what's happening or not happening.

If things used to work before, do you know what was changed ? It's just hard to guess from the information we have here.

Hello,

ProGet will issue a 401 (Unauthorized) when the request has not been authenticated or if the authenticated credentials are not correct.

We don't have instructions or troubleshooting steps for nerdctl, but in the docker client you will use a login first., before a push.

Here's some more info:
https://docs.inedo.com/docs/proget-docker-private-registries#creating-and-using-a-docker-registries-in-proget

Cheers,
Alana

Hi @OtterFanboy ,

Thanks for the feedback and for finding additional bugs; I just fixed these as BM-3826 - at least I think.

1. Files was definitely broken (I must never have tested it last time)
2. That was probably related to something else (syncing issues?)... but hard to say; in theory, it should give a "red error box" when there's a problem w/ Git instead of a 500
3. This flow was using the wrong properties to create the resource name
4. Same

I also corrected the "new application flow" so that Rakko will do a "repository detection" to help select a pattern to use.

Let us know if you find other UI bugs (browsing, etc.) with the "Generic Git" repository; it should work the same as Git-service integrations, but we didn't do very good regression testing on it.

Hi @Bob_4018 ,

Thanks for the heads-up! We've resolved this, and the link should now work:

http://cdn.inedo.com/downloads/otter/OtterInstaller22.0.9_Offline.exe

Cheers,
Alana

Hi @martin-helgesen_8100 ,

You can aggregate multiple feeds using connectors:
https://docs.inedo.com/docs/proget-feeds-connector-overview

If you want to put all the packages in one feed, then after adding the connector you can use a "feed downloader":
https://docs.inedo.com/docs/proget-feed-importing

Cheers,
Alana

Hi @jim-borden_4965 ,

It looks like that isn't supported; but it should be relatively easy :)

I added a change (PG-2305) that we'll try to get in the next release (scheduled Mar 24).

Cheers,
Alana

Hi @e-rotteveel_1850 ,

ProGet doesn't currently support deleting individual "files" like this under a package, but it's something we can add for sure. The UI definitely seems to act "suboptimal" with some of these conda packages, but we'll fix it!

I suspect it is best for us to wait until ProGet 2023, since we're doing a big data model changes. I'm going to keep this "open" internally, along with the "constrains" request you made earlier (https://forums.inedo.com/post/13876) :)

Cheers,
Alana

@sebastian no problem :)

Actually... I just realized that I have a backup of your database that you sent to us for analysis/review. And your vulnerability assessments were one of our test datasets to see how migration goes.

So we can give you a very accurate and specific answer once we get to that phase of the ProGet 2023 release

Hello @sebastian ,

Great questions, and I'll do my best to help. This is a little complicated I think :)

[1] PGVC vs OSS Index
From a technical standpoint, PGVC is implemented as an "offline database", which offers a lot of performance benefits - namely ProGet can know about vulnerabilities in packages you're not yet using, and display those on remote packages. ProGet will download updates on a nightly basis.

Regarding the "Quality of data", it's really hard to say. I think everyone just aggregates from the same sources like NVD:

• PGVC leverages the Open Source Vulnerability (OSV) platform developed by Google and backed by Microsoft, etc. It’s an open platform.
• OSS Index is just Sonatype, and it’s closed (proprietary).

We decided to invest in PGVC because OSS Index has been rate limiting more and more, and the quality of results have been declining over the years. We believe PGVC (and the underlying OSV platform) will ultimately be superior.

[2] Instant Availability & Overnight Scanning
As I mentioned above, PGVC is an offline database. This means ProGet can immediately query that database to show you vulnerabilities on packages you may want to use or are currently using. This is not possible with OSS Index due to rate limiting.

The "vulnerability scan job" (which both OSS Index and PGVC scan do) will basically compare all packages you have in ProGet (local/cached) against the vulnerability source. This is to show you about vulnerabilities discovered in pakcages you're using.

[3] Migration
We are planning on some guidance about this. In theory, its should be possible because both the PGVC and OSS Index use CVE-ID. But the OSS Index sometimes uses their own ID instead of a CVE-ID.

We'll study some datasets and see what we can bring over. It might be a SQL Script or a tool inside of PRoGet.

[4] Using Both
I want to say, that you should just pick one source. Otherwise you’ll get a lot of duplicate vulnerabilities. Either one should be sufficient for package scanning, as they both aggregate the same publicly-available data sources.

However, it wouldn't hurt to try using both... just to see what comes up for vulnerabilities. If you delete a vulnerability source, it will delete all the assessments -- so that is a quick way to at least test (you can delete the PGVC vulnerability source).

Hi @OtterFanboy ,

FYI, Generic Git repo browsing in the UI has now been fixed as BM-3822 and will be included in the next maintenance release (March 10). If you'd liek a patch/prerelease sooner, just let me know!

Cheers,
Alana

Hi @priyanka-m_4184 , thanks for updating; yes in this case, it must be a really large package file? 30min is long time, and the upack client must be timing out then.

Hi @e-rotteveel_1850 ,

Thanks so much, this will help quite a lot and should be easy to follow! I downloaded those package and attached them to our internal tracker.

So basically... it sounds like we should just treat constrains (no t ) like we do depends? And if we can display it in the UI, then we will.

I peeked at the code, and it's a bit more complex than I hoped... mostly because of how we have to index/cache "connector" data as a SQL Lite database. But hopefully not that complex.

Anyway, I'll update once we have an idea of when we can get this field in.

Cheers,
Alana

Hi @Justinvolved ,

The first thing I would try is enabling verbose logging; this is on the advanced tab of the operation, or you can set with Verbose: true parameter.

That will give a file-by-file comparison, and show you what's being copied and not. You will see texts like:

Copying C:\apxltd\artifact\Inedo.DependencyScan.dll to C:\apxltd\artifact2\Inedo.DependencyScan.dll...
Inedo.DependencyScan.dll already exists in C:\apxltd\artifact2.
Source timestamp: 11/12/2022 8:26:38 AM, Target timestamp: 11/12/2022 8:26:38 AM
Source size: 54272, Target size: 54272
Size and timestamp are the same; skipping Inedo.DependencyScan.dll...

Hopefully that will help trace this a little bit better.

Cheers,
Alana

Hello @jmartschinke_3948 ,

Thanks for letting me know; this appears to be some kind of regression from a python-related change in 2022.20 .

It will be addressed ASAP via PG-2290 in the next maintenance release (coming ASAP).

Cheers,
Alaa

@OtterFanboy got it, thanks!

So this particular change is a bit more involved unfortunately. For example, if you clicked on "Windows Server", you wouldn't find any drifted servers - which would be confusing, because the environment would show as drift.

I'll note this as something to think about for v2023

Hi @e-rotteveel_1850 ,

Thanks for the suggestion! We could definitely use your help in getting a few more details on how to implement this... we only learned CONDA by trying to implement a repository

I couldn't find any info about constraints from searching their documentation.

Can you provide us with an example package we can upload to ProGet (public package is easiest, but private is fine too)? And also, show us what it should look like in the repodata.json file?

Cheers,
Alana

Hello @OtterFanboy ,

Thanks for the bug report! We'll fix it via OT-483.

That tab only displays if "Configuration drift:" is set to something other than "None", but there should be a "no roles" type of message instead of a crash.

Cheers,
Alana

Hello @OtterFanBoy123 ,

I was able to reproduce this; it's a UI regression BuildMaster 2022's new OtterScript editor. This won't impact deployment, the script, or anything like that.

I've logged this as BM-3817 and we'll try to get this fixed in the next maintenance release :)

Cheers,
Alana

@lm thanks for letting us know that it worked!

I'm glad you were able to find the issue on GitHub. Looking closer, it is a NuGet client bug; I posted an update on the issue. The only workaround is what you described (setting a separate API key).

Hi @cshipley_6136,

We aren't trying to pass the buck here, but given the symptoms, it's almost certainly not a software problem I'm afraid. Under the hood, ProGet uses Microsoft's SQL Server driver/components, which uses the operating system's networking drivers/components to communicate to the server. In this case, the error is originating in the operating system's networking components.

Based on the symptoms (intermittent network-level errors), it's most certainly a problem with the network hardware, components, or configuration. Since the problem is intermittent, running nslookup similar commands on the container won't really identify any issues; those commands also communicate on different protocols/ports than SQL Server.

We're not really experts at troubleshooting network problems, but we have seen a few over the years. Have you had a chance to bring this up with your Network/Operations team? What have they tried or investigated so far?

Cheers,
Alana

Hi @lm , thanks for the feedback!

I'm not sure about #1, but I was curious about #2, and spotted a typo in our database updating code.

If you run this against your ProGet database, this will work and should unblock you:

GRANT EXECUTE ON TYPE::[IndexedSymbolEntry] TO [ProGetUser_Role]

This will also be fixed in in ProGet 2022.21 as PG-2287.

Cheers,
Alana

Hi @cshipley_6136 ,

In general, when the database has temporary problems (timeouts, unavailability, etc.), then errors will be shown to users (like the above) when they make a web request. Every web request is a new try, basically, and will cause the same error. But once the database is back up, new requests should work fine.

However, if the application fails during initialization (i.e. the first web request after starting the service), then it effectively requires restarting the application (container/service). This scenario is tricky to work-around.

Otherwise, what was the nature of the user auth issue? If it was an incorrect password, then that would require fixing the connection string (which is passed as an environment variable usually) and then restarting the container.

Hope that helps...

Alana

Hi @cshipley_6136 ,

Thanks for the additional information; we were able to figure out what the underlying issue was with your help. Essentially, it a combination of command caching and some other factors that caused this false-positive behavior. The error should have eventually been triggered after a little while, but it's hard to say.

In any case, we've changed this to use a one minute cache, so this kind of error will be detected much quicker. Either databaseStatus will say Error, or the handler will return a 500 status.

This will be fixed in PG-2284 , which is scheduled for this Friday's maintenance release (2022.20).

Cheers,
Alana

Thanks again for these reproduction instructions! I confirmed the behavior very easily, though it doesn't look like a trivial fix (at least to me)

We are targeting PG-2278 for the upcoming maintenance release (Friday), but it might be delayed if we have other priorities until the following maintenance (Feb 24). Of course we can provide a prerelease/patch version as soon as we code the fix :)

Thanks for the additional insight! To clarify... I know very little about how caching works, and I was just reporting what changed recently so we can track down where to look :)

So just to confirm... you're saying that this used to work in ProGet 6.0, but it's not working after you upgraded to ProGet 2022? If that's the case, then it would very likely be the platform change.

@pfeigl said in Assets do not return Last-Modified header (anymore?):

Anyways, I guess our question simply is: Is it reasonable for you to (re-)add this header in a future version? It feels like a simple change, as the asset UI already shows this exact field.

Yes, we just need to track down exactly what the issue is :)

Our platform code does seem to look for an If-Modified-Since header, and then sends a 304 if the dates are within a minute of each other. So I guess that works.

But then this code, when sending response headers, looks pretty suspicious to me.... I wonder if he should be setting the Last-Modified header instead of Date

Hi @joacim-svensson_8194 ,

This use case isn't very common (and isn't one we necessarily designed for), so it's not so intuitive to do in the UI. To handle this, you can create a create a second vulnerability source, and then use that source in the UI. Let us know how that goes!

Cheers,
Alana

Hi @Justinvolved ,

Hundreds of configuration files in one application -- definitely too much. I'd probably seek a different solution if that's the case. But having one or a few per application is okay.

There is an Http-Post Operation that you can probably use to make API calls. Conceptually it's similar to the PowerShell Invoke-WebRequest method.

I'm not sure what the configuration file would look like (array? map variable?), but there's also a variable function, ($Filecontents()), that could read a file, and an $Eval() function that can can convert text into variables.

That said... it might be a bit challenging to do all this in OtterScript. It's not really designed for this. You may be better off writing a global PowerShell script that can process input from a configuration file that you deploy to the working directory. That would also be easier to test as well.

Cheers,
Alana

Thanks @e-rotteveel_1850 , these repro steps will be very helpful to debug/fix the problem.

With platforms we know little about (Python, Conda) figuring out the repro steps is often the hardest part

Please stay tuned; we'll post an update once we identify or fix the problem

Hi @e-rotteveel_1850 ,

I haven't really tested this, just reporting on information I found in our notes :)

The problem that we fixed via PG-2220 was reproduceable as follows:

1. Create Conda feed w/ connector to default public repository
2. Access this page in UI: feeds/MyCondaFeed/ca-certificates/versions

It also gave errors in the ProGet API. But the underlying issue was related to unexpected ("invalid") metadata from the remote Conda repository's API (index files), specifically with sorting (comparing) those leading zeros.

Python specifications give me a headache, but it has something to do with the PEP-440 Normalization Rules not being followed under the hood

Regardless, it sounds like this is a different bug...

Can you identify how we can repro/fix (without using CONDA client)?

1. Create new CONDA feed (no connector)
2. Download ???? files from the public site (I couldn't tell from this page)
3. Upload those files to ProGet
4. Web UI shows ???? instead of ???
5. API url ???? doesn't download file (as expected)

Cheers,
Alana

Hi @pfeigl ,

The handler for asset file downloading hasn't changed recently.

The last major changes were in ProGet 6.0, where (among other things), the ability to control client-side caching was added. The change you found (PG-2068) fixed a bug related to UTC/Local time differences in those cache headers.

In ProGet 2022, we changed the overall platform (.NET Framework -> .NET6). The platform is what's responsible for reading/responding to cached/head requests.

I'm not sure what the behavior was prior to ProGet 2022... but if you're finding that caching isn't working as expected, I would inspect the cache control headers and see if you can find what the underlying issue is. So far as we can tell/test, it's working as it's supposed to now.

Cheers,
Alana

Whoops, posted too fast :)

@e-rotteveel_1850 said in Uploading ca-certificates (2023 version) to a ProGet conda feed does not work:

I tested with Proget 2022.0.19, but the file is still renamed to "2023.1.10"

The package store is internal to ProGet, and we don't support accessing or modifying those files directly. The folder structure or naming of the files won't impact usage in the ProGet UI or API.

Hi @e-rotteveel_1850 ,

The underlying problem is that ca-certificates has invalid package versions, at least according to Conda's own versioning specification.

2022.07.19 is not supposed to be permitted in a repository... and yet it's there. ProGet follows the Conda specification, which says packages with leading zeros should be "normalized" to 2022.7.19. I guess they treat their specs more like "guidance" than "specifications"

In any case, as you noted, this was addressed in a newer version (PG-2220 in ProGet 2022.10), so upgrading should take care of it.

Cheers,
Alana