Posts made by atripp

Hi @norbert ,

Based on the error messages, the problems is occurring somewhere between the ProGet and choco.

• Chocolatey is reporting that the remote connection was forcibly closed.
• ProGet is reporting that the client unexpectedly disconnected.

It's the equivalent of someone "cutting the cord" halfway through the communication, and there's no way to know where that happened or why.

You'll need to investigate intermediate equipment. Typically it's a firewall, ,proxy, or other gateway that is misconfigured.

Cheers,
Alana

Hello,

You can find it under Admin > License Key; here's more information that might be helpful:
https://docs.inedo.com/docs/myinedo-activating-a-license-key

Cheers.
Alana

Hi @philippe-camelio_3885 ,

I don't believe that root is required to SSH into a Linux server.

What did you mean by implicit sudo? I guess you could run sudo from your shell scripts, but I wasn't sure if that's what you meant.

Cheers,
Alana

Hello @jonathan-brown_3775 ,

This is considered a "false positive", so please disregard those results :)

Those are JavaScript files that run inside your browser to display certain charts about feed disk usage in the Web interface; there is no possibility whatsoever of a security vulnerability or attack coming from these javascript files inside of ProGet.

Thanks,
Alana

Hi @pooms_7365 ,

Thanks, I found where the error is in the code; it was in displaying the "code has not been synced" message. Easy fix (BM-3845) and it will ship in next maintenance release :)

And wow... 15gb!

Cheers,
Alana

Hi @ForgotMyUsername ,

1. Checkout-Code operation using Generic Git repository with recurse submodules...

I can't reproduce this, but I can see how that issue might occur. Your repository must have a GitLink entry in the target tree (i.e. folder you're checking out), but there's no corresponding reference entry in the .gitmodules file. I've updated that to be a warning., and just pushed the Git 2.2.3 extension.

It will be included in the next maintenance release of BuildMaster, or you can download/upgrade now.

2. Every minute an exception appears...

Nice find; this will be fixed in next maintenance release (2022.10) via BM-3844

3. Clone application fails when trying to clone application using generic git repository...

I believe this is related to importing "releaseless-builds", and this may be fixed in 2022.9 via an internal commit (i.e. one we didn't log an issue for). Clone does an export then import. If that doesn't fix it, please export the application and provide it to us, so we can inspect the file for issues.

Regarding [5], can you send logs? The scenario you describe (existing repo with the same name) is already tested and handled, but there's likely something that's missing.

Hi @ForgotMyUsername ,

Unfortunately this is not going to be possible to debug with the information you've provided.

Except for the misspelling of Certficiate (which would have no bearing on executing the script), a cursory glance of the code shows that everything is otherwise wired up as expected:

• IisSiteBindingConfiguration.cs
• IisSiteConfiguration.cs

So, I'm hoping you help troubleshoot/debug some more, to show us where the issue is.

IIS settings are ultimately stored in the applicationHost.config stored under %windir%\System32\inetsrv\config\. We do not directly work this file (as you can see from the code), but it's the easiest way to review the settings in IIS that Otter makes.

To help us reproduce these issues, can you send:
(1) Relevant appPool and site elements from the config file
(2) OtterScript code that should work with those entries
(3) Actual vs expected results in the config file

And one note on the bindings... unfortunately this is a difficult problem in general to solve because IIS Sites have multiple bindings, but users often don't want to clear bindings. We used to have a Bindings property (it's still htere) that lets you specify a map, but it was too difficult/confusing to use.

Cheers,
Alana

Hi @ForgotMyUsername,

Under-the-hood, Otter uses an in-process PowerShell Host (i.e. a runspace) to execute scripts, and does not invoke PowerShell.exe:
https://github.com/Inedo/inedox-scripting/blob/master/Scripting/InedoExtension/PowerShell/PowerShellScriptRunner.cs#L181

And as you can see in the code, Otter is properly creating/disposing of those runspaces.

I'm not sure where the powershell.exe processes are coming from, but it sounds like you have some problematic scripts. These will cause a lot of memory to be consumed and never released. In addition, perhaps they are calling other scripts using powershell.exe? Or something? It's really hard to guess...

In any case, Otter already has a mechanism called Isolation (with isolated) that can be used to help solve problematic scripts like this.

When something is run as isolated, then a new Otter.Agent.exe process is created to handle the unit of work (such as a script execution). When the work is done, that process is terminated; if it doesn't terminate, then it is killed within 5 seconds.

Cheers,
Alana

Hi @jimbobmcgee,

OtterScript is a DSL and doesn't have a lot of "general purpose" things like maths, as you've noticed. This is by design.

But you can just use PowerShell for that; for example, set $MyResult = $PSEval((5 + 3) * 4 / 3^0.5);

There isn't any noticeable overhead.

PowerShell is probably also more suitable for string manipulation as well, especially when you're working with diskpaths and the like.

Cheers,
Alana

Hi @h-morgenthaler_3015, we are currently investigating this. ,It sounds like an issue we fixed in ProGet 2022 already, but maybe it didn't get merged over. Please stay tuned.

Hello, unfortunately we really don't know what the underlying issue could be. Perhaps it's only an error on the "test" button, and the plug-in will work?

This is not an Inedo-created plugin, and we're not familiar with Java or Jenkins to know how to to debug it. It should still work (and I know we have some users who got it working), but otherwise we don't really know how to debug it I'm afraid :(

The source code is here, but I don't know if it's at all helpful:
https://github.com/jenkinsci/inedo-proget-plugin

Cheers,
Alana

Hi @jimbobmcgee ,

The "Secret key" field is just an arbitrary string that needs to match on both the Agent and the Otter Server; it needs to be unique across your servers, as its used to uniquely identify an incoming agent connection. The UI generates a random string, but you can enter whatever you'd like.

There needs to be a Server record on the Otter server (i.e. on the "Servers" page) before an agent can connect to Otter. This is true for either Incoming or Outgoing communication modes. This is often called "server registration".

One option is to servers self-register in Incoming mode; you can have the the following script on first boot:

1. Generate a random string ("secret key"); you could use a GUID
2. use the Infrastructure API from the server to "register" the server using that key
3. edit the agent configuration file to have that key
4. start the Inedo agent service

Hope that helps,
Alana

Hi @Justinvolved ,

Oooh - I thought we removed those flags/notices from BuildMaster 2022

Looks like we only removed the code that clears them, and almost all of the code that sets them (with the exception of the CEIP page).

Anyway, I've vanquished the code for good via BM-3839.

In the mean time, please run sql this command to clear the notices:

DELETE [Configuration] WHERE [Key_Name] IN ('Service.DelayExecutionsWhenRestartRequested','Service.RestartRequested', 'Web.RestartRequested')

Cheers,
Alana

Hi @d-gleissner_8620 ,

The best way to handle this scenario is with two feeds (unapproved and approved packages); install from the unapproved in your development environment, and then to promote packages (in bulk) from the unapproved to the approved feed.

Unfortunately, it's not feasible for a tool like ProGet to handle dependency resolution; a dependency is specified as a range (for example, PackageA requires PackageB 3.2 or later), and it's up to the environment (i.e. the client doing the installation, PSGet) to determine which dependencies are currently available and not available.

For example, if you already had "PackageB 3.3" installed, then PowerShell would not install it, even if "Package B 3.4" is the latest version. It gets even more complicated when it comes to platform dependencies and all that. Ultimately, ProGet can't have such environmental knowledge.

hoep that helps!

Cheers,
Alana

Hello,

This is by design; looking over the code, it seems the command says "If metadata file is provided, value will be ignored." I can't answer why that's the design/behavior.

You're welcome to submit a pull request to update the behavior. Here is the relevant code:
https://github.com/Inedo/upack/blob/master/src/upack/Pack.cs

My main concern is, what if someone is relying on the value being ignored , either accidently or intentionally. I don't have a great idea for how to address that concern... though my first thought is to deprecate the manifest argument and rename to manifestFile.

So it's not a trivial change, but you're welcome to give it a shot. It's unlikely we will implement this ourselves anytime soon, as we have a lot of other feature requests on ProGet that take priority, and UPack gets a decent number of pull requests from the community.

Cheers,
Alana

Thanks @sebastian !! Your RegExFu is impressive

Definitely more robust - so I replaced it, and now we will match that "silly" test case of ((a) OR (b))

Hi @martin-noack_4528 ,

Nice points :)

We added this to our ProGet 2023 roadmap as a "Nice to Have"; we're a little behind on the release now so I don't know if it will make it, but we will reevaluate after 2023.0 is released at that point.

Cheers,
Alana

Hi @sebastian

Just an update; this was committed to the PG2023 code base, and seems to work on a few packages I tried (but I can't find too many).

Basically we just will enumerate the code matches in this Regex: ^\(?(( OR )?(?<code>[0-z-_\.]*))+\)?$

That will catch (a OR b) and a OR b, but if one were to be silly and add ((a) OR (b)) then it would revert back.

Doesn't seem worth getting any more complex than that :)

Hi @js-enthoven_2797,

Unfortunately I'm not very familiar with debugging apt :(

The only one I was familiar with was the InRelease-related error, which you worked around.

Otherwise, I haven't seen these before, but they seem to be warnings, so maybe it's okay and is working? It does say "All packages are up to date".

I don't really know what they mean, or if it's related to ProGet, apt configuration, or the package I searched for the text of these errors ("No Hash entry in Release file"), and there are a lot of suggestions on what to do... they are all different, and I have no idea what might work.

The "Connection timed out" is really strange too. Maybe it's related to proxy, or something? A blank page is to be expected if you have no packages in that scope; otherwise you will see a number of "Paragraphs", one for each package.

Cheers,
Alana

Hi @js-enthoven_2797 ,

This issue is unrelated to internet access or whitelisted URLS.

This is the error message you will get from apt if a private repository does not support the InRelease API endpoint. ProGet does not support this endoint, so you will get this error.

Here is some information about how to bypass this warning in apt:
https://www.linuxfordevices.com/tutorials/linux/fix-updating-from-such-a-repository-cant-be-done-securely-error

Basically you have to explicitly trust the PRoGet repository.

Please let us know if that works,

Alana

Hi @js-enthoven_2797,

I'm not totally certain, but I believe this is the result of the InRelease endpoint (i.e. the clear-signed index) is not being implemented by ProGet.

You can ignore the "weak security information" message, and configure apt to use this repository. If you're using HTTPS, the clear-signed index adds no additional security; it's a vestigial feature these days, and is was designed for when HTTPS wasn't available. You can ignore/override those warnings.

We may implement the InRelease endpoint, but there are some problems/bugs with the way BouncyCastle (the encryption library we use) generates "armored output streams". It's a lot of effort for no real value (other than just not having those outdated errors by default).

Cheers,
Alana

Hi @v-makkenze_6348 ,

I can see that you're using the OSS Index? Did you also add PGVC as a source?

I didn't review or try to reproduce this particular case yet... but we have seen this "duplicate data" problem happen from time to time with OSS Index. It's a data-quality issue; ProGet maintains an "external ID", and sometimes OSS Index will report duplicate

ProGet will display the External ID when you click on the vulnerability; that should be unique.

However, based on the description... I wonder if that's the case here? Does this seem to happen with cached packages only, as they've been removed? Any other insight you could provide would be very helpful, so we can investigate this further.

Cheers,
Alana

So sorry @msimkin_1572 , I just double checked.. the Endpoint URL should be formatted like v3/index.json not index.json.

http://my-internal-server/nuget/InternalLibs/v3/index.json

It "looked" right when I typed it, but I should have double checked. Normally I just copy/paste this from the Endpoint URL in the feed page.

Hi @PMExtra ,

Thanks for the details on this, very helpful :)

We actually recently discovered this too (though a different underlying problem) - it was should be addressed in 2022.27 via PG-2321.

Can you try that and see if it helps? We can investigate further if not.

Cheers,
Alana

Hi @msimkin_1572 ,

Sorry I didn't realize this before... the URL needs to be the Feed Endpoint URL, which would end in /index.json. I also updated the documentation to make this clear.

The URL you have now is the v2 endpoint url, which doesn't have a feed description document.

Hopefully it'll work as soon as you add /index.json to the URL :)

Cheers,
Alana

Hi @kichikawa_2913 ,

PG-2290 was the regression fix, and it was included in ProGet 2022.22

Cheers,
Alana

Hi @philipp-cender_3322 ,

Looks like you're making good progress :)

I ran the command .\inedoagentservice.exe run but the command only stated that it does: "Starting agent connector to the otter-host-fqdn on port 8630"

That's okay to see; it means that its working as expected. I there was an error, you would see it.

But for Source 10.67.0.17 something like a token exception is stated

In the screenshot, it seems okay and doesn't report an error. So I think it's okay?

The exception message is some kind of OS-level error, and I'm not sure what it means exactly. But it's a SSL/TLS issue. In this case, if you search for the text of the error ("Die Anmeldeinformationen, die dem Paket übergeben wurden, wurden nicht erkannt" -- but perhaps English is better), you can probably get some details on how to fix it. It could be some obscure operating system configuration.

If i open otter at Port 8630 (Agent Listener) on Firefox webbrowser i get something like that: PR_CONNECT_RESET_ERROR

It's not possible to "browse" such a connection; the Inedo Agent uses a proprietary, TCP-based binary protocol. So you will always get errors if you try to browse, telnet, etc.

Cheers,
Alana

Hi @msimkin_1572 ,

The blog article was technically incorrect on the arguments, so I changed that.

However, the documentation is updated and should be accurate:
https://docs.inedo.com/docs/proget-feeds-nuget-symbol-and-source-server

Basically, if you have both a nupkg and a snupkg in your folder, then you should only have to use

dotnet nuget push kramericalib.4.1.2.nupkg  --source https://proget.kramerica.corp/nuget/internal-nuget

The .snupkg file will be stored on the same feed.

The symbol server url hasn't changed (it's still /symbols/feed).

hope that helps :)

Hi @cjchambers_1799 ,

Thanks for clarifying that; that handler outputs logs in plain text format (content-type is like txt/plain or something like that, not html).

That url is intended mostly to download/export the log files for APIS and other tools, but we put it in the UI for discoverability. And sometimes it's convenient to CTRL-F everything.

Since it's plain-text, it's not possible to do any kind of formatting (even a user-style sheet). But I guess you could CTRL-F [Error] or something, and it will "highlight" it in the browser at the least.

Cheers,
Alana

Hi @cjchambers_1799 ,

I'm not entirely sure what screen that is.....I think the execution-in-progress page (with the moving bar)?

So you're using the built-in Dark Mode then? Can you (temporarily) try switching to Light Mode, just to make sure it's showing up in the proper/expected colors?

If that's the case, likely just a CSS bug/fix we need to make :)

I didn't test yet, but just want to be sure I know we're looking at right thing.

Cheers,
Alana

Hi @cjchambers_1799 ,

Log entries are stored with an associated log level (error, warning, info, debug), and when you Log-Error it writes that item to the log with an error level.

When displaying log entries, error-level entries are displayed in Red, warning are in Yellow, and Debug is in Light-Grey.

So, I think this already behaves the way you are asking.

Cheers,
Alana

Hi @jw ,

In ProGet 2023, we are recording a "publish user" when a package is published. I'm not sure how we will show/expose this (probably history page for now), but that's really easy to change once we collect the data.

We're really hoping to get to get 2023.0 released later this month :)

Cheers,
Alana

Hi @Justinvolved ,

That error message is coming from here:
https://github.com/Inedo/inedox-git/blob/master/Git/Git.InedoExtension/Operations/CanonicalGitOperation.cs#L73

Basically, it means that value you've specified for From is not a known Secure Resource. It's not a common error, and is likely the result of deleting/re-adding something.

I would remove the From argument from your OtterScript altogether (just leave it as Git::Checkout-Code;). I don't think you need it. Your build should already associated with a repository, branch, and commit.

Cheers,
Alana

Hi @Justinvolved ,

Looks like there's a Visual Editor bug with Exec statement; we'll investigate/fix ASAP .

But as for the Git, you mention it's happening during an execution (i.e. build process)? Anyway, can you share more of the execution log?

Are you able to browse the Git repository in the Web UI okay?

If that's the case, the error is on the build agent (or BuildMaster server) in the service process. That uses a different set of local repositories than the Web UI. There may be a troubleshooting step we can take to manually clear the local repository (on the service-side), but Id like to gather more info

Cheers,
Alana

Hi @philipp-cender_3322 ,

The "inbound connection" is complex and a relatively new feature, and I don't have a ton of experiencing troubleshooting - so I'll do my best :)

So far, everything looks okay to me.

On the Otter Server, are you seeing any errors related to the server under Admin > Diagnostic Center? I see the server is in an "Error" state.

On the remote server, does the service stay running? If so, that's indicating it's able to establish a connection. But one thing you can try is to stop the service, and run in interactive mode (i.e. run InedoAgentService.exe run on the commandline). That will show you information about the connection.

Cheers,
Alana

@e-rotteveel_1850 thanks for explaining that, that's great to know!

Sometimes it's almost impossible to learn how these feed/package types are actually used, especially since we don't develop in those languages and really just focus mostly on API reverse-engineering ;)

FYI We are targeting late April for 2023.0 release

Hi @jim-borden_4965 !

I think that the "delete old versions" option of Retention Rules might be what you're looking for; that will let you keep the last "X" versions of each package. That, in combination with "unused versions" (i.e. not recently downloaded) typically cover nearly all desired retentions.

As far as a "feed independent" API, that's on our list as a "nice to have", and I don't know how much of an API we'll get in the first version of ProGet 2023. The "hard part" is usually specifications/docs, so if you have any ideas we'd be very open!

Currently, our idea is base it off of the upack api:
https://docs.inedo.com/docs/upack-feed-api-endpoints

Some things will be more difficult (or impossible?) than others. Especially for multi-platform package types like ruby, python. But TBD.

Cheers,
Alana

Hi @hashim-abu-gellban_3562 ,

I haven't set up Clair (v2) recently... but a little while back (as part of a ProGet regression test), I did set it up and it worked. I remember it wasn't very straight-forward and I had to redo a few things because I fat-fingered some of the Docker commands.

I'll see if I can some additional help on this, please stay tuned...

Cheers,
Alana

Hi @dhurley_5516

How to enable download statistics (disabled by default and we need enabled)

This is controlled by restrictPackageStatistics property.

How to support both ODATA (v2) and JSON-LD (v3) (default is only ODATA (v2))

There's no property for this; however it should be enabled by default (it is not). We can also add this as useApiV3

Which property in JSON controls the "Remove signature file" option (enabled by default but we need disabled)

There's also no property for this....; however it IS currently disabled by default . We can also add this as stripSignature

I made these changes as PG-2317, and they'll be in ProGet 2022.27 (which should ship next Friday).

Hi @hashim-abu-gellban_3562 ,

Currently ProGet works with Clair v2. Unfortunately, Clair v4 (there is no v3 by the way) is basically a "different product" and the API is completely different. The vulnerabilities that are scanned/reported are the same, it's really just the back-end. We are exploring updating to v4 (a major change) or just creating our own container scanner for PGVC; both are major undertakings.

That being said, it sounds to me like Clair v2 is currently running okay.

What's really through us off is the error message that you're getting...

Fetching updates for Clair_Index_Docker...
Persisted object is not a VulnerabilitySource.

That's an internal error to ProGet, and basically ProGet is failing to even try to query Clair. This must be a new regression (there are some new preview features for vulnerabilities), but we just can't figure out how you are getting that particular error message.

Essentially, it means the configuration in the ProGet database is incorrect; the Configuration_Xml column from select * from VulnerabilitySources should look something like this:

<Inedo.Extension.Clair.VulnerabilitySources.ClairVulnerabilitySource Assembly="Clair">
  <Properties ApiUrl="http://localhost:6060/" AuthenticationHeader="MySecretKey" />
</Inedo.Extension.Clair.VulnerabilitySources.ClairVulnerabilitySource>

Any insight or more information would be really helpful - especially if you can query the ProGet database to see what's in the table.

This is why Rich asked if you can "edit" the Vulnerablity Source in the ProGEt UI, because that should give the exact same error if the config is invalid.

Thanks

Hi @mhdos_4222 ,

I'm not sure if that's related (and it was, why changing network would have any impact last time).

When the current user is not authorized to perform the required task in a Docker feed, this is code that ProGet runs:

string[] scopeParts = new[] { "repository", fullRepositoryName, ex.SecuredTask == (int)ProGetSecuredTask.Feeds_ViewFeed ? "pull" : "push" };
context.Response.AppendHeader("WWW-Authenticate", $@"Bearer realm=""{authUrl}"",service=""{context.Request.Url.Host}"",scope=""{string.Join(":", scopeParts)}""");
WriteError(context, new DockerException(401, "UNAUTHORIZED", ex.Message), feed, w =>
{
    w.WriteStartObject();
    w.WritePropertyName("Type");
    w.WriteValue(scopeParts[0]);
    w.WritePropertyName("Name");
    w.WriteValue(scopeParts[1]);
    w.WritePropertyName("Action");
    w.WriteValue(scopeParts[2]);
    w.WriteEndObject();
});

I do know that the Docker authentication stuff is very sensitive/complex, and I don't think anyone can answer why ProGet writes pull instead of pull,push. But whatever we're doing now works...

This code has basically been the same for about five years now, and we don't want to just change it because that will probably cause something to break.

Do you have any documentation/evidence/specs or anything that points to pull,push being a correct response?

Thanks,
Alana

Hi @mhdos_4222 ,

I'm afraid I'm not sure how to help troubleshoot this very well; behind the scenes, the server (i.e. ProGet) side of things is fairly simple. Images are basically just manifest files and blobs, and ProGet will add those to the repository when receiving commands to a somewhat basic REST API.

What I would do is use a tool like Fiddler to capture the HTTP traffic between the client and server, and see if you can identify if there are any failed or missing requests. For example, maybe the client is never uploading 3cc66... for some reason. Or perhaps, the request is getting "eaten" but your ingress-controller for some reason.

I would also try to take your somewhat complicated configuration out of the equation, and just go with the most basic setup possible, like this: https://docs.inedo.com/docs/proget-how-to-install-on-aws-lightsail

You can then compare/contrast the HTTP traffic and find where there are issues.
Cheers,
Alana

Hi @e-rotteveel_1850 ,

Sorry but I had this mis-categorized internally , so I didn't see the reply.

This actually requires a fair amount of under-the-hood changes, because of the way we maintain an index of the conda packages in a SqlLite database. It's not terrible, but it's also not complex.

This was added as a "nice to have" in PG2023 :)

Cheers,
Alana

Hi @sebastian ,

FYI; for now, we'll plan to add support for OR when reading a SPDX from a manifest; we'll add this to the "nice to haves" in PG2023!

Cheers,
Alana

Hi @e-rotteveel_1850 , sorry I missed the notification on this b/c of how I categorized this.

Yes.. Q2 (late April for now) is the current plan, though this is something we would likely do after the main release.

Hi @falk-winkler_2111 ,

I believe in this case, you'll just enter the access token as the password, and select "Bearer" as the authentication type.

Cheers,
Alana

@ade8s_7742 so glad to hear that! It's really rare, but glad it wasn't database corruption!!

Hi @doejohn_7742 ,

The underlying error message "Invalid object name 'dbo.RpmPackages' is implying that there's something pretty wrong with the database; in this case, a missing table (RpmPackages).

If that's the case, it wouldn't be easy to fix or troubleshoot unfortunately, and would require SQL expertise, etc. We don't have any general advise for this, but we may be able to help some paid users depending on the issue (it's quite time consuming as you can imagine).

HOWEVER -- it could also be something really simple, like your username (i.e. set in the connection string when you upgrade) is in the wrong schema (needs to be dbo schema), so I would check that too. Our script is supposed to detct that, but sometimes it doesn't work.

Cheers,
Alana

Hi @darturow_6059 ,

I think the equivalent to a "Generic Repository" would be ProGet's asset directory:
https://docs.inedo.com/docs/what-is-an-asset-directory

However, there is no "connector" possible, except to another ProGet asset directory.

Cheers,
Alana

Thanks for clarifying @avaleriusdebeffort_3858

In this case, please do not monitor the console output. There is no useful information that you will gather from that, and it's entirely a diagnostic tool. It cannot be adjusted

If you wish to monitor the health of ProGet, please check the /health API endpoint. That will contain all of the information that you need.