I have an Otter job template for which I have defined a number of custom variables, some of which are marked as Required and some are not.

If I view previous executions of that job in /jobs/history, there is a small (play) button icon against each execution, for which I believe the intention is to re-run any one particular execution with the variable prompts pre-populated with the previously-entered values for that execution.

If there were no custom variables defined on the template or, for that execution, if all the custom variables were populated with a value, clicking that button works as intended.

If, however, I did not set a value for one of the properties (i.e. it was left as an empty text box), clicking that button results in an HTTP/500 server error. The Diagnostic Centre page yields the following:

An error occurred in the web application: The given key 'MyCustomVariableName' was not present in the dictionary.

URL: http://xxxx:8626/jobs/from-template?jobTemplateId=Default%3A%3AJobTemplate%3A%3AMyTemplateFolder%2FMyTemplateName&jobId=140
Referrer: http://xxxx:8626/jobs/history
User: Admin
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0
Stack trace:    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Inedo.Otter.WebApplication.Pages.Jobs.CreateJobFromTemplatePage.VariablePrompts..ctor(IEnumerable`1 variables, Dictionary`2 selectedValues, VariableTemplateContext context)
   at Inedo.Otter.WebApplication.Pages.Jobs.CreateJobFromTemplatePage.CreateChildControls()
   at Inedo.Otter.WebApplication.Pages.OtterSimplePageBase.InitializeAsync()
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::HTTP Error on 19/12/2024 18:55:42::

I'm fairly certain it boils down to a simple null-ref check in the CreateJobFromTemplatePage class (decompiled from Otter.WebApplication.dll):

public VariablePrompts(IEnumerable<TemplateVariable> variables, Dictionary<string, string> selectedValues, VariableTemplateContext context)
{
    object item;
    CreateJobFromTemplatePage.VariablePrompts variablePrompt = this;
    foreach (TemplateVariable variable in variables)
    {
        if (this.variableValues.ContainsKey(variable.Name))
        {
            continue;
        }
        TemplateVariable templateVariable = variable;
        if (selectedValues != null)
        {
            item = selectedValues[variable.Name];    // <-- here
        }
        else
        {
            item = null;
        }
        if (item == null)
        {
            item = variable.InitialValue;
        }
        templateVariable.InitialValue = (string)item;
        VariableTemplateInput variableTemplateInput = variable.Type.CreateInput(variable, context);
        variablePrompt.get_Controls().Add(variableTemplateInput);
        this.variableValues.Add(variable.Name, variableTemplateInput);
    }
}

My guess is that you are not serializing unset/empty values or that they are not being deserialized correctly to the selectedValues dictionary by the caller. Whether or not that is a larger problem is unknown, but you can probably bandage it here by simply changing to...

if (selectedValues != null || !selectedValues.ContainsKey(variable.Name))
{
    item = selectedValues[variable.Name];
}

...without affecting the surrounding logic.

I have enabled the listening connection on my Otter install, and am trying to get a second server that has the agent installed to dial home to the Otter server.

The firewall is open, a self-sign certificate has been created on the Otter server, and its thumbprint has been configured in the Otter server's listener config. A public export of the self-signed cert has been installed in the Trusted Roots store on the server with the agent, and the windows dialog claims the certificate chain is therefore OK.

I have created a server object, with the server type of "pull", and pasted the secret key from the object into a Connections/Server node the InedoAgent.config file on the server. Also, Connections/@Enabled="true" in that file.

However, the server object in Otter is stuck in the Error state.

The Agent Listener Dashboard shows connections from the server with the agent, every 30s or so. The Diagnostics Centre shows errors in a matching timeframe, with:

Bad handshake from SERVERWITHAGENTIP:52768: System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognized at System.Net.SSPIWrapper.AcquireCredentialsHandle(ISSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED* scc) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED* secureCredential) at System.Net.Security.SslStreamPal.AcquireCredentialsHandleSchannelCred(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) --- End of inner exception stack trace --- at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) at System.Net.Security.SecureChannel.AcquireServerCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer) at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at Inedo.Agents.Connections.PullServerConnection.ReceiveHandshakeAsync(CancellationToken cancellationToken) at Inedo.Agents.AgentListener`1.ProcessIncomingConnection(TConnection channel)

If I set LogFile in the InedoAgent.config file, I see repeated entries for:

07/06/2023 06:13:44 DEBUG: Attempting to establish connection with OTTERSERVER:46336...
07/06/2023 06:14:14 DEBUG: Attempting to establish connection with OTTERSERVER:46336...

DNS and firewall look fine:

> Test-NetConnection OTTERSERVER -Port 46336

ComputerName     : OTTERSERVER
RemoteAddress    : OTTERSERVERIP
RemotePort       : 46336
InterfaceAlias   : Ethernet0
SourceAddress    : SERVERWITHAGENTIP
TcpTestSucceeded : True

I can add the standard .NET trace listeners to the InedoAgentService.exe.config, but I'm not sure what I'm looking for in the massive infodump the resulting trace file then contains.

What am I missing?

I am trying to determine if an Otter server object has been created, and am using the recommended (non-native) API to do this.

At the moment, in order to perform this test (at least as per the documentation), I have to issue an /api/infrastructure/servers/list call and loop through all the results, matching on the Name property.

As this project grows, this is going to amount to a whole heap of JSON parsing, and the list is only going to get longer and longer. That's going to have a growing impact on the memory and processing time of my scripts, not to mention the log storage (as each API call response appears to be logged).

Instead, I would like to call an API method which returns the given object by name (I believe name is unique within Otter).

I've already tried to see if an undocumented RESTful or RPC-like method already exists, but both...

• /api/infrastructure/servers/SERVERNAME
• /api/infrastructure/servers/get/SERVERNAME

...return HTTP/400 Invalid action type (I have also tried RPC-like methods search and find, with the same result). I had hoped that one of them might return HTTP/200 and the JSON of the single found object; or HTTP/404 if it did not exist.

Can I please raise the request that something along these lines be considered for the recommended API?

Or if it already exists, could the documentation be updated to include it?

@atripp said in Apply-Template adding unexpected CR newline chars:

In any case, we should probably switch to an enumeration like template operat has (TemplateNewLineMode) -- but a fourth option to the both (None).

Creating a None value may be risky. It's just that None is such a "default-sounding" word, that someone might one day try and make it the default, instead of Auto, and everyone would have to update scripts that worked before.

For Create-File, I picked a bool to indicate that you can either attempt to transform the newlines file or not, and named it so that the default (i.e. false) would keep the current behaviour. That way, it becomes an opt-in parameter so, if anyone is relying on the way Create-File currently handles newlines, they shouldn't have to change their scripts to retain the old behaviour.

If you standardise the logic for both Apply-Template and Create-File to both support a NewLines property in the same way, having the default value remain Auto would still mean that the current behaviour is retained without changing the scripts.

Perhaps Binary would indicate both a non-default status, and that the source isn't transformed...

(Of course, the term Auto might itself lead someone to expect that binary/non-text source files would not be subject to line-ending alteration -- I don't know how you get around that. Naming things is hard )

@atripp
Thanks for looking.

Without wanting to hold you to a particular date, what is the typical release timeframe?

I just need to work out whether to work around this issue for now, or wait it out until the next release...

@atripp

It didn't seem right to me either, so I assumed I was missing something obvious.

Am I right in saying that for server is only expected to work for Custom server targeting, though?

And for server can take a scalar variable argument, such as...

for server localhost {
	Log-Debug "some arbitrary stuff on the Otter server"

	set $serverIWantToConfigure = "TheServerIChoseInMyProperty";

	for server $serverIWantToConfigure {
		Log-Debug "some arbitrary stuff on the server I want to configure"
	}
}

...?

@atripp

Here's the Create-File code, where there seems to be another replacement:
https://github.com/Inedo/inedox-inedocore/blob/master/InedoCore/InedoExtension/Operations/Files/CreateFileOperation.cs#L77

That appears to use the same Regex (or close-enough) as I have proposed for Apply-Template above, but it always replaces with fileOps.NewLine, which I assume is server-specific.

If someone genuinely needed to create a file which used a different line-ending than the platform's default, that logic falls short.

Perhaps, instead, you could ratify an additional property which controls this:

+++
[ScriptAlias("RawMode")]
[Description("If true, does not attempt to rewrite newlines")]
public bool RawMode { get; set; }

...then alter the line to gate that newline rewrite behind that property:

---
var text = Regex.Replace(this.Text ?? string.Empty, @"\r?\n", fileOps.NewLine);

+++
var text = this.Text ?? string.Empty;
// OPT: reuse the interned regex from Apply-Template above?
if (!this.RawMode) text = Regex.Replace(text, @"\r?\n", fileOps.NewLine);

You are then doing a further translation, further down at #L92, where you use StreamWriter to explicitly set the newline for Linux operations.

If fileOps.NewLine is already abstracting this, it may not actually be necessary but, in any case, I don't think it does what you think it will, because you only call writer.WriteAsync(text) and not writer.WriteLineAsync(text).

I think the StreamWriter.NewLine property only applies when WriteLine/WriteLineAsync are called, and only to indicate the char added to the end of the string in each call -- it doesn't replace existing newlines in the supplied string.

I suspect, therefore, the line at L92 can therefore become:

    using var stream = await linuxFileOps.OpenFileAsync(path, FileMode.Create, FileAccess.Write, Extensions.PosixFileMode.FromDecimal(mode.Value).OctalValue);
--- using var writer = new StreamWriter(stream, InedoLib.UTF8Encoding) { NewLine = linuxFileOps.NewLine };
+++ using var writer = new StreamWriter(stream, InedoLib.UTF8Encoding);
    await writer.WriteAsync(text);

@atripp

I wonder if this is the issue in Apply-Template?
https://github.com/Inedo/inedox-inedocore/blob/master/InedoCore/InedoExtension/Operations/General/ApplyTemplateOperation.cs#L90

At first glance, it would certainly appear to be that line which creates the \r characters I am seeing.

I'm guessing the original string must have \r\n in it or something.

I assume that >>swim strings>> which contain newlines persist them as \r\n, but that might be platform-specific. If I can therefore also assume that the only supported platforms are Windows and Linux (so \r\n or \n), then it might be enough to replace that line with:

---
if (this.NewLineMode == TemplateNewLineMode.Windows)
    result = result.Replace("\n", "\r\n");

+++
string targetNewline;
switch (this.NewLineMode)
{
	case TemplateNewLineMode.Windows:
		targetNewline = "\r\n";
		break;
	case TemplateNewLineMode.Linux:
		targetNewline = "\n";
		break;
	case TemplateNewLineMode.Auto:
		auto:       // jump here after warning, if not handled
		targetNewline = Environment.NewLine;  // should this be fileOps.NewLine...?
		break;
	default:
		this.LogWarning($"unsupported NewLine value '{this.NewLineMode}'; Auto assumed");
		goto auto;  // jump to Auto case
    }

// TODO: intern this in a singleton utility class?
var newlineSearcher = new Regex(@"
	(?>             # atomic capture and discard matched
	    \r?         # optional \r char
	    \n)         # definitive \n char
	", RegexOptions.Multiline
	 | RegexOptions.Compiled 
	 | RegexOptions.IgnorePatternWhitespace);

result = newlineSearcher.Replace(result, targetNewline);

There is another replacement happening at https://github.com/Inedo/inedox-inedocore/blob/master/InedoCore/InedoExtension/Operations/General/ApplyTemplateOperation.cs#L97 which I don't think would be needed, if the above is applied...

---
var fileOps = await context.Agent.GetServiceAsync<IFileOperationsExecuter>().ConfigureAwait(false);
if (this.NewLineMode == TemplateNewLineMode.Auto)
    result = result.Replace("\n", fileOps.NewLine);

...but it may be appropriate to hoist the instantiation of fileOps and use fileOps.NewLine in place of Envrionment.NewLine in the switch above (I assume that fileOps.NewLine uses the platform of the current for server, rather than the platform on which Otter is running).

@atripp said in Basic arithmetic in OtterScript:

You're right, you would have to switch to localhost; however, you wouldn't have to "switch back"

But anytime I try to switch anywhere with for server, I am met with: Server context switching is not allowed on this plan execution.

I can only get for server to work if I run an ad-hoc job with Custom server targeting, but if I use ad-hoc jobs, I can't prompt for variables.

I can only prompt for variables with a job template, but the option for Custom server targeting is not available in job templates.

So switching is indeed a pain, regardless of direction.

I am experiencing an unexpected bug in the following minimum-repro OtterScript:

set $literal = >-@>
    this is line 1
    this is line 2
    this is line 3
>-@>;

Apply-Template(
    Literal: $literal,
    OutputFile: $PathCombine($SpecialWindowsPath(CommonApplicationData), none.txt)
);

Apply-Template(
    Literal: $literal,
    NewLines: Auto,
    OutputFile: $PathCombine($SpecialWindowsPath(CommonApplicationData), auto.txt)
);

Apply-Template(
    Literal: $literal,
    NewLines: Windows,
    OutputFile: $PathCombine($SpecialWindowsPath(CommonApplicationData), windows.txt)
);

Apply-Template(
    Literal: $literal,
    NewLines: Linux,
    OutputFile: $PathCombine($SpecialWindowsPath(CommonApplicationData), linux.txt)
);

When run against localhost (a Windows server), the resulting files appear to be created with additonal \r chars in between each line:

I was expecting that, for NewLines: Windows, the byte-sequence for line-endings would be \r\n and for NewLines: Linux, the byte-sequence would be \n. Instead, I appear to have \r\r\n and \r\n.

(I assume not specifying is the same as Auto, and matches the Windows behaviour because of the target server's platform. I have not tried against a Linux target server.)

Note that I am not referring to the ones at the top and bottom of the resulting files -- those are clearly because I have written newlines before and after my fish-quotes. Nor the spaces at the start of each line, those also exist in the fish-string.

The script was entered via the /scripts/edit2 text editor rather than /osve.

Have I misunderstood some nuance of the Apply-Template function, or fish-strings in general, or is this a bug?

PS: I thought I might work around this with...

Apply-Template
(
    OutputVariable => $out,
    Literal: $literal,
    NewLines: Windows
);

set $linux_out = $RegexReplace($out, "[\r]+\n", "`n");
Log-Debug $linux_out;

set $win_out = $RegexReplace($out, "[\r]+\n", "`r`n");
Log-Debug $win_out;

Create-File
(
    Name: $PathCombine($SpecialWindowsPath(CommonApplicationData), linux_pp.txt),
    Text: $linux_out,
    Overwrite: true
);

Create-File
(
    Name: $PathCombine($SpecialWindowsPath(CommonApplicationData), windows_pp.txt),
    Text: $win_out,
    Overwrite: true
);

...but it appears that Create-File has its own nuance, and all were turned into \r\n newlines. I assume, therefore, it is not possible to use Create-File to create a raw file containing the exact content of a variable, with no pre-processing?

Version 2022.10 (Build 1)