Setting variables inside async blocks

Guest

I have a variable that I initialize outside of all blocks (so that it won't be scoped to a server or deployable). Then, I attempt to set it to a new value inside an async block. I know the set inside the async is working, because I can log the value. But, after the blocks are finished, the value has reverted back to the default set before the async.

Is there a way to persist the value after the async, or in other words, to have a thread safe variable?

Code:

set $ScriptError = P;

Log-Information $ScriptError;

with async
{
    set $ScriptError = X;

    Log-Information $ScriptError;
}

await;

Log-Information $ScriptError;

Output in Log:

P
X
P

What I want:

P
X
X

This is a simplified version of my code. In reality, I am running scripts on several servers simultaneously, and each server has its own variable in which to report status.

Product: BuildMaster
Version: 5.3.1

Guest

Thanks for the detailed reproduction case; I was able to reproduce the same behavior you observed. I'll post an update as soon as I hear back from the engineers about the details.

Guest

This is by design (maybe not documented though...); an async block makes a copy of all variables, and the parent/main variables are inaccessible. The reason for this is in following the "principle of least astonishment" - with OtterScript's scoping rules (especially with templates involved), it would be very challenging to debug accidental "cross-thread" variables.

Our current thoughts are to use a new keyword that would explicitly declare a variable that's shared across all async contexts. Obviously we want to be very careful in designing this, and there's a lot of questions we need to answer ourselves.... your feedback is welcome as always!

#should we also allow setting a value here, like "= something"?
global $myvar; 

# we presume this should just be an empty string, since you can't have null 
# values in OtterScript, only strings
Log-Information $myvar; 

# perhaps this should be an error; global may only be declared before 
# other statements run, otherwise what it's declared in a template, or some
# random other spot like that
global $myvar2; 

# this part is relatively easy...
set $myvar = hello1;
with async { set $myvar = hello2; }
await;

#will be "hello2"
Log-Information $myvar;

Or, perhaps, a better keyword would be shared or something.

Of course, an immediate work-around would be to write to a configuration variable or file of sorts.

Guest

Rather than creating a new keyword to create a thread-safe variable, It may be simpler to create a "critical section" block, wherein variables set would modify the original variable rather than (or in addition to) the copy created for the async block. Naturally, the critical section would be accessible by only one thread at a time.

Something like this:

set $ScriptError = P;

Log-Information $ScriptError;

foreach $i in @Range(1, $NumberOfServers)
{
    with async
    {
        #various processing here 
        
        with critical
        {
            set $ScriptError = X;
            
            Log-Information $ScriptError;
        }    
    }
}

await;

Log-Information $ScriptError;

In that example, every thread created by the loop would be able to simultaneously execute the "#various processing here" section, but each thread would have to wait its turn to access the critical block where the variable would be set. Because of the critical section, we can know that only one thread at a time is operating on the variable.

Of course, this is an academic example. The script author would have to write some logic to prevent the variable from being needlessly overwritten by each pass of the loop, using an if/else block, for example.

Introducing critical section blocks would also solve concurrency problems with file write access, or network resource access where multiple connections aren't permitted.

Guest

Great idea; this was also on our "should we put this in OtterScript v1, or just get the thing shipped" list :)

    with lock=token
    {
       ... only one can run at a time ...
    }

It wouldn't quite solve the variable thing, because then the set statement would a different operation mode inside of a certain block: update not only the local copy, but then all of the ancestor thread's variables of a same name. This behavior mode would be hidden if you're using templates, etc.

Fortunately, new keywords aren't so bad (usually documentation is the hardest part); about our only opportunity for conflict is in colliding operation names. But I think that happens in any language... if you name an operation something that might be a keyword (global or shared or new), you shouldn't be surprised.

At some point, I want to get together a doc showing the OtterScript ideas; maybe we'll just integrate it into the regular docs as "proposed" or something.

The other one that just came up, FYI, was a way to declare strings without escaping/eval. Maybe we'll do that with a function though, like $Literal($this $will $not $be $evaluated).

Guest

I can appreciate the difficulty in devising a solution. Can I look forward to seeing something in a future release?

For the problem I'm trying to solve, I'm working around this issue by sending an email from within the async block to alert me that there is an error, rather than accumulating the error status from each thread and aggregating it into one email at the end as I had originally planned.

Guest

Just a heads up, that we've updated the docs with our proposed specifications, always open to feedback!

We don't have an ETA; the changes aren't really that difficult, but once we implement them we're stuck with them.