1. Home
  2. Categories
  3. Support
  4. [BM] Error checking out code on Linux server

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

[BM] Error checking out code on Linux server

  • P

    I'm getting the exception below. I'm brand new to BuildMaster. I have a build pipeline that just does a git checkout on a remote linux server and nothing else. The server connection is good, and in fact, running the build does create the /tmp/buildmaster-agent folder. Not the /tmp/Inedo folder however. 7944ac1a-0401-4852-ab04-791a79968791-image.png

    Include Debug
    Using SshAgent agent on ########
    Unhandled exception: System.InvalidOperationException: Error initializing remote process.
    at Inedo.Agents.Hosting.RemoteHostedAgentProcess.SendMessageAsync(AgentCommand command, CancellationToken cancellationToken)
    at Inedo.Extensibility.Operations.RemoteExecuteOperation.ExecuteAsync(IOperationExecutionContext context)
    at Inedo.BuildMaster.Windows.ServiceApplication.Executions.PlanExecuter.OtterScriptExecuter.Inedo.ExecutionEngine.Executer.IExecutionHostEnvironment.ExecuteActionAsync(ActionStatement actionStatement, IExecuterContext context)
    Cleaning up...
    Deleting /tmp/buildmaster-agent/_E241 on #########...
    /tmp/buildmaster-agent/_E241 on ######### deleted.
    Cleanup complete.

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @phwhite_9282,

    This is a low-level error, and could mean a number of different things on the SSH side.

    Can you let us know:

    1. What version of BuildMaster are you using?
    2. What is the operating system of the linux server?

    Thanks,
    Alana

    0
  • P

    We're on 23.0.2. The linux server is running RHEL8. It seems like the SSH connection is successful, since I do see the /tmp/buildmaster-agent folder is being created.

    0
  • P

    On a whim, I changed the tmp path to /home/buildmaster/tmp/buildmaster-agent and I get what looks like a git error instead.
    Not sure about the original error, I would have said it was a permissions error, but the 'buildmaster' user has write permissions to the /tmp folder.

    Using SshAgent agent on #######
    Updating local repository for https://svc_buildmaster@bitbucket.coh.org/scm/riapps/foo.git...
    Repository path is /home/buildmaster/tmp/buildmaster-agent/.gitrepos/bitbucket.coh.org$scm$riapps$foo.git
    Repository does not exist or is not valid.
    Cloning from https://svc_buildmaster@bitbucket.coh.org/scm/riapps/foo.git...
    Unhandled exception: System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library '/tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so' or one of its dependencies.
    In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: /tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so: failed to map segment from shared object at System.Runtime.InteropServices.NativeLibrary.LoadFromPath(String libraryName, Boolean throwOnError) at System.Runtime.InteropServices.NativeLibrary.Load(String libraryPath) at System.Runtime.Loader.AssemblyLoadContext.LoadUnmanagedDllFromPath(String unmanagedDllPath) at Inedo.Extensibility.ExtensionLoadContext.LoadUnmanagedDll(String unmanagedDllName) at System.Runtime.Loader.AssemblyLoadContext.ResolveUnmanagedDll(String unmanagedDllName, IntPtr gchManagedAssemblyLoadContext) at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwOnError) at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError) at System.Runtime.InteropServices.NativeLibrary.TryLoad(String libraryName, Assembly assembly, Nullable`1 searchPath, IntPtr& handle) at LibGit2Sharp.Core.NativeMethods.ResolveDll(String libraryName, Assembly assembly, Nullable`1 searchPath) in /_/LibGit2Sharp/Core/NativeMethods.cs:line 124 at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags) at LibGit2Sharp.Core.NativeMethods.git_libgit2_init() at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary() in /_/LibGit2Sharp/Core/NativeMethods.cs:line 148 at LibGit2Sharp.Core.NativeMethods..cctor() in /_/LibGit2Sharp/Core/NativeMethods.cs:line 54 --- End of inner exception stack trace --- at LibGit2Sharp.Core.NativeMethods.git_clone(git_repository*& repo, String origin_url, FilePath workdir_path, GitCloneOptions& opts) at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts) in /_/LibGit2Sharp/Core/Proxy.cs:line 278 at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options) in /_/LibGit2Sharp/Repository.cs:line 810 at Inedo.Extensions.Git.RepoMan.FetchOrCloneAsync(RepoManConfig config, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E412252\Src\Git\InedoExtension\RepoMan.cs:line 89 at Inedo.Extensions.Git.Operations.CanonicalGitOperation.FetchOrCloneAsync(IRemoteOperationExecutionContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E412252\Src\Git\InedoExtension\Operations\CanonicalGitOperation.cs:line 123 at Inedo.Extensions.Git.Operations.CheckoutCodeOperation.RemoteExecuteAsync(IRemoteOperationExecutionContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E412252\Src\Git\InedoExtension\Operations\CheckoutCodeOperation.cs:line 61 at Inedo.Extensibility.Operations.RemoteExecuteOperation.RemoteExecuteJob.ExecuteAsync(CancellationToken cancellationToken) at Inedo.Agents.AgentCommand`1.Inedo.Agents.IAgentCommandWithResponse.ExecuteAsync(Stream responseStream) at Inedo.Agents.AgentCommandDispatcher.ExecuteCommandAsync(AgentCommand command, IClientConnection connection)Cleaning up...
    Deleting /home/buildmaster/tmp/buildmaster-agent/_E243 on #####
    .../home/buildmaster/tmp/buildmaster-agent/_E243 on ###### deleted.Cleanup complete.

    0
  • P

    Just upgraded to 23.0.9, same issue

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @phwhite_9282 ,

    We're a bit baffled by this one, and think the issue is related to "some kind of configuration" on your RHEL server and the /tmp folder. Likely, security related. For testing purposes, can you make the BuildMaster user account (i.e. what you SSH in as) a root user?

    Here's some more context... to help troubleshoot this further.

    The first error was clearly related to some "strange" SSHD rejection (problem initializing a process), but changing the agent's temp path fixed it. We don't know what that error is specifically... but behind the scenes, BuildMaster starts by transferring executable files to the Agent's Temp directory (which clearly succeeded) and then executing those files (which failed).

    The second error is an error loading a file from disk ('/tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so). This usually means the file isn't on disk -- do you see that file on disk? The agent would have unpacked it there.

    Our working theory is that some kind of security configuration is quarantining things that are written to /tmp and preventing execution. Which is unusual, because that's a common place where basically everything uses as a temp space.

    Thanks,
    Alana

    0 P 2 Replies
  • P

    @atripp
    I think you are right. Yes, the file exists. I saw it didn't have execution permissions so I gave it that but it still fails.
    I tried running 'ldd' on it but that failed too: "ldd: warning: you do not have execution permission for `/tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so'
    not a dynamic executable"
    However if I copy it to another location I can run ldd on it.

    Can I change the location of the /tmp/Inedo/ folder? I'll reach out to one of our Linux admins too, to see if it can be resolved on our side.

    0
  • P

    @atripp
    So /tmp is being mounted with 'noexec'. I might be able to get our security department to grant an exemption, but it'll probably take a long time.
    Anyway to change where buildmaster stores its libraries?

    Edit: I was thinking, I can probably get an exemption for one machine, but if we start using buildmaster in a bigger way, we'll want to be deploying apps to a number of different machines and I don't think I will be able to get exemptions for all of them. If it's not possible to change the path currently, then that would be an important feature request for us.

    0
  • P

    Also I just tried doing the same thing with another server, a Centos7 server, that we had been able to successfully build on and deploy to from an old version of BuildMaster (4.9.10) in the recent past. This server does not have noexec mount at /tmp.
    I configured it with the same creds and a similar tmp directory as the old buildmaster was using, but I am getting the same original error above:

    Unhandled exception: System.InvalidOperationException: Error initializing remote process.
    at Inedo.Agents.Hosting.RemoteHostedAgentProcess.SendMessageAsync(AgentCommand command, CancellationToken cancellationToken)
    at Inedo.Extensibility.Operations.RemoteExecuteOperation.ExecuteAsync(IOperationExecutionContext context)

    I suppose we are looking at 2 separate issues. I'd like to resolve both, but the noexec /tmp issue is the more important one for me.

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @PhilipWhite ,

    Thanks for confirming that; after researching this, it seems that mounting /tmp with noexec is "old-school security measure" to prevent issues that have since been fixed, but is not an uncommon practice today. So we will make a code change to have Use Agent Temp folder instead of the system temporary directory for extension files extraction - this will be implemented via BM-3927 , hopefully in the next maintenance release (Tomorrow).

    As for the Centos7 server, the underlying error message would be buried in the sshd logs. My guess is something with noexec, but who knows?

    The Linux integration in BuildMaster v4 and BuildMaster 2022+ are a bit different:

    • BuildMaster v4 just issues SSH commands and transfers files
    • BuildMaster 2022+ transfers an agent executable, then uses SSH to instruct the agent to run the commands

    So this added complexity seems to be causing some issues, but we'll figure it out!

    FYI: eventually we intend to publish a Linux-version of the Inedo Agent, primarily to make it easier to configure than SSH.

    Best,
    Alana

    0 P 1 Reply
  • P

    @atripp
    Thanks, I installed 23.0.10 this morning and the git checkout task succeeds on the RedHat server. Next step is getting the docker build task to succeed. I may open a separate post for that.

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation