SSH FileOperation prepending characters to file

[Andrew Sumner]

This is happening both in my custom extension and the FileTransfer extension. I'm trying to copy a shell script and it becomes unusable because of those leading characters.

If I write to a local file the prepended characters are not included, if I write to the IFileOperationsExecuter for the SSH Agent they are.

When I inspect the transfer buffer while debugging - using System.Text.Encoding.Default.GetString(buffer).Substring(0, 50) in immediate window - I get "#!/bin/sh \necho "Hello World, we've made it!"\n\0" which has prepended the characters. Some research suggests this might be length information for the stream.

If I vi the transferred file on linux I see "..." at the beginning of the file.

This is the offending bit of code, full source code can be found on GitHub.

    private void TransferFile(IFileOperationsExecuter srcFileOps, string srcFileName, IFileOperationsExecuter destFileOps, string destFileName)
    {
        this.LogInformation("Transfer {0} to {1} over SSH", srcFileName, destFileName);
    
        FileStream srcStream = null;
        Stream destStream = null;

        try
        {
            srcStream = File.Open(srcFileName, FileMode.Open, FileAccess.Read);
            //srcStream = srcFileOps.OpenFile(srcFileName, FileMode.Open, FileAccess.Read);
            destStream = destFileOps.OpenFile(destFileName, FileMode.Create, FileAccess.Write);

            const int KB_32 = 32 * 1024;
            Byte[] buffer = new Byte[KB_32]; 
            int bytesRead;

            while ((bytesRead = srcStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                destStream.Write(buffer, 0, bytesRead);
            }
        }
        finally
        {
            if (srcStream != null) srcStream.Close();
            if (destStream != null) destStream.Close();
        }
    }

Product: BuildMaster
Version: 4.6.4

[Tod]

Those symbols are the text representation of the UTF-8 byte-order-mark. I'm not sure where in the stack those could be getting added. From a quick glance, it seems you're simply copying the file bytes from one stream to another which shouldn't be using encoding at all.

Are you absolutely sure the source file does not have those 2 bytes at the beginning of the file in both cases? I'm not sure how or where the source file comes from, but I see:

string onlyFileName = Path.GetFileName(this.DestinationFileName ?? this.ArtifactName);
string srcFileName = srcFileOps.CombinePath(srcFileOps.GetBaseWorkingDirectory(), onlyFileName);

which seems to indicate it's written out in a previous action. Can you tell me how that source file is generated (i.e. with a BuildMaster action or something else?)

[Andrew Sumner]

I'm downloading the file from github: trial.sh

This test doesn't add the extra characters, only seems to happen when I use this.Context.Agent.GetService<IFileOperationsExecuter>(); on SSH Agent.

    [TestMethod]
    public void test()
    {
        FileStream srcStream = null;
        Stream destStream = null;

        try
        {
            srcStream = File.Open("c:/temp/trial.sh", FileMode.Open, FileAccess.Read);
            destStream = File.Open("c:/temp/trial2.sh", FileMode.Create, FileAccess.Write);

            Byte[] buffer = new Byte[32 * 1024];                int bytesRead;

            while ((bytesRead = srcStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                destStream.Write(buffer, 0, bytesRead);
            }
        }
        finally
        {
            if (srcStream != null) srcStream.Close();
            if (destStream != null) destStream.Close();
        }   
    }

[Tod]

That linked file has the  characters at the start as well... which is why you're seeing this behavior :)

You can't see those characters on most text editors in Windows (including Notepad, Notepad++, etc.) because they understand that is the UTF-8 BOM and will not display it.

If you save the file encoded as ANSI the BOM will be removed. You can also manually re-save the file yourself if you set the srcStream position to 3 in your test, and change the destStream to save to a different file.

[Tod]

Also for convenience, as of .NET 4.0 there is a CopyTo() method on Stream that will do the buffering for you, so you can just call:

srcStream.CopyTo(destStream);