Error Pushing Docker Images - Non-negative number required

[Rob]

I am experiencing errors pushing to a Proget Docker Repository. I have previously been able to push but now I get the following display when pushing from PowerShell:

PS C:\Users\rob-adm\source\repos\Lance> docker push 192.168.1.6:81/rob/test/monitor:latest
The push refers to a repository [192.168.1.6:81/rob/test/monitor]
43bc2880ee23: Preparing
8c88eee9b989: Preparing
b1a6003e4bf1: Preparing
bb26ae9bb86c: Preparing
730f4287f9b8: Preparing
9655be662841: Preparing
9bbee8f1f7db: Preparing
7a30647a9723: Preparing
1e0e918b167f: Preparing
e6b995dc63ac: Preparing
f358be10862c: Preparing
9655be662841: Waiting
9bbee8f1f7db: Waiting
7a30647a9723: Waiting
1e0e918b167f: Waiting
e6b995dc63ac: Waiting
f358be10862c: Waiting
b1a6003e4bf1: Pushed
43bc2880ee23: Pushed
730f4287f9b8: Pushed
bb26ae9bb86c: Pushed
9655be662841: Pushed
8c88eee9b989: Pushed
e6b995dc63ac: Layer already exists
7a30647a9723: Pushed
1e0e918b167f: Pushed
9bbee8f1f7db: Pushed
f358be10862c: Retrying in 5 seconds
f358be10862c: Retrying in 4 seconds
f358be10862c: Retrying in 3 seconds
f358be10862c: Retrying in 2 seconds
f358be10862c: Retrying in 1 second

docker : received unexpected HTTP status: 500 Internal Server Error
At line:1 char:1

docker push 192.168.1.6:81/rob/test/monitor:latest

CategoryInfo : NotSpecified: (received unexpe...al Server Error:String) [], RemoteException

FullyQualifiedErrorId : NativeCommandError

A look at the Proget Recent Errors logs shows two different types of errors:

Error Type 1

Logged: 11/28/2017 12:54:48 PM
Level: Error
Category: N/A
Message: Unhandled exception processing /v2/Rob/test/monitor/blobs/uploads/42b04a3d-e1c4-4528-a986-d3d0f83866df
Details: System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
at System.IO.FileStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Feeds.Docker.DockerFeed.<WriteBlobChunkAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.SimpleHandlers.Docker.DockerHandler.<ProcessBlobUploadAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.SimpleHandlers.Docker.DockerHandler.<ProcessRequestAsync>d__5.MoveNext()

Error Type 2

Logged: 11/28/2017 12:58:51 PM
Level: Error
Category: HTTP
Message: An error occurred in the web application: An error occurred while communicating with the remote host. The error code is 0x80070057.
Details: URL: http://192.168.1.6:81/administration/service
Referrer: http://192.168.1.6:81/administration
User: Admin
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299
Stack trace: at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.Stream.<>c.<BeginWriteInternal>b__46_0(Object )
at System.Threading.Tasks.Task1.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.IO.Stream.EndWrite(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.IO.SlimMemoryStream.<CopyToAsync>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<ProcessRequestAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at Inedo.Web.Handlers.AsyncHandlerWrapper.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

We are using Version 4.8.6 (Build 3)

Any assistance you could provide would be greatly appreciated.

Thank you
:)

Product: BuildMaster
Version: 5.8.2

[ben]

Hello Rob,

Sorry for the delay. Does the error occur with any Docker image that is available on Docker Hub (or, would you be willing to share your Docker image)?

Also, are you using docker-machine or something similar to run Docker on a virtual machine, or is Docker running natively on the machine with PowerShell?

[Rob]

Good Morning

Thanks for the note!

My system is stand-alone and not connected to the Internet. We have multiple images that it fails with. RabbitMQ, SQL Server, and windowsservercore images all fail.

Docker runs for us natively in PowerShell. Our platform is Windows Server 2016. Docker 17.06.

Thanks!

[ben]

This may be fixed by ILIB-19, which makes buffering of Docker layer uploads happen in a temporary file instead of in memory. It's very possible that SlimMemoryStream has a bug that only manifests when it has several gigabytes of data in it.

microsoft/windowsservercore has a 7GB foreign layer, and I assume you're using the allow-nondistributable-artifacts option on the Docker daemon because you mentioned the system is not connected to the internet, so the next version of ProGet will stop that 7GB upload from being buffered in memory.