Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Jenkins pipeline uploadProgetPackage fails on large uploads?



  • I'm having to trial-and-error, and have found this:

    if I execute uploadProgetPackage with single file (xxx.exe), it all uploads fine.
    If I execute with folder containing that EXE (binDir/), or (binDir/**/), it fails with
    java.io.IOException: Remote call on Channel to /10.5.50.53 failed

    Is there a limit to the package size being transferred from slave to master to ProGet server to ProGet DB?? Also, I cannot find any logs on the slave... where are the logs?? I've looked in the server (via web) and also do not see logs.

    Environment:

    • Jenkins Server: 2.32.3 (on Win10), Inedo Plugin 0.7<br>
    • ProGet Server: 4.7.6 build5 running on same system as Jenkins Server
      using ProGet provided MySQL DB
    • Jenkins Slave: Win10

    example success

    
    13:31:16 [x64dbg] [ProGet] Sending POST as user 'build-auto' to http://XXXMYSERVERXXX.com:81/upack/Experiment/upload
    13:31:16 [x64dbg] [ProGet] With application/zip content:
    13:31:16 [x64dbg] 	... Content of file C:\Baseline_x64dbg.unpack ...
    13:31:16 [x64dbg] 
    
    [Pipeline] [x64dbg] echo
    13:31:19 [x64dbg] upload succeeded
    

    example failure

    
    13:39:25 [x64dbg] [ProGet] Sending POST as user 'build-auto' to http://XXXMYSERVERXXX.com:81/upack/Experiment/upload
    [Pipeline] [x64dbg] error
    [Pipeline] [x64dbg] }
    [Pipeline] [x64dbg] // dir
    [Pipeline] [x64dbg] }
    [Pipeline] [x64dbg] // node
    [Pipeline] [x64dbg] }
    [Pipeline] [x64dbg] // timestamps
    [Pipeline] [x64dbg] }
    [x64dbg] Failed in branch x64dbg
    [Pipeline] // parallel
    [Pipeline] echo
    >>>>>>>>> ERROR:hudson.AbortException: java.io.IOException: Remote call on Channel to /10.5.50.53 failed
    

    I'm not sure where the IP address 10.5.50.53 is coming from - the system running both Jenkins and Proget servers is 10.1.1.180. I will be checking that with my IT department.
    In case its important, I've surrounded the Proget uploadProgetPackage with try{...} catch () {...}

    Meanwhile, can you direct me to any documentation on the plugin and the upload (examples, restrictions, etc)??
    thank you!

    Product: ProGet
    Version: 4.7.6



  • Are you able to upload this same package via upack.exe or the browser?

    There shouldn't be any upload limits...

    here are some additional links:

    https://github.com/jenkinsci/inedo-proget-plugin

    https://wiki.jenkins-ci.org/display/JENKINS/Inedo+ProGet+Plugin



  • Apologies for the lengthy time-to-respond. I've been side-tracked, but back on this now.
    Update:

    • I've been able to setup my own ProGet Server (4.7.13), with a valid 30 day license. I have it setup with one test feed. My Jenkins Server is 2.60.2, and the Indedo plugin is 0.7. I'm using pipeline scripts (groovy) only.
    • tests using Jenkins plugin and uploading a folder with logs (test files) of about 8-20Kb works.
    • When I then use same plugin, but upload folder of binaries (2.9Gb), it packs for about 2 min, and then I see
      ProGet Upload error was: java.io.IOException: Remote call on Channel to /10.5.50.98 failed
    • I've found the API documentation and see PUT or POST /upack/«feed-name»/upload«...» but it appears (the CURL example) that I must first make a ZIP archive on my own, and then issue the curl PUT using the name of that ZIP archive file. (yes?) [and I will be experimenting today..]


  • More Data:

    • after some experimentation with the Jenkins Plugin, I've discovered that it is not a "size" issue (eg 2.x Gb), but rather that I seem to only be able to specify a SINGLE folder to pack using the Jenkins Plugin syntax:
      dir(parent) { uploadProgetPackage artifacts: kidFolder+'/**/*', caseSensitive: false, excludes: '**/*.unpack', title: myTitle, description: descrip, feedName: feedID, groupName: myGroupName, packageName: myPackageName, version: '1.2.3', metadata: myMetaAry.join('\r\n') }
    • the above does not work. It fails -- I've lost the log but can re-create the error pretty easily..
    • I've setup another experiment using the Universal Package (make a zip called myTest.upack, with a upack.json inside) and issue a curl command mostly per the Inedo docs:
      curl.exe" -X PUT "http://myProGetSrvr:81/upack/myUnivFeed/upload" --upload-file "C:\Jenkins\workspace\experiment\myTest.upack" --user "****"
    • this works (sortof).
    • I see the curl transfer stats, and there does not appear to be an error, but when I echo the STDout, I only see the command (above) and nothing else.
    • when I look on the ProGet server, I see no new packages anywhere.
    • Aha!! I looked at the Event Log, and see that the package was created at 12:47 today, and modified at 5:59 today (when I ran the test script one last time)!! and indeed, I see the correct contents when I download that package.

    I think I'm better, but it sure would help to know what responses I get from a cURL -X PUT... and do I need to use PUT, or can I use POST(??)



  • even more data;

    • the curl -X PUT... works, but only for zipped files smaller then 1Gb

    as I mentioned in earlier posts, the data is usually close to 2.x Gb, so using a web-ish upload (eg curl) is problematic on many levels.

    Is there an equivalent to FTP for uploads/downloads to handle file transfers between the ProGet server and a client system?



  • Final results for curl -X PUT...

    • I was able to change C:\Program Files\ProGet\WebApp\Web.config (yes, my server is on Windows 10), and increase the transfer size from 1Gb to 3Gb
    • I then restarted the ProGet Web Service
    • I then tried to upload a zip file 2.02Gb in size..and, it worked.


Inedo Website HomeSupport HomeCode of ConductForums GuideDocumentation