Visual Studio adding nuget to path for source files, port hard-coded into Proget output

I have a source + symbols package and when I am attempting to step into the code, Visual Studio is not able to download the source. It makes the following requests:

<my server>/symbols/<my feed>/<dll name>.pdb/<random stuff>/<dll name>.pdb
...
<my server>:<port>/nuget/source-files/<my feed>/<package name>/<version>/<path to source>

The first gives a 200 response, the second is a 500 error. If I remove the 'nuget' part of the path and the port (i.e. <my server>/source-files/<my feed>/...) I am able to view the source file. I can fix the path with a simple URL Rewrite rule in IIS, but I don't know how to fix the port problem. The issue appears to be that ProGet is hard-coding the port into any URIs it generates. Is there some way to disable that?

Product: ProGet
Version: 3.8.4

UPDATE: By setting ProGetBaseUrl (hint came from the release notes for 3.8.5, which I have now upgraded to), Visual Studio is no longer adding the port to the request for source or adding the extra 'nuget' but it is still not getting the source file. I don't know why my URL Rewrite rules are not directing that request to the web farm I have placed the stand-alone ProGet service in.

I've got it working now, but only with the port. I'd rather not have to do that, but it will work for now.

I'd really like to know why I can't use a reverse proxy in front of ProGet to avoid needing to specify port numbers.