Unable to debug into NuGet package

Hi,

When trying to debug into a NuGet package, it immediately jumps to the next line. If I try to force the debugger to step into the line of code, Visual Studio(2012) displays the No Symbols Loaded page. Under the 'Symbol load information' section, it displays PDB format is obsolete.

Any help is appreciated.

Thanks.

Product: ProGet
Version: 2.2.10

I also have this issue. I've determined that it's only a problem when the library is built on my CI server (TeamCity) and not locally. I haven't been able to figure out what the difference is, and neither has Inedo.

Interesting.

For the record, I'm seeing the issue even when the package is built on someones local environment.

I can't figure out what the difference is between the environments. Both have VS2012 installed, both have .NET 4.5.1 installed, both are using the 64-bit version of MSBuild, building the .csproj directly, with identical command lines.

Are you seeing it with all your packages or only certain ones? I can debug successfully if I exclude 'x' number of files from the project. And it's not always the same ones, different combinations work sometimes as well. Very sketchy!

Are you able to simplify your package to only a couple of files? Just curious if you would see the same behavior I do.

I've tested single-file projects, and have successfully debugged with them. On all my multiple-file projects, I've failed. I haven't tried to nail down a specific number of files or modules where it starts failing.

Hi Mark and Jim,

Just so you are aware we are actively attempting to debug this issue on our end and still have not determined the root cause.

Since there are about 5 different places all for the same issue, I will say that this is effectively the canonical one from here on out.

Tod,

Can I assume you received my various emails, including the .nupkg files (one which fails and one which succeeds)?

Yes, thank you - those are the ones we are using for debugging.

If there's anything else I can do to help, just let me know.

So far as we can tell, TeamCity must be modifying the PDB files for unit test/source capturing, or something like that. We're simply using the pdbstr.exe tool to change the indexes, which does not produce obsoleted formats when created via the compilers normally.

What about Jim, who isn't using TeamCity?

Also, I can build the project, then the package, outside of TeamCity on that machine, and the package still fails. Therefore, it isn't TeamCity, but something else on the machine that is causing the problem.

Hm, right; had forgotten that.

So... to confirm... pdbs built from two seemingly identical machines will yield different results (one obsolete, one not) when streaming the pdb from ProGet?

I think it's very unlikely this is a ProGet bug, it's probably bug in consuming this arcane format. We're doing standard symbol serving stuff:

Visual Studio requests a PDB based via hash (/feedName/symbols/HASH)
ProGet looks in database for symbol based on hash
If found, ProGet looks for corresponding NuGet package (i.e. one you uploaded) on disk
ProGet takes .pdb file from package, puts it in temp location
ProGet runs PdbStr.exe v6.2.92 to create a new pdb
ProGet returns the new pdb, which visual studio then uses

We could try embedding a different version of PDBStr, but i'm wondering if it has to do with the folder structure that's used to create the build? I wouldn't be surprised if PdbStr has a bug with long file paths or something.

When I tested a manual build on the failing machine, I had my sources in the path C:\1\package_name. Doesn't get much shorter than that.

I wonder if a case could be opened with Microsoft on the pdbstr.exe tool?

Mark,

Just to confirm...

You say it works when you build locally? Have you tried re-naming the directory before you debug into the package? By default the pdb's will be pulled directly from the directory where they were built. Another way to tell is to look in the Modules window in VS to where the symbols were loaded(symbols cache or local file system).

I'm just curious because I am able to reproduce locally. Unfortunately because of a NDA, I cannot send my dll.

Yes, it does work when I build the package locally (on a different machine). Like you said, I had to remove the original files, or VS picked them up. I confirmed it was pulling the PDB from the server after I discovered that.

It does not work when I build from the command line on the TeamCity machine (not having TC do the build).

We are experiencing the same problems. We are using Teamcity to build nuget packages that are pushed to ProGet. All packages are built with the same build configuration template in Teamcity and they all reside in the same Visual Studio solution. Interestingly though, some of the packages .pdb files are not loaded (with the message "PDB format is obsolete").

If I download a "failing" nuget package directly from Teamcity and refer to it's .pdb in Visual Studio, source code debugging works without any problems. This is confirmed for all failing pdb's, so I don't think Teamcity is the source of the problem.

I compared the different .pdb files in a text editor and noticed that the working ones begins with "Microsoft C/C++ MSF 7.00", but the failing ones looks truncated at the beginning of the file.

Are the .pdb files that are created with pdbstr.exe saved on disk before they're sent to Visual Studio?

I've also noticed that for the bad pdb files there is no, 'Microsoft C/C++ MSF 7.00' header at the top
The bad pdb's also do not contain a section like, 'SRCSRV: source files ---------------------------------------'
Using the srctool.exe -r on a bad pdb returns no results
I extracted some of Inedo's code, ran it locally and it was successful. Multiple versions of pdbstr.exe were used successfully.
Using a locally installed(IIS) version of ProGet was successful.
The only place I get the failure is with ProGet installed on Windows server 2012(IIS).

My ProGet is installed on Server 2008 R2, not Server 2012.

Thanks for the detailed debugging; this one is a giant pain to figure out, as you're noticing!

Jim -- can you try running that code you extracted on the Server 2012 that's failing? I'm wondering if pdbstr.exe is silently failing due to missing dependencies, and producing these "bad" pdb files.

Steve,

It's probably not the answer you were looking for, but it DOES work if I copy my console application to the 2012 server and run it. If I take the resulting pdb from the console app, I can copy it to my local symbol server and use it to step into the code. It correctly contacts the ProGet server and downloads the source. Also, my code uses the same pdbstr.exe that ProGet does(C:\Windows\Temp\inedopgsrcsrv\pdbstr.exe).

As noted above the bad pdb is smaller in size(KB) to the correctly generated pdb.

Thanks Jim; exactly what I was wondering.

Can you by chance send this console app and src to support - at - inedo . com? This could probably help debug this too.

I was able to reproduce the bad pdb in terms of exact byte size, by writing the stream out before the call to GetIndexedPdb is made. My guess is the Process that starts pdbstr is failing silently. Unfortunately nothing is being logged in that case. Is it possible for someone to add some logging? I would be willing to test in my environment.

Good news! We're finally able to reproduce this on our end.

It turns out the version of pdbstr.exe we've embedded (from the 2008R2/W7 SDK) has a hidden dependency on some ancient dll (mspdb60.dll). That DLL is installed with the SDK, and in some Visual Studio SKUs, and pdbstr.exe will silently fail if it's not inclused.

We're investigating further, and will ship a maintenance release to address this soon.

Just to let everyone know that's been following this thread - ProGet 2.2.12 has been released which should address all of these issues. Thanks to everyone for your help in tracking this down!

It's now working!

Thanks.

I can also confirm that it's working now. Thanks for the steady support.

It's working for us too.

Thanks!