Symbol Server is returning an old PDB

Hi,

It looks like old PDB files are being served by the symbolserver preventing us from stepping into the source code whilst debugging.

The 'Load Symbols Information' Log shows the PDB being retrieved for a dll with guid 8CDA77C3E36F4FB4B433026AE4021CB0 and age 17 correctly:

C:\Workspaces\XXX\XXX\Source\AppServer\bin\Debug\XXX.Performance.pdb: Cannot find or open the PDB file.
C:\Workspaces\XXX\XXX\Source\AppServer\bin\Debug\XXX.Performance.pdb: Cannot find or open the PDB file.
c:\Workspaces\XXX\XXX\Source\Performance\obj\Release\XXX.Performance.pdb: Cannot find or open the PDB file.
C:\WINDOWS\XXX.Performance.pdb: Cannot find or open the PDB file.
C:\WINDOWS\symbols\dll\XXX.Performance.pdb: Cannot find or open the PDB file.
C:\WINDOWS\dll\XXX.Performance.pdb: Cannot find or open the PDB file.
C:\symbols\XXX.Performance.pdb\8cda77c3e36f4fb4b433026ae4021cb011\XXX.Performance.pdb: Cannot find or open the PDB file.
 **SYMSRV:  XXX.Performance.pdb from http://packages.ops.XXX.com/symbols/Release: 83456 bytes** 
**http://packages.ops.XXX.com/symbols/Release: Symbols downloaded from symbol server.**
C:\symbols\XXX.Performance.pdb\8CDA77C3E36F4FB4B433026AE4021CB011\XXX.Performance.pdb: PDB does not match image.

Using the chkmatch tools to compare headers we see that the served PDB has an age of 3, not 17 as expected:

C:\Workspaces\XXX\XXX\Source\AppServer\bin\Debug>chkmatch -c XXX.Performance.dll C:\symbols\XXX.Performance.pdb\8CDA77C3E36F4FB4B433026AE4021CB011\XXX.Performance.pdb
ChkMatch - version 1.0
Copyright (C) 2004 Oleg Starodumov
http://www.debuginfo.com/


Executable: XXX.Performance.dll
Debug info file: C:\symbols\XXX.Performance.pdb\8CDA77C3E36F4FB4B433026AE4021CB011\XXX.Performance.pdb

Executable:
TimeDateStamp: 568b8a87
Debug info: 2 ( CodeView )
TimeStamp: 568b8a87  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 284  RVA: 00005810  FileOffset: 00003a10
CodeView format: RSDS
Signature: {8cda77c3-e36f-4fb4-b433-026ae4021cb0}  Age: 17
PdbFile: c:\Workspaces\XXX\XXX\Source\Performance\obj\Release\XXX.Performance.pdb

Debug information file:
Format: PDB 7.00
Signature: {8cda77c3-e36f-4fb4-b433-026ae4021cb0}  Age: 3

Result: Unmatched (reason: Age mismatch)

The same check comparing the PDB and DLL within the symbols nupkg (downloaded from the same ProGet server) shows that they are correctly matched:

C:\Users\XXX\Downloads\XXX.Performance.1.0.9\lib\net40>chkmatch -c XXX.Performance.dll XXX.Performance.pdb
ChkMatch - version 1.0
Copyright (C) 2004 Oleg Starodumov
http://www.debuginfo.com/


Executable: XXX.Performance.dll
Debug info file: XXX.Performance.pdb

Executable:
TimeDateStamp: 568b8a87
Debug info: 2 ( CodeView )
TimeStamp: 568b8a87  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 284  RVA: 00005810  FileOffset: 00003a10
CodeView format: RSDS
Signature: {8cda77c3-e36f-4fb4-b433-026ae4021cb0}  Age: 17
PdbFile: c:\Workspaces\XXX\XXX\Source\Performance\obj\Release\XXX.Performance.pdb

Debug information file:
Format: PDB 7.00
Signature: {8cda77c3-e36f-4fb4-b433-026ae4021cb0}  Age: 17

Result: Matched

Do you have any suggestions on what to do next?

Thanks,

Martin

Product: ProGet
Version: 3.8.6

ProGet only alters the pdb/symbol files when requested via the /symbols API endpoint, and the only alteration is to inject SRCSTREAM headers to point to ProGet's source server. The symbol files are always read straight from the corresponding package on disk, and there is no caching of these requests.

This is an arcane technology, so it's not easy to debug.

You can see the ProGet's symbol file index by clicking on the [view] button next to "Symbols and source files are available" on the corresponding package. Those strings (in your case, 8CDA...CB011) are embedded in the DLL, and is what visual studio uses to request. If there's a match, then ProGet sends back the matching symbol file, altering as described above.

If you're getting symbol mismatches, most likely it's because the package and DLL don't match. But it's hard to say. ProGet's simply returning whatever symbol was uploaded.

Thanks for the prompt reply Alana.

VS is indeed requesting the correct string, but inspecting the inserted text in the returned PDB:

PGSERVER=http://packages.ops.xxx.com/source-files
PGFEED=Release
PGPKGID=XXX.Performance
PGPKGVER=1.0.0
HTTP_EXTRACT_TARGET=%pgserver%/%pgfeed%/%pgpkgid%/%pgpkgver%/%var2%

The appropriate package version is 1.0.9 for that library.

However, digging into the previous versions it looks like the returned symbol file is for the first package (v1.0.1) containing the assembly with version 1.0.0.0.

Is it possible ProGet is picking the symbols file based on the assembly version, rather than the package containing the dll with matching Guid/Age?

Has anyone had time to take a look at this? It pretty much breaks ProGet for us if it can't tell which package version an assembly with a fixed version number is from.

To clarify the issue, the dll version is fixed at 1.0.0.0 in packages 1.0.1-1.0.9 and is packaged with the correct symbols file (matching GUID and Ages). This has been confirmed manually by downloading each version from ProGet and using the chkmatch utility. ProGet also displays the correct GUID and Age for each package in the web interface.

However, when debugging packages 1.0.2-1.0.9 ProGet only returns the symbols from 1.0.1.

I looked through the code, but nothing seems to be jumping out. It's working as described, just looking for a matching guid and age based on the signature.

Unfortunately, this is not an easy problem to debug on our end... since we can't reproduce it, and no one else has reported it. Are you able to use reflector or something to inspect the SymbolserverHandler? That should at least walk through the precise search logic, and something might jump out.

It looks like the problem is in the Symbols_GetSymbols/NuGetPackageSymbols_GetSymbols stored proc:

CREATE PROCEDURE [Symbols_GetSymbols]
(
    @Feed_Id INT,
    @Symbol_Id BINARY(16),
    @SymbolFileName_Text NVARCHAR(255),
    @Symbol_Age INT
)
AS
BEGIN

SET NOCOUNT ON

    SELECT *
      FROM [Symbols]
     WHERE [Feed_Id] = @Feed_Id
           AND [Symbol_Id] = @Symbol_Id
           AND [SymbolFileName_Text] = @SymbolFileName_Text
    ORDER BY ABS(@Symbol_Age - [Symbol_Age]) DESC, [Symbol_Age] DESC
END

This is returning the worst age match first with the code taking the first returned row without checking if the age matches.

To get the correct behaviour, you need to alter the last line to:

    ORDER BY ABS(@Symbol_Age - [Symbol_Age]) ASC, [Symbol_Age] DESC

Wow, great find :)

This will be addressed in the next maintenance version.