Nuget connector stuck in failed state ("'0x00' is an invalid start of a property name")

mayorovp_3701

Configuration: nuget feed with 2 nuget connectors. First connector is nuget.org connector, second connector is self-connector to another feed.

Periodically, all external packages become unavailable, and web interface shows warning:

An error occurred while querying connectors for this feed.

'0x00' is an invalid start of a property name. Expected a '"'. LineNumber: 0 | BytePositionInLine: 1.

Once this problem occurs, it persists until proget server is restarted. Once server is restarted, the problem disappears until it happens again.

That all looks like some cached network error, but I have no clue what URL was requested and what response was.

Diagnostic Center contains no events for nuget.org connector.
System logs contains no errors too.

atripp

Hi @mayorovp_3701,

That's a really strange error; it's basically saying that, somehow, a 0x0 character found its way into some JSON returned by the API. This character is invisible, and you'd need to use a kind of hex editor or developer tool to even see it.

I guess, in theory it could be inserted by some intermediate device (firewall, gateway, etc), but who knows at this point. I can't imagine how that could happen on either NuGet or ProGet, but that's the first place to start looking.

I suspect the server restart is unrelated; that certainly wouldn't cause a random 0x0 unless there's something really broken with the computer.

From here, you'll want to keep isolating the issue, and try to figure out which connector is "bad"

If it's NuGet.org -- the issue is most certainly a network/gateway that's doing that.
If it's ProGet -- it's likely some strange bug, where 0x0 got inserted to the Database for a connector or feed or something. We saw that during some migrations, but it's realy hard to track down.

I would just keep experimenting. If it's related to a reboot, just stop/start the service. That should be the same.

Thanks,
Alana

mayorovp_3701

@atripp Actually zero byte in position 1 looks like attempt to read UTF16-LE-encoded json as UTF8.

I would just keep experimenting.

Proget refuses to tell me what it is trying to do, I even cannot find this error in logs. This error is visible in feed UI only. What kind of experiments can I do next time?

If it's related to a reboot, just stop/start the service.

Of course, by "server" I mean "ASP.NET Core server process", not "virtual machine".

atripp

Hi @mayorovp_3701

Actually zero byte in position 1 looks like attempt to read UTF16-LE-encoded json as UTF8

Oh that's a great observation! Yeah that sounds like a reasonable explanation. But still... how could that even be possible?

It's not like ProGet is going to randomly swap an encoding like that, and it's not like NuGet is going to store .json files incorrectly.

As for experimentation, next time it happens:

remove one of the connectors from the feed to which one
navigate to the JSON endpoints of the connector in question, to see if you see the bad JSON
try to identify a pattern of behavior that causes this
watch for HTTP access logs to see if you can find the exact URL that's being accessed at the time of the connector failure (assuming it's a self-connector)
be prepared to attach a MITM proxy to ProGet (Admin > Proxy)

Thanks,
Alana