ProGet indexing errors

We are consistently getting indexing errors when deploying to nuget:

The process cannot access the file 'XXXXXXX.1.0.1505.nupkg' because it is being used by another process.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Inedo.NuGet.Packages.NuGetPackage.ReadFromNupkgFile(String fileName)
at Inedo.ProGet.Extensibility.PackageStores.DirectoryPackageStore.TryReadPackage(String fileName)

Using process explorer, we have verified that the only program that has a lock on the file is 'ProGet.Service.Exe'

Product: ProGet
Version: 3.0.3

Is this an existing package being overwritten, or is this a brand new package that is being pushed?

The feed indexer does not prohibit other reads (i.e. FileShare.Read) so the only way I could see this happening is if the file is attempting to be overwritten at the same time it is being read by the indexer.

Unfortunately this happens even with new packages. Is there any diagnostic information that I can pull and send it to you guys?

Any updates on this? We really like this product, but this is kind of a dealbreaker for us.

We are still investigating this issue at the moment. Is there any other information you can send, perhaps Process Monitor logs that show when the file gets opened and then attempts to be opened again? Also, how often are these packages being pushed to the feed?

We've tried to reproduce this issue on our test systems by pushing and trying to install several packages at virtually the same time but have not experienced the issue.

We actually push a lot of packages - our build system creates a package for every commit which gets pushed to proget. We are a fairly busy shop, so proget probably handles more packages than other places. I'll do a process monitor log for a period of about 10 mins or so - who can I send it to?

You can send it to support@inedo.com

Thanks for the log file. Unfortunately, it looks like the indexer is behaving as normal. The log seems to indicate that IIS opens the file for write, and then the service attempts to index it right away but receives a sharing violation because the file hasn't been fully written yet (which is normal and would not cause the issue you're seeing because there is no request from IIS to read the file yet). Once written, the file is correctly closed, and then it looks like there is a read attempt by the service to index it again once the directory change notification hits (i.e. the file has been closed).

There are still no reads from IIS at this point, so no errors should ever be returned to NuGet from that sharing violation (it's also caught and handled on the service side anyway).

Do you by chance have UseIntegratedWebServer set to "true" in All Settings? It doesn't appear to be from the process monitor log, but worth checking.

Beyond that, do you have any additional information as to when this error occurs exactly? It again appears that you are downloading a file at the time of the error, but you mentioned deploying to NuGet. Is this error returned by the NuGet client when performing an upload or download operation, or something else entirely?

UseIntegratedWebServer is set to false. This is an error reported under the 'indexing errors' section of the admin console of proget. It's an issue for us because packages now take ~20 minutes to appear in proget once they're uploaded, which interferes with our automated build process.

The screenshot I sent you shows that the indexer is trying to open the file less than one millisecond after it's first created by IIS. Doesn't that indicate that the directory change notification is firing too early? This is a windows 2008 R2 environment if that helps.

That's correct in that it's too early to read the file, which is why it results in the sharing violation (the file is opened for write by w3wp.exe without FileShare.Read). The new file in the directory triggers the FileWatcher (which doesn't find the file because it's not completely written yet). However, the NotifyChangeDirectory operation in the feed directory will trigger the FileWatcher again after the file is completely written and closed in which case it would get picked up by the indexer, and that would make the package appear in the feed.

Is there a way to run the monitor again to cross-reference the times that the error appears in the ProGet error log with a violation within the Process Monitor log? I am thinking that these particular sharing violations are not related to the errors you're seeing.