1. Home
  2. Categories
  3. Support
  4. Debian Feed (New) connector errors

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Debian Feed (New) connector errors

  • D

    Trying to set up the new Debian feed and connector on ProGet 2023.29

    I can create the feed and connector, but the healthcheck fails with:

    Connector debian-buster health check reported error: unable to open database file

    Connector Settings:
    Type Debian2
    Name debian-buster
    Url http://ftp.debian.org/debian/

    I have tried restarting the Service and the ISS site and app pool, but still get the same connector error. Any ideas?

    0
  • gdivis
    inedo-engineer

    Hi @dan-brown_0128,

    I would try increasing the timeout period in the connector settings to see if that helps. Are you running ProGet on Windows or Docker/Linux?

    0
  • D

    @gdivis -- We are running ProGet on Windows

    Upping the timeout doesn't seem to make a difference. The "unable to open database file" message is logged almost immediately whenever the connector healthcheck runs.

    When viewing a new Deb feed with the connector added, it only shows the message "An unexpected error occurred while listing packages: unable to open database file. Additional information has been logged in the diagnostics center." and doesn't clear (24 hours).

    Here's the stack trace that shows in the messages. Not sure what Sqlite DB it's trying to access. The main database is SqlServer

    Details:code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file
at System.Data.SQLite.SQLite3.Open(String strFilename, String vfsName, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool)
at System.Data.SQLite.SQLiteConnection.Open()
at Inedo.ProGet.Feeds.Debian2.Debian2ConnectorIndex.Open(String fileName, Debian2Connector connector)
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetIndex()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetAndUpdateIndexAsync()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.UnionAsyncIterator`1.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Union.cs:line 131
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.AsyncEnumerable.GroupedAsyncEnumerable`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/GroupBy.cs:line 1094
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.SearchPackagesAsync(String query, Int32 maxCount, Boolean includePrerelease)
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsyncInternal()
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsync()
    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @dan-brown_0128 ,

    Thanks for the stack trace and additional information; that definitely helps pinpoint the issue.

    Debian require that the entire remote repository is downloaded (which can sometimes take more than 10 seconds on slow connections), and this repository information is cached in a disk-based SQL Lite database. And that error message is coming from SqlLite, which is implying some kind of file access issue.

    The database is stored in «Storage.DebianPackagesLibrary»\C«connector-id»\index.sqlite3", and you can find the values under Admin > Advanced Settings.

    Do you see that index.sqllite3 file on disk? Any reason that ProGet wouldn't be able to write to that directory?

    Thanks,
    Steve

    0
  • D

    I checked on the file share, and the sqllite database is not found. ProGet definitely can write to the directory, and does for all other feeds.

    Just to confirm -- even if ProGet is using SqlServer for all other database operations, does the Debian feed connector require SqlLite? Our corporate policy currently prohibits the use of sqllite and that could very well be why we don't see the file on disk

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @dan-brown_0128,

    ProGet uses SqlLite databases as a cache for a variety of operations. I would first try to delete and the recreate the connector, to see if it makes any difference. That will create a new disk path.

    If ProGet does indeed have full control over that directory, you'd need to use some sort of file system monitoring tool like ProcMon to see what activity is happening when attempting to create/open the database file.

    It wouldn't surprise me if there is some kind of "malware" or "rootkit" installed on your server that is interfering with ProGet's operation. We see these cause all sorts of strange problems, especially when they start "quarantining" package files that users upload because they contain "dangerous executable code" like .dll files 🙄

    Best,
    Steve

    0
  • D

    @stevedennis -- we were able to rule out antivirus and similar. Windows Defender shows no threats or actions, and nothing on our file servers.

    We were able to have success by switching the Storage.DebianPackagesLibrary path to a local path (E:\ProGetPackages\.debian vs \\server.corp\qa\progetpackages\.debian).

    With the local path, the connector passes heathcheck and I see remote packages in the feed. Unfortunately this will not work well for our production use since we're using a HA setup.

    I did try copying the connector folder (and sqlite file) back to the network share and changed the storage setting back to the network share. When I ran the heathcheck, the sqlite file was deleted. Looks to me that there's something happening specific to the Debian2 connector and storage paths using a network location (\\server\path notation). Is this something you can validate too?

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @dan-brown_0128,

    Hmm, that's strange, and I can't imagine anything in ProGet that would yield the behavior that you're describing. We've been following this pattern for a few years now and haven't had any issues like this. Behind the scenes, here is what's happening:

    1. If the index.sqllite3 file exists, then ProGet attempts to opens it
    2. If the file cannot be opened (corrupt, old schema version, etc), then it is deleted
    3. ProGet then instructs SqlLite library to open a database with the file;
    4. If the file does not exist, SQL Lite will create it

    There's really no difference between writing to a disk path or a UNC path - that's all handled at the operating system level. Few things to note...

    • A health check for a Debian2 feed simply downloads and updates the index file - i.e. it's the same thing that happens when you browse the feed the UI.
    • Connector Health Check is performed by the ProGet Service, which is a separate program than the ProGet Web application

    Cheers,
    Steve

    0
  • D

    @stevedennis - If we go the route of storing the sqlite DB onto a local disk on each node of a cluster, are there any risks that would come of that? Either packages being out of sync, or slowdowns?

    We're still researching the network path on our end

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @dan-brown_0128 ,

    I'm afraid that's not really possible; the sqllite databases are stored in the package store, which needs to be shared/common across servers.

    Cheers,
    Steve

    0
  • D

    That's what I suspected. We'll do some more testing on our end about UNC/mounted drives and see what comes up. Are there any debug logs I can get from the ProGet service that might give more info?

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @dan-brown_0128 ,

    The error that you discovered is about all that would be logged on ProGet; you'll need to use a tool like ProcMon to monitor file system activity for the processes.

    Best,
    Steve

    0
  • D

    I was finally able to try this with a mounted network drive (using SysInternals), and that works

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation