Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Debian Feed (New) connector errors

    Scheduled Pinned Locked Moved Support
    13 Posts 3 Posters 45 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      dan.brown_0128
      last edited by

      @gdivis -- We are running ProGet on Windows

      Upping the timeout doesn't seem to make a difference. The "unable to open database file" message is logged almost immediately whenever the connector healthcheck runs.

      When viewing a new Deb feed with the connector added, it only shows the message "An unexpected error occurred while listing packages: unable to open database file. Additional information has been logged in the diagnostics center." and doesn't clear (24 hours).

      Here's the stack trace that shows in the messages. Not sure what Sqlite DB it's trying to access. The main database is SqlServer

      Details:code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file
at System.Data.SQLite.SQLite3.Open(String strFilename, String vfsName, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool)
at System.Data.SQLite.SQLiteConnection.Open()
at Inedo.ProGet.Feeds.Debian2.Debian2ConnectorIndex.Open(String fileName, Debian2Connector connector)
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetIndex()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetAndUpdateIndexAsync()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.UnionAsyncIterator`1.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Union.cs:line 131
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.AsyncEnumerable.GroupedAsyncEnumerable`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/GroupBy.cs:line 1094
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.SearchPackagesAsync(String query, Int32 maxCount, Boolean includePrerelease)
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsyncInternal()
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsync()
      stevedennisS 1 Reply Last reply Reply Quote 0
      • stevedennisS Offline
        stevedennis inedo-engineer @dan.brown_0128
        last edited by

        Hi @dan-brown_0128 ,

        Thanks for the stack trace and additional information; that definitely helps pinpoint the issue.

        Debian require that the entire remote repository is downloaded (which can sometimes take more than 10 seconds on slow connections), and this repository information is cached in a disk-based SQL Lite database. And that error message is coming from SqlLite, which is implying some kind of file access issue.

        The database is stored in «Storage.DebianPackagesLibrary»\C«connector-id»\index.sqlite3", and you can find the values under Admin > Advanced Settings.

        Do you see that index.sqllite3 file on disk? Any reason that ProGet wouldn't be able to write to that directory?

        Thanks,
        Steve

        1 Reply Last reply Reply Quote 0
        • D Offline
          dan.brown_0128
          last edited by

          I checked on the file share, and the sqllite database is not found. ProGet definitely can write to the directory, and does for all other feeds.

          Just to confirm -- even if ProGet is using SqlServer for all other database operations, does the Debian feed connector require SqlLite? Our corporate policy currently prohibits the use of sqllite and that could very well be why we don't see the file on disk

          stevedennisS 1 Reply Last reply Reply Quote 0
          • stevedennisS Offline
            stevedennis inedo-engineer @dan.brown_0128
            last edited by

            Hi @dan-brown_0128,

            ProGet uses SqlLite databases as a cache for a variety of operations. I would first try to delete and the recreate the connector, to see if it makes any difference. That will create a new disk path.

            If ProGet does indeed have full control over that directory, you'd need to use some sort of file system monitoring tool like ProcMon to see what activity is happening when attempting to create/open the database file.

            It wouldn't surprise me if there is some kind of "malware" or "rootkit" installed on your server that is interfering with ProGet's operation. We see these cause all sorts of strange problems, especially when they start "quarantining" package files that users upload because they contain "dangerous executable code" like .dll files 🙄

            Best,
            Steve

            1 Reply Last reply Reply Quote 0
            • D Offline
              dan.brown_0128
              last edited by

              @stevedennis -- we were able to rule out antivirus and similar. Windows Defender shows no threats or actions, and nothing on our file servers.

              We were able to have success by switching the Storage.DebianPackagesLibrary path to a local path (E:\ProGetPackages\.debian vs \\server.corp\qa\progetpackages\.debian).

              With the local path, the connector passes heathcheck and I see remote packages in the feed. Unfortunately this will not work well for our production use since we're using a HA setup.

              I did try copying the connector folder (and sqlite file) back to the network share and changed the storage setting back to the network share. When I ran the heathcheck, the sqlite file was deleted. Looks to me that there's something happening specific to the Debian2 connector and storage paths using a network location (\\server\path notation). Is this something you can validate too?

              stevedennisS 1 Reply Last reply Reply Quote 0
              • stevedennisS Offline
                stevedennis inedo-engineer @dan.brown_0128
                last edited by

                Hi @dan-brown_0128,

                Hmm, that's strange, and I can't imagine anything in ProGet that would yield the behavior that you're describing. We've been following this pattern for a few years now and haven't had any issues like this. Behind the scenes, here is what's happening:

                1. If the index.sqllite3 file exists, then ProGet attempts to opens it
                2. If the file cannot be opened (corrupt, old schema version, etc), then it is deleted
                3. ProGet then instructs SqlLite library to open a database with the file;
                4. If the file does not exist, SQL Lite will create it

                There's really no difference between writing to a disk path or a UNC path - that's all handled at the operating system level. Few things to note...

                • A health check for a Debian2 feed simply downloads and updates the index file - i.e. it's the same thing that happens when you browse the feed the UI.
                • Connector Health Check is performed by the ProGet Service, which is a separate program than the ProGet Web application

                Cheers,
                Steve

                1 Reply Last reply Reply Quote 0
                • D Offline
                  dan.brown_0128
                  last edited by

                  @stevedennis - If we go the route of storing the sqlite DB onto a local disk on each node of a cluster, are there any risks that would come of that? Either packages being out of sync, or slowdowns?

                  We're still researching the network path on our end

                  stevedennisS 1 Reply Last reply Reply Quote 0
                  • stevedennisS Offline
                    stevedennis inedo-engineer @dan.brown_0128
                    last edited by

                    Hi @dan-brown_0128 ,

                    I'm afraid that's not really possible; the sqllite databases are stored in the package store, which needs to be shared/common across servers.

                    Cheers,
                    Steve

                    1 Reply Last reply Reply Quote 0
                    • D Offline
                      dan.brown_0128
                      last edited by

                      That's what I suspected. We'll do some more testing on our end about UNC/mounted drives and see what comes up. Are there any debug logs I can get from the ProGet service that might give more info?

                      stevedennisS 1 Reply Last reply Reply Quote 0
                      • stevedennisS Offline
                        stevedennis inedo-engineer @dan.brown_0128
                        last edited by

                        Hi @dan-brown_0128 ,

                        The error that you discovered is about all that would be logged on ProGet; you'll need to use a tool like ProcMon to monitor file system activity for the processes.

                        Best,
                        Steve

                        1 Reply Last reply Reply Quote 0
                        • D Offline
                          dan.brown_0128
                          last edited by

                          I was finally able to try this with a mounted network drive (using SysInternals), and that works

                          1 Reply Last reply Reply Quote 0

                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                          With your input, this post could be even better 💗

                          Register Login
                          • 1 / 1
                          • First post
                            Last post
                          Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation