Unfortunately, when symbol serving doesn't work, it can be a pain to diagnose...
Can you "start from scratch", documenting your steps along the way, so that I can try to reproduce exactly what you're doing?
Start by making a very simple, hello world sort of library (maybe one class, with some basic code you can easily step/through and debug).
After that, then create some brand new feed (it sounds like you want two feeds? a symbols and a package feed?), then configure the new feed in Visual Studio. Then follow the other steps, like seeing if you can find the symbols in Visual Studios, etc.
If you can share the exact steps you did, and the package you upload, then I can reproduce the error you're seeing by following steps using the package.