I'm sorry to hear you've had a rough experience. I've never tried to load hundreds or thousands of clips for a project (maybe a hundred or so, max) but I'm not surprised that PD has difficulty with that many clips.
If you're willing to change your workflow a little, I'm confident that you'll be able to do everything you want to accomplish in PD. If you're so frustrated that you need to move on, that's completely understandable.
While you (and/or PD) are having trouble in a couple different areas, there's actually a single solution/workaround that will address all of these issues (assuming that you want to continue with PD), and the answer is simply: Don't try to do your entire gigantic, complex project at once.
The first step would be to mentally break your project into sections, maybe like chapters or maybe like scenes, whatever makes sense to you. I can't imagine that you'd need more than a few dozen clips for each section, so pick one to start with, create a new project, and then load all of the related clips into the Media Library with the shadow files checkbox enabled.
While PD is creating the shadow files, you'll see a yellow tag in the lower right corner of each clip in the Media Library (unless you've set it to display Details. For this step, you should set it to displaying thumbnails of any size). Once every clip shows a green tag, you're ready to go.
You can actually do all kinds of basic editing during the conversion process, so feel free to place the clips on the timeline and maybe even trim them upfront.
PD actually
does convert all clips in the media library, regardless of whether they're ever put on the timeinle or not. Likewise, removing them from the timeline doesn't have any impact on keeping or deleting the shadow files. Since PD couldn't process all the files you'd imported due to running out of disk space, I think you saw a mix of files with and without shadow files, some of which were on the timeline.
By default, PD will keep shadow files for 30 days, but in your case you'd definitely want to manually delete them as you complete each section to make room for the next round. Once you've got each section ready, produce the clip, then create a new project for the next section and continue.
If you're willing to focus on one section at a time, you won't run into any programmatical or storage limitations, and you'll end up with a series of fully produced sections that you can bring together for your final production.
You can also combine produced sections at any point, like combine the first ten and produce them to a bigger pre-final clip. You could add each newly produced section to that video as you go along, or produce any number of sections in whatever sequence you see fit. The final step would simply be to load all your produced sectional clips onto the timeline, add whatever transitions you want between them, then produce it.
Although it may seem like it's more work to create dozens of sectional projects, I think it actually gives you more control and also more time to think about what you want each section to convey to your audience. Overall I think it's a great exercise in patience, long-term focus and goal setting, but the main thing is that PD will be able to do everything you want under these conditions.
Hopefully this approach will work for you, and I wish you well in this endeavor, no matter how you decide to move forward.
YouTube/optodata
DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors
Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°