PD15 : Slow Shadow File Generation

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Solved by

Jul 07, 2023 21:17

I was editing last week and PD was lagging badly. I noticed that the drive used for storage for editing was at 100% load and only a little over 100MB/s max. Everything else was very low load. I was editing 4K vids, with shadow files enabled, and it had a queue of vids to make shadow files for. With just shadow file creation, the disk was only doing 10-13MB/s.

I trudged through the rest of the project, and then replaced my old storage drive with a 2TB NVMe, figuring it would drastically improve speeds. Wrong. I've loaded a bunch of GoPro 4K footage and it's being a slug again with shadow file creation.

I've been watching the performance meters and while creating shadow files the NVMe is only at 0-1% running the same 10-13MB/s as the old drive. The CPU is only around 10-11%, RAM sticking at 19-21%, GPU barely doing anything most of the time at 0-20%, and the SSD where the shadow files are headed is only at about 1%.

So nothing is working hard at all, but shadow files are still extremely slow to create. If I produce a file, at least the CPU will goto 100% so I know PD is capable of putting more load on the system. Why doesn't it utilize more resources for shadow files so they can finish quickly?

I edit large amounts of video... hours and hours of GoPro footage, sometimes with 4 simultaneous cameras so 4 stacked 4K files synched up in the editor. I need shadow files for that so it would be nice if this could be improved. I searched and found people with newer versions of PD with issues with shadow files and recently even some saying that it stops creating them after a couple of files with 365. One person said going back to PD14 that they had as well worked better in that case. So I don't think not upgrading is the issue.

TIA.

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Jul 09, 2023 09:26

Quote Why doesn't it utilize more resources for shadow files so they can finish quickly?

Shadow file creation in PD is a CPU only encoding task and CL has elected to heavily throttle the process so it doesn't use so many resources and make the machine slow to respond, I guess perhaps to aid editing during the generation.

A 2TB NVMe will do nothing to accelerate as it's not a R/W capacity issue.

You could always entertain the manual approach, https://forum.cyberlink.com/forum/posts/list/96844.page#post_box_389513

Jeff

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Jul 09, 2023 12:23

Another option would be to instead convert your source clips to an intermediate codec like MagicYUV, especially since your new SSD will be able to handle the high bitstream rate from multiple 4K clips at once. See this discussion for more info.

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Jul 09, 2023 13:19

I'm aware of the manual method. It's a shame there's no option to allow shadow file creation as a priority rather than a background process.

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Jul 09, 2023 13:48

Quote I'm aware of the manual method. It's a shame there's no option to allow shadow file creation as a priority rather than a background process.

Yep, when CL and the forum used to solicit feedback on releases, a user pref option suggestion for shadow file generation was provided here https://forum.cyberlink.com/forum/posts/list/46732.page#post_box_242512 . That was 2015, nearly 8 years later and users are still hoping for the same. Others are just happy when it works.

Jeff

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Jul 09, 2023 14:27

Quote

Yep, when CL and the forum used to solicit feedback on releases, a user pref option suggestion for shadow file generation was provided here https://forum.cyberlink.com/forum/posts/list/46732.page#post_box_242512 . That was 2015, nearly 8 years later and users are still hoping for the same. Others are just happy when it works.

Jeff

I've used PD for a long time now and bought a few versions. I've been thinking about upgrading, but nothing very important to me seems to have changed much since my old version. I'm trying to hang in there and hope they'll fix stuff like this... but I'm also looking around. I'd really like to stick with what I have so many hours in front of, but not if many problems of the past are still problems.

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Jul 09, 2023 15:38

Quote I've used PD for a long time now and bought a few versions. I've been thinking about upgrading, but nothing very important to me seems to have changed much since my old version. I'm trying to hang in there and hope they'll fix stuff like this... but I'm also looking around. I'd really like to stick with what I have so many hours in front of, but not if many problems of the past are still problems.

Concerning shadow file generation, they have changed a few things since PD15 like:
1) Can select 3 different framesizes for shadowfiles, 720, 1280, 1920.
2) Changed from MPEG files to MP4 with H.264 codec
3) For same 720x480 shadow file framesize of PD15, they reduced the bitrate nearly in half, ~8kbps to ~4kbps when going to H.264

Bottom line though, the produce time to create the shadow file is the same, PD15 to PD21, so don't upgrade for an improvement there. On my platform, I can encode the same shadow file specs in 1/3 the time using my hardware so it's still a very underwhelming process. Just a guess, but they have so many issues with hardware decoding and encoding, maybe the reason on using CPU only for shadow file generation as its most robust in their software.

Jeff

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Jul 09, 2023 15:50

Quote

Concerning shadow file generation, they have changed a few things since PD15 like:
1) Can select 3 different framesizes for shadowfiles, 720, 1280, 1920.
2) Changed from MPEG files to MP4 with H.264 codec
3) For same 720x480 shadow file framesize of PD15, they reduced the bitrate nearly in half, ~8kbps to ~4kbps when going to H.264

Bottom line though, the produce time to create the shadow file is the same, PD15 to PD21, so don't upgrade for an improvement there. On my platform, I can encode the same shadow file specs in 1/3 the time using my hardware so it's still a very underwhelming process. Just a guess, but they have so many issues with hardware decoding and encoding, maybe the reason on using CPU only for shadow file generation as its most robust in their software.

Jeff

Thanks for the info. Even at CPU only, I've got 90% left if it would just use all of the processing power.

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Jul 09, 2023 16:09

Quote Thanks for the info. Even at CPU only, I've got 90% left if it would just use all of the processing power.

Probably not really. You probably have Hyper threading if Intel or SMT if AMD activated and PD simply can't effectively use those as a real core so 90% left rather inflated.

As a simply test, drop your 4K source in the timeline with shadow files off, Pref > Hardware Acceleration both off and produce to MPEG2, 720x480 profile with CPU and see how long it takes. My guess, same duration as your shadow file creation time if using the same shadow file specs for the same 4K source file. PD simply has too many internal latencies to 100% utilize hardware on these lower end profiles.

My guess, in your case, PD simply can't CPU decode your 4k source rapidly and that's the throttle.

Jeff

This message was edited 1 time. Last update was at Jul 09. 2023 16:11

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Jul 09, 2023 16:19

Quote

Probably not really. You probably have Hyper threading if Intel or SMT if AMD activated and PD simply can't effectively use those as a real core so 90% left rather inflated.

As a simply test, drop your 4K source in the timeline with shadow files off, Pref > Hardware Acceleration both off and produce to MPEG2, 720x480 profile with CPU and see how long it takes. My guess, same duration as your shadow file creation time if using the same shadow file specs for the same 4K source file. PD simply has too many internal latencies to 100% utilize hardware on these lower end profiles.

My guess, in your case, PD simply can't CPU decode your 4k source rapidly and that's the throttle.

Jeff

You're right. It didn't go to 100% on that profile. Only 20-21%, so 10% more. What's more annoying is that it took under a minute for a 9 minute 4K30 file to that format with the GPU working. My old 1050Ti was at almost 100%. If that could just happen normally, even in worst case scenarios with lots of files I'd have all of my shadow files auto-generated in under an hour vs all day. It's right there... if we would just use it.

I never realized how much different some formats were. Hardly any GPU use on MP4 for me and 100% CPU.

This message was edited 1 time. Last update was at Jul 09. 2023 16:20

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Jul 09, 2023 16:56

Quote You're right. It didn't go to 100% on that profile. Only 20-21%, so 10% more. What's more annoying is that it took under a minute for a 9 minute 4K30 file to that format with the GPU working. My old 1050Ti was at almost 100%. If that could just happen normally, even in worst case scenarios with lots of files I'd have all of my shadow files auto-generated in under an hour vs all day. It's right there... if we would just use it.

That's the benefit of doing it manually, which you said you knew about. The option 2 suggested in 2015 thread, ability to use hardware to speed things up. But as referenced, PD21 offers nothing new here so a workaround is all that can be suggested. The workarounds can improve throughput substantially though.

Quote I never realized how much different some formats were. Hardly any GPU use on MP4 for me and 100% CPU.

It's not MP4 that creates your observation, for instance, same profile to MKV will have same observations. Load really depends what's doing the decoding and what's doing the encoding. Any modern GPU substantially more efficient at this than even high end CPU's with PD.

Jeff

justsomedude

Newbie Joined: Jan 02, 2011 16:32 Messages: 11 Offline

Jul 09, 2023 16:58

Quote

That's the benefit of doing it manually, which you said you knew about. The option 2 suggested in 2015 thread, ability to use hardware to speed things up. But as referenced, PD21 offers nothing new here so a workaround is all that can be suggested. The workarounds can improve throughput substantially though.

It's not MP4 that creates your observation, for instance, same profile to MKV will have same observations. Load really depends what's doing the decoding and what's doing the encoding. Any modern GPU substantially more efficient at this than even high end CPU's with PD.

Jeff

I may have to just give up and do it myself. Just so close to being a great auto feature. Oh well. Thanks for the help.