CyberLink Community Forum
where the experts meet
| Advanced Search >
Creation of shadow files from 4K material is very slowwwww, work around available
Reply to this topic
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
It appears that PDHanomanSvr.exe is creating the shadow files.

CPU usage of that program is typically 15% rarely going to 20% and priority is automatically set to "below normal"

Setting a higher priority makes no difference.

PD13 is impratical to use with 4K material w/o the shadow files.

In some cases it has taken me 24 Hrs to create the shadow files for app 8hrs of raw material.

There should be a way to speed this up if there is no other activity in PD13 !!!



Eugene

This message was edited 3 times. Last update was at Mar 23. 2015 16:00

73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
[Post New]
Eugen, maybe unrelated, I see that you have a LGA1366 platform. To help speed up the process, you might want to go on eBay and get an i7-980X (Extreme Edition 3.33Ghz 6-Cores) - I saw one for $250.
Depending of your motherboard support you could go for a Xeon too... like a Xeon X5690 (3.46GHz, 6-Cores) for $215.

This message was edited 1 time. Last update was at Mar 22. 2015 16:36

Reply
ynotfish
Senior Contributor Private Message Location: N.S.W. Australia Joined: May 08, 2009 02:06 Messages: 9870 Offline
[Post New]
Hi Eugene -

I've come across the same problem with generating shadow files, only in testing. There was one clip - 4K H.265 HEVC - that was still not finished generating after 24 hours so I aborted the mission. There's a little process I've applied successfully that may (or may not) help you. It can be applied in many circumstances. In your case, it completely bypasses PDR's shadow file generation.

This explanation of this substitution procedure probably needs a screen capture, but I'll give it a shot:


  1. You have a working folder [F1] containing your 4K clips named (say) 000.mp4, 001.mp4, 002.mp4 etc - don't open PDR yet

  2. Use Format Factory to convert all your 4K clips - (edit) or render in PDR - to (say) 720p & place the converted files in a separate folder [F2]

  3. Rename the converted files 000.mp4, 001.mp4, 002.mp4 etc so the naming protocol exactly matches your originals

  4. Open PDR & import the 720p clips - go ahead and do all the editing you need to do adding whatever extra photos audio you want.

  5. Save the project and close PDR

  6. (while PDR's not watching) Replace the 720p clips in [F2] with the original 4K clips in [F1]

  7. Reopen the project and produce.


That sort of replicates the shadow file process and speeds up the works by using a file converter first. It relies completely on being able to rename the original & converted files to match.

Cheers - Tony

P.S. If the weather clears here, the new Sony FDR-X1000V is going for its first ride with me today. laughing

This message was edited 3 times. Last update was at Mar 26. 2015 14:47


Visit PDtoots. PowerDirector Tutorials, tips, free resources & more. Subscribe!
Full linked Tutorial Catalog
PDtoots happily supports fellow PowerDirector users!
Reply
JL_JL [Avatar]
Senior Contributor Private Message Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 5038 Offline
[Post New]
Basically the same approach suggested previously
http://forum.cyberlink.com/forum/posts/list/39786.page or http://forum.cyberlink.com/forum/posts/list/15941.page#78030

Not that anything will come of it, but detailed suggestion also added to "Cyberlink, listen to this suggestion" http://forum.cyberlink.com/forum/posts/list/75/40351.page

Jeff

This message was edited 2 times. Last update was at Mar 24. 2015 00:13

Reply
ynotfish
Senior Contributor Private Message Location: N.S.W. Australia Joined: May 08, 2009 02:06 Messages: 9870 Offline
[Post New]
Agreed Jeff -

It's certainly an area for improvement.

I knew I must have read it somewhere. Damn! All my good ideas come from somone else. embarassed

Cheers - Tony
Visit PDtoots. PowerDirector Tutorials, tips, free resources & more. Subscribe!
Full linked Tutorial Catalog
PDtoots happily supports fellow PowerDirector users!
Reply
[Post New]
Quote: Not that anything will come of it, but detailed suggestion also added to "Cyberlink, listen to this suggestion"
...
Jeff


PDHanomanSvr.exe last changed on Sept 4, 2014. So... I won't hold my breath.

Fun fact: The decoders (h264, h265) are signed January 05, 2015.

This message was edited 2 times. Last update was at Mar 22. 2015 21:01

Reply
The Shadowman
Senior Contributor Private Message Location: UK Joined: Dec 15, 2014 13:06 Messages: 1831 Offline
[Post New]
Hi Tony

Your suggestion for conversion works very well for me. My clips, prior to conversion with FF, were 96 mbps. After the conversion they were down to only 15.66 mbps which PD seems to be able to handle very well.

FF is much quicker to convert files than PD is making shadow files so a lot of time is saved. And sure enough, after editing is completed, the restored 4K files are back to 96 mbps - so no loss in quality.

Thank you for the suggestion

Robert Panny TM10, GH2, GH4,
Reply
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
Tony, Jeff and sonic67,

thanks for the inputs. Yes, I did look at this before but it sort of faded out of my mind.
So I now printed out the idiot proof (essential) instructions and will give it a whirl. Above I mentioned 24 hrs, I think it was longer than that too.

As to replacing the CPU, it is not the money but my talent for screwing things up.
Something as simple as the recent upgrade of the GPU was done with an elevated pulse, blood pressure and trembling hands. I would likely bend 200 of the 400 CPU connector pins.

Some owners of the Sony FDR-AX100 have a problem with the file numbering system of that camera. The numbers start with 0001 each time a blank SDXC is inserted, when these are added to PD it can create a mess.

Will report on this


Eugene

This message was edited 2 times. Last update was at Mar 23. 2015 16:16

73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
Well it works like a charm and FF converted 130 XAVCS 4K video files, 1.2GB in 28 minutes and PD turned out a edited 4K file with titles transitions etc.

Would have taken about 10x longer with PD.

Thank you guys!!


Eugene

Anyone aware of a conversion program that makes use of the GTX960 ?

The 960 works well with the latest Media Player Classic 64.

This message was edited 1 time. Last update was at Mar 23. 2015 16:10

73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
JL_JL [Avatar]
Senior Contributor Private Message Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 5038 Offline
[Post New]
Quote:

Anyone aware of a conversion program that makes use of the GTX960 ?


PD13

Jeff
Reply
[Post New]
http://www.erightsoft.com/SUPER.html

Just use "Custom" at install time to avoid loading any unecessary software. Decline any other offers.

It uses LAV decoder (HA decoding for nVidia). Encode on lower quality with audio stream copy. Select the same file format for output. Increase the priority.

.
[Thumb - Super_settings 2.png]
 Filename
Super_settings 2.png
[Disk]
 Description
 Filesize
87 Kbytes
 Downloaded:
254 time(s)
[Thumb - Super_settings.JPG]
 Filename
Super_settings.JPG
[Disk]
 Description
 Filesize
288 Kbytes
 Downloaded:
219 time(s)

This message was edited 10 times. Last update was at Mar 23. 2015 19:39

Reply
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
Thanks Jeff, I meant making use of its real time encoding and decoding capability, like MPC 64 does for decoding.

The PD shadow files are unfortunately all in one folder, regardless of project name. At the moment my computer has about 25GB of those.
The SF filing system uses the last digit to differentiate the project name:

C0260_0
C0260_1
C0260_2

etc.

While the external creation of those files is very fast, it does not offer the convenience of going back six month later and make changes to the edit w/o the replacement procedure

I wonder if there is a simple way to duplicate the shadow file identification system so that the externally created files can be permanently left in the PD system.

In my case it would require adding "_X" or "_XX" to each file created in the FF converter.

So PD would just covert a few shadow files on it's own in order to create a last digit numerical value to establish the project identification.

An other way might be to create a separate SF folder for each project?



Eugene

This message was edited 1 time. Last update was at Mar 23. 2015 19:11

73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
Myk
Senior Member Private Message Location: The Hartland of Michigan Joined: Feb 05, 2015 16:09 Messages: 205 Offline
[Post New]
I've used Super for many years. Converts anything into anything. .
.
Never, under any circumstances, combine a laxative and sleeping pill on the same night.

PowerSpec G310
ASRock Extreme 6 A85X
AMD A10 6800K Quad Core Processor 4.1GHz
16GB DDR3-1600 RAM
EVGA GTX960 Super OC GPU
Windows 7 Pro
Reply
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
Including 4K ?

I will check it out. How is the conversion quality?


Eugene 73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
[Post New]
Well, it's quality is adjustable. But it doesn't matter, those are just temp files after all...

And yes, it does 4K decoding, see my attachements above. And you can select a different folder for output.

This message was edited 3 times. Last update was at Mar 23. 2015 19:49

Reply
JL_JL [Avatar]
Senior Contributor Private Message Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 5038 Offline
[Post New]
Quote: Thanks Jeff, I meant making use of its real time encoding and decoding capability, like MPC 64 does for decoding.
Not sure I understand the point. For 4K/60p material for me, I can create a manual shadow of 1280x720/60p resolution at the rate of 60sec source > 25sec to generate with my GTX970 and PD13. So a 60min source would convert in 25min I have no experience with the niche GTX960. Decode support benefit not significant for this manual shadow creation from what I have seen. Name or file subsititution only done after editing and when creating final output so one can always come back and edit again latter and retain editing ease while providing 4K source output quality.


Quote: The PD shadow files are unfortunately all in one folder, regardless of project name. At the moment my computer has about 25GB of those.
Not sure I understand. PD generated shadow files are in folder ShadowEditFiles in the "Output" location set in preferences. So each PD project can/could have its own folder. Simple user project management.


Quote: I wonder if there is a simple way to duplicate the shadow file identification system so that the externally created files can be permanently left in the PD system.
Not sure, never attempted that. Casual look now it would appear PD maintains the info in Win Registry so probably not a real convenient way.

Jeff

This message was edited 1 time. Last update was at Mar 23. 2015 21:43

Reply
Eugen157
Senior Contributor Private Message Location: Palm Springs area, So.CA Joined: Dec 10, 2012 13:57 Messages: 660 Offline
[Post New]
Thank you Jeff.



I was not clear in formulating my question about the 960. It would be nice to use the inherent encoding capability


to create a, for example HVEC 4K 265 video from a 4K 264 video in nearly real time or at least a lot faster than PD13.



I do not use or better have not used, individually assigned project output folders, so all projects are in that folder called PD13 output including the shadow files.

PD assigns a different last digit number to identify the shadow files from the various projects. like:

C0260_1.MPG

C0260_2.MPG



When converting externally the file name is

C0260.MPG.

C0261.MPG etc



in order for PD13 to accept this file as its own it has to be changed to



C0260_X.MPG

C0261_X.MPG



So some software is needed to do this.



Eugene

This message was edited 6 times. Last update was at Mar 24. 2015 16:37

73s, WA6JZN ex DL9GC
CYBERLINK PLEASE ADD UHD BLU RAY BURNING SOFTWARE
PD14,
Win10,64bit.CPU i7 6700,16GB ,C= 480 GB SSD ,GPU GTX1060 6GB 1 fan. Plus 3 int, 4 ext HDD's for video etc.LG WH16NS40 reads UHD.
4K 24" ViewSonic monitor.Camera Sony FDR-A
Reply
[Post New]
Quote:
I was not clear in formulating my question about the 960. It would be nice to use the inherent encoding capability
to create a HVEC 4K 265 video from a 4K 264 video in nearly real time or at least a lot faster than PD13.

Eugene


I understand that's planned for next edition of PD...
Reply
JL_JL [Avatar]
Senior Contributor Private Message Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 5038 Offline
[Post New]
Quote: I was not clear in formulating my question about the 960. It would be nice to use the inherent encoding capability

to create a, for example HVEC 4K 265 video from a 4K 264 video in nearly real time or at least a lot faster than PD13


Oh yes, that part of the problem, sorry for not reading between the lines, I was focused on the topic of shadow files. That capability was released in Nvidia 347.09 in Dec/2014 driver package mentioned here http://forum.cyberlink.com/forum/posts/list/42163.page#217296 but awaiting CL incorporation into PD. I'm not sold on h.265 yet so I will sit by and see the technology unfolding. But as you indicate, it's important to you so why not. Basic NVENC support for h.264 encoding on newer Nvidia cards was released in May/2013 and still not fully functional in current PD13 beta patch, this nearly 2 years later. Actually much of the NVENC encoding support was not functional when PD13 was released, support came via a patch and full support of interlaced formats still lacking.

Another add it to the suggestion list, already there but more mention can't hurt. I wish these suggestions all had a running list with tally votes, Maybe the good old user star system!!! I'd sure hate to see another round of GUI and icon color enhancements vs some product core technology improvements for PD14.

Jeff
Reply
[Post New]
^ I totally agree... I would like some core improvements versus more glitter.
Reply
Reply to this topic
Powered by JForum 2.1.8 © JForum Team