Shadow Files - several questions

AnalyticalGuy

Newbie Joined: Sep 22, 2015 03:19 Messages: 14 Offline

Solved by

Sep 22, 2015 22:43

Hi. I've been using PowerDirector for a few months now, ravenously searching this forum whenever I have questions, and almost always finding just the answers I need. (Thanks guys.)

I've also accumulated enough questions to which I couldn't find answers that I'll start a thread of my own -- about Shadow Files. (I'm mostly pursuing Shadow Files because of trouble scrubbing within the Trim tool).

My specs:

Windows 10 Pro, 3.4 GHz Intel Core i7-3770 Processor, 16 GB RAM

NVIDIA GeForce GT 640

PowerDirector runs on my 480GB SSD C: drive.

Video files sit on my 2 TB E: drive.

The Trim tool:

With most video files, I can "scrub" really quickly in the popup Trim tool. I can basically hold down the Up arrow on the keyboard and watch the video move at, say, half speed. But with a few video files (large files? 24 FPS files?), the Trim tool is suddenly very slow. If I hit the Up arrow 5 times in quick succession, I can watch over the next 15 seconds as the video preview slowly crawls through the 5 frames... One. Slow. Frame. At. A. Time.

I'm thinking this might be sped up with Shadow Files? (I'll ask about Hardware Acceleration in another topic.)

Shadow Files:

I've read about how to tell when a Shadow File has been created. (The little icon turns from yellow to green.)

Q: Is there any way to see how long a shadow file will take to create? If not a progress bar, then maybe some rough estimates, based on a hypothetical 1 GB, 1-hour MP4 file?

Any ballpark would help. For example, it would help to know if a 1 GB, 1-hour MP4 will take more like 1 hour, 5 hours, or 24 hours...?

Q: If Windows 10 decides to sleep, does shadow processing continue while the computer sleeps?

Q: Can Windows see that the computer is busy with the "shadow processing," and therefore not sleep?

Based on what I saw today, the answer to both of those are no. I left my computer processing a large file (2 hours, 2 GB) and went to work. Nine hours later, I came home, saw that my computer had decided to sleep, and woke it up. When I did, the icon on the file was still yellow.

This leaves me with some questions about going back and forth between multiple projects (some with shadow files, some without).

Q: If I'm in the middle of a long (multi-hour) shadow processing session, and I quit PowerDirector, only to restart it later, will it resume the shadow processing or start over?

(In other words, do I have to leave PowerDirector running until this particular round of shadow file is 100% complete?)

And finally, here's a scenario:

(1) I turn on Shadow Files to work on a Project A, from source video A. (I turn on shadow files because source video A is huge and I want editing to go more smoothly.)

(2) The shadow processing completes, and the little icon turns green.

(3) Project A will take several weeks, and I'll have to come back to it. In the meantime, I save Project A, and go on to work on projects B and C for a while.

(4) For projects B and C, I turn off Shadow Files.

(5) I come back to project A later, and I turn Shadow Files back on.

Q: Will I have to wait several hours for the shadow files to be created from scratch? Or did they stay on the hard drive, despite having turned shadow files off and on for a couple days?

A lot of these questions might be moot if I learn that creating shadow files doesn't prevent the glacial movement inside the Trim tool. But, either way, it would help me to have a mental model of how to work with Shadow Files -- how much I need to complete them in a single session, prevent Windows from sleeping, avoid doing other projects without shadow files until I'm done, and so on.

Thanks in advance to whoever has some answers! :

AG

This message was edited 1 time. Last update was at Sep 22. 2015 22:46

AnalyticalGuy

Newbie Joined: Sep 22, 2015 03:19 Messages: 14 Offline

Sep 23, 2015 18:22

I've done a few more tests, and I think I have answers to most of my questions. I'll post them, to save others from having to do a bunch of typing (except to correct any mistakes in my report.

Quote: The Trim tool:

With most video files, I can "scrub" really quickly in the popup Trim tool. I can basically hold down the Up arrow on the keyboard and watch the video move at, say, half speed.

Quick correction: This is probably more like 30% speed. Meaning, it takes 3 seconds to scrub through 10 seconds of healthy source video.

Quote: But with a few video files (large files? 24 FPS files?), the Trim tool is suddenly very slow. If I hit the Up arrow 5 times in quick succession, I can watch over the next 15 seconds as the video preview slowly crawls through the 5 frames... One. Slow. Frame. At. A. Time.

Actually, if I hold the Up arrow with one of these grueling videos, I get something like 1 to 2.5 frames per second, depending on the amount of motion in the scene. I'll call it 5% speed.

Quote: I'm thinking this might be sped up with Shadow Files? (I'll ask about Hardware Acceleration in another topic.)

If you want to read no further, the answer is: Yes.

Even if I don't need to bother with shadow files for 90% to 95% of my source files, on the couple that were so hard to read or process, spending the hours creating shadow files suddenly made them smooth as silk in the Trim tool.

Quote: Shadow Files:

I've read about how to tell when a Shadow File has been created. (The little icon turns from yellow to green.)

Q: Is there any way to see how long a shadow file will take to create? If not a progress bar, then maybe some rough estimates, based on a hypothetical 1 GB, 1-hour MP4 file? Any ballpark would help. For example, it would help to know if a 1 GB, 1-hour MP4 will take more like 1 hour, 5 hours, or 24 hours...?

A: My experience: It just converted my 2-hour, 2 GB source video file into a 4.1 GB shadow file over the course of 2.75 hours. I'm not sure whether to abstract that to "approx. 25 MB per minute" or "approx 0.8 source minutes per minute."

Quote: Q: If Windows 10 decides to sleep, does shadow processing continue while the computer sleeps?

A: Apparently not.

Quote: Q: Can Windows see that the computer is busy with the "shadow processing," and therefore not sleep?

A: Also, apparently not. At least not Windows 10.

Quote: Based on what I saw today, the answer to both of those are no. I left my computer processing a large file (2 hours, 2 GB) and went to work. Nine hours later, I came home, saw that my computer had decided to sleep, and woke it up. When I did, the icon on the file was still yellow.

This leaves me with some questions...

Q: If I'm in the middle of a long (multi-hour) shadow processing session, and I quit PowerDirector, only to restart it later, will it resume the shadow processing or start over?

(In other words, do I have to leave PowerDirector running until this particular round of shadow file is 100% complete?)

A: When you turn off PowerDirector, it will immediately delete the in-progress shadow file it was creating.

Note: I tested this on a small video file -- about 4 minutes long. It's conceivable that PowerDirector behaves slightly differently with a longer source file, creating a "build" of the shadow file every half hour or so. But I doubt it.

Quote: And finally, here's a scenario:

(1) I turn on Shadow Files to work on a Project A, from source video A. (I turn on shadow files because source video A is huge and I want editing to go more smoothly.)

(2) The shadow processing completes, and the little icon turns green.

(3) Project A will take me several weeks of editing, and I'll have to come back to it. In the meantime, I save Project A, and go on to work on projects B and C for a while.

(4) For projects B and C, I turn off Shadow Files.

Quick note: Step 4 does not delete the shadow file that was created in step 2.

Quote: (5) I come back to project A later, and I turn Shadow Files back on.

Q: Will I have to wait several hours for the shadow files to be created from scratch? Or did they stay on the hard drive, despite having turned shadow files off and on for a couple days?

A: Short answer: The shadow files remain on the hard drive.

Best guess is that they will remain for whatever duration you have set under the preference called "Auto delete temporary files every __ Days" (30 days by default).

I can't guess if PowerDirector means 30 days of not being used, or how it determines when a file was last used. But the key thing is that they don't disappear the moment you turn off Shadow Files.

More detail:

At the end of step 5, If I come back, reopen Project A, and don't turn Shadow Files back on, then I'm back to something like my old grueling scrubbing speed (5%, or 1 to 2.5 frames per second).

If I enable Shadow Files again, I'm back to a very nice scrubbing speed -- something like 30% to 80% of actual speed -- possibly faster than any other source videos, but not necessarily by much.

Bottom Line:

I can turn off Shadow Files functionality for the projects that don't need them.

Any shadow files I created will remain on my hard drive (for 30 days, maybe?).

If I come back to the project for which I enabled Shadow Files, I can use the files I created previously by re-checking the checkbox.

I hope this is helpful to whoever googles for it in the future!

AG

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Sep 24, 2015 01:55

Hi AnalyticalGuy. Welcome to the forum!

You have some thoughtful questions and I'm glad to see that you've worked out the answers to many of your own questions. To be perfectly honest, I think it would be much easier for other members to help and comment if you limited your questions to just a couple at a time. There's plenty of time to eleborate or ask further questions once the thread is active.

There's so much here that it's hard to read through everything you've requested help with. While it's nice how you quoted the things you've learned about each subject, it also means you have to read almost everything twice to get through your post.

In any case, here are the things I can add to:

Quote: Shadow Files:
I've read about how to tell when a Shadow File has been created. (The little icon turns from yellow to green.)
Q: Is there any way to see how long a shadow file will take to create? If not a progress bar, then maybe some rough estimates, based on a hypothetical 1 GB, 1-hour MP4 file? Any ballpark would help. For example, it would help to know if a 1 GB, 1-hour MP4 will take more like 1 hour, 5 hours, or 24 hours...?

A: My experience: It just converted my 2-hour, 2 GB source video file into a 4.1 GB shadow file over the course of 2.75 hours. I'm not sure whether to abstract that to "approx. 25 MB per minute" or "approx 0.8 source minutes per minute."

That doesn't seem right.

For one thing shadow files are supposed to be low-res versions of your source clips, so I don't know why yours is twice the size. Also, my 1GB 1920x1080 60p MP4 source clip was converted to a 180MB 720x480 30p MPEG clip in 15:40, which occurred at a processing rate of 1.1MB/s. I've attached the MediaInfo details for each one.

My computer was idle while I was running this test, so if you were doing other things or had PD minimized, that might explain why yours took almost 3 hours but it doesn't explain the gigantic file size.

If I use PD13 to manually produce my source clip to the same format as the shadow file, it takes only 2 minutes and is processed at the rate of about 8MB/s (roughly 7x the shadow files creation speed).

Quote: ...(5) I come back to project A later ... did they stay on the hard drive, despite having turned shadow files off and on for a couple days?
A: Short answer: The shadow files remain on the hard drive.
Best guess is that they will remain for whatever duration you have set under the preference called "Auto delete temporary files every __ Days" (30 days by default).

I can't guess if PowerDirector means 30 days of not being used, or how it determines when a file was last used. But the key thing is that they don't disappear the moment you turn off Shadow Files.
AG

PD only checks whether to delete temporary files when you open it, so your temporary files might sit there for a year if you don't use PD during that time. If you open PD fairly regularly, you can count on it managing the deletions pretty much on schedule.

Filename	BTS polymer clay 001_0.MPG.txt	Download
Description	shadow file clip details
Filesize	4 Kbytes
Downloaded:	311 time(s)

Filename	BTS polymer clay 001.MP4.txt	Download
Description	source clip details
Filesize	4 Kbytes
Downloaded:	287 time(s)

AnalyticalGuy

Newbie Joined: Sep 22, 2015 03:19 Messages: 14 Offline

Sep 25, 2015 01:03

Quote: Hi AnalyticalGuy. Welcome to the forum!

You have some thoughtful questions and I'm glad to see that you've worked out the answers to many of your own questions. To be perfectly honest, I think it would be much easier for other members to help and comment if you limited your questions to just a couple at a time. There's plenty of time to eleborate or ask further questions once the thread is active.

Yes, I noticed that the string of questions came out pretty long. I think I was hoping this thread might become the one-stop shop for "Are there any cases where I should use shadow files, and if so, what do I have to look out for in terms of timing and accidental deletion?"

I'll try to keep this new reply as short as possible :

Quote:

Quote: Q: Is there any way to see how long a shadow file will take to create? My experience is that it just converted my 2-hour, 2 GB source video file into a 4.1 GB shadow file over the course of 2.75 hours.

That doesn't seem right.

For one thing shadow files are supposed to be low-res versions of your source clips, so I don't know why yours is twice the size. Also, my 1GB 1920x1080 60p MP4 source clip was converted to a 180MB 720x480 30p MPEG clip in 15:40, which occurred at a processing rate of 1.1MB/s. I've attached the MediaInfo details for each one.

My computer was idle while I was running this test, so if you were doing other things or had PD minimized, that might explain why yours took almost 3 hours but it doesn't explain the gigantic file size.

I agree that it doesn't sound "right," but I'd be glad to give you the data.

Source video:

About 2.1 GB, about 2 hours, data rate about 2300kbps.

Video Codec: "H264 - MPEG-4 AVC"

With the unusual display resolution of 1808 x 1080.

Shadow File creation

I began at 10:29 AM Wednesday.

I checked the shadow file size, in progress, a couple times before leaving. At 10:53 AM, it was up to 564 MB.

At 10:57 AM, it was up to 707 MB.
(707 MB in 28 minutes = 25.25 MB per minute.)

I came home a few hours later to find the shadow file complete. File Properties listed the Last Modified time as 1:18 PM, or 169 minutes after 10:29 AM.

4.1 GB in 169 minutes = about 24.25 MB per minute.

Shadow File video:

About 4.1 GB, still about 2 hours, data rate of exactly 8300kbps.
(In fact, every shadow file PowerDirector has created, across both my computers, has been 8300kbps.)

Video Codec: "MPEG-1/2 Video (mpgv)"

With a display resolution of 720x480 (surprisingly low, considering the data rate?)

I can now scrub smoothly within the Trim tool, so I'm not complaining.

I'm just offering some data, since the numbers seemed unusual to me as well.

Thanks again.

I'll try to keep the next question(s) shorter!

AG

This message was edited 4 times. Last update was at Sep 25. 2015 01:08

AnalyticalGuy

Newbie Joined: Sep 22, 2015 03:19 Messages: 14 Offline

Sep 25, 2015 01:11

P.S. Sorry about the weird spacing in my posts. I'm not mastering the rules of BB code as quickly as I might like.

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Sep 25, 2015 02:00

Yeah, the weird spacing came with the new BB code. A single return becomes two lines of space and there's no way I've found to fix or adjust that. Clicking on the Preview button at the bottom is the best way to see what your post will look like before you go live, and you can also go back and edit it if you want.

Also, please know that I wasn't questioning your numbers, just expressing surprise that they were so different from what I saw. I measured the read and write rates directly using Resource Monitor and watching the read and write streams, and I could see that they stayed in a narrow range the entire time the shadow file was being created.

The 8300kbps bit rate in the shadow file should actually be the max bit rate, and in my file the average of the variable bit rate is 5671kbps (for just the video data). If the bit rate was actually at 8300k for 2 hours, your shadow file would be about 7.5GB, so your average rate is probably closer to 4500kbps.

720x480 is the standard resolution and the high(ish) bit rate means you'll have decent detail available when editing.

If you were willing to download and use MediaInfo (as listed in the Welcome sticky) we could the shadow files compare apples to apples. I made a small donation to get the adware-free version.

The specs on your source video also look unusual to me. I know you meant to type 1920x1080, but shouldn't the bit rate be more like 23,000kbps (23Mbps) instead of 2300k (2.3M)? What frame rate was it shot at? If it was around 5fps then 2300k might be accurate. A very low framerate would also explain why the shadow file is bigger, since I'm assuming it's 30fps.

If your source clip is actually lower resolution than full HD, that could also explain the situation.

Either way, I'll wait for more details (hopefully in the form of a MediaInfo report

AlinDenver1

Newbie Joined: Aug 11, 2007 10:34 Messages: 4 Offline

Nov 21, 2015 13:13

Wait before shadow files are done?? When I start a project by loading all my clips, should I wait for the shadow files to complete before moving them to my time line? Do the shadow files have to be accessed from somewhere to load them into the library or do they automatically replace the original clips?

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Nov 21, 2015 13:40

You can move the clips to the timeline and generally get things ready for editing, but it's better to wait until all the little tags on the clips (when looking at the Media Library) turn from yellow to green before you do any serious work.

There are actually 2 reasons for waiting: PD is busy creating the shadow files and won't have the full use of the CPU to edit or preview clips in real-time, and the unprocessed clips will still be full resolution and you won't see the benefits of working with the lower-resolution shadow files.

There's nothing to worry about if you start editing before the shadow files are finished. You won't damage or corrupt anything, just be aware that you might see some stuttering or choppy previewing if the S.F. aren't ready.

There's also another way of working with clips that would otherwise benefit from shadow files.

The Magic+PD *thread* explains how preconverting your clips can give you real-time editing performance in Full HD preview quality. The other advantages over shadow files are that you can do the conversion process in a batch mode and have all the clips ready before you start editing, and the conversion itself is much faster. You can also easily see the progress, whereas the tags for shadow files can only tell you which clips have been converted and which ones haven't.

While the thread is in the PD14 forum, you can use the Magic+PD approach with any version of PD

AlinDenver1

Newbie Joined: Aug 11, 2007 10:34 Messages: 4 Offline

Nov 21, 2015 13:53

Thanks for the quick reply! I'll start waiting for the shadow files to finish from now on. I'll check out the Magic+PD if my preview screen still freezes ... sure wish CyberLink would integrate this or similar into their software. I waste so much time just waiting for the preview screen to start after moving my time line ... sometimes five minutes sometimes it starts playing right away.
Al in Denver