Serious frame access issues with H.264 M2TS clips

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Solved by

Mar 21, 2019 22:03

I've been working with HD MTS clips since 2011 and have almost always produced to M2TS, since the quality is very good and YouTube happily accepts them as is. Over the years I've had many kinds of frustrating experiences due to the technical kinds of edits I tend to make, as I often need to place a keyframe, cut or trim on exactly the right frame.

Things got noticeably worse when the 2514 patch came out for PD17, but I had a difficult time finding any common ground between various problems I kept seeing - even though they kept coming up again and again. Now, after some hunches and a ton of forensic work, I have a big part of the answer.

It's a frame access problem with PD17 (specifically 17.0.2514.2) and H.264 M2TS clips.

The gist of the issue is that for some reason, PD17 is unable to access every frame in this type of clip when doing normal timeline editing. It's not much of an issue with playback or producing, and it's only when you need access to a specific frame that the real problems start.

I made a 5 second test video with a unique number in each frame and produced it many different ways with several different hardware configurations in both PD14 and 17. In general, PD14 handles the clips reasonably well, with only a few specific frames "unreachable" from the timeline regardless of whether the test clips were produced by PD14 or PD17.

PD17 is a whole other story, with as many as 1/2 of the frames in any given second absolutely unavailable from the timeline, and clpis it's produced have more issues than when produced with the same settings by PD14. This explains why it's been so hard to have a clip behave consistently when editing.

An important thing to note is that I each produced clip actually has every single frame present. You can see that everything is intact on other platforms with easy frame access, like VirtualDub2, so this is solely a timeline/editing access issue in PD17.

Here's a video showing a clip produced by PD14 alongside one produced with the exact same settings in PD17. The first section is what PD17 does when previewing them frame by frame, and the second half is the exact same clips in PD14:

In my testing, I've found that H.264 clips produced as MP4s do not have this issue, neither do H.265 clips produced as M2TS. The problem is much worse with nVidia HW encoding (both with v411 and v417 drivers). Using a GTX 780Ti gives different results than using an RTX 2070, but both are seriously flawed.

Using CPU encoding or Intel QuickSync avoided the biggest frame issues (blocks of frames that PD17 cannot access from the timeline), but when produced by PD17, all profiles had a significant number of inaccessible or displaced frames. Interestingly, whether the Intel or nVidia card is seen by PD17 determines which artifacts CPU-only producing will generate.

Here's a partial screenshot of my data table. Anyone who's bored (or masochistic) enough can download it and see all the gory details here.

So, anyone who wants to test this out can check out any of my test clips in this OneDrive folder. Try putting a few on the timeline and see what the previews look like when you step frame by frame like in my video. I also have the original project that created my test clips in thisfolder. It has a 1.5GB source clip, but you can skip downloading that and simply use any 60p clip when you open it.

I'd really like to make sure that the debacle I've experienced isn't limited to my machine.

YouTube/optodata

DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors

Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°

tomasc

Senior Contributor Joined: Aug 25, 2011 12:33 Messages: 6464 Offline

Mar 22, 2019 00:32

I did check out my 1080/p60 m2ts original and m2ts cpu rendered PD17 clips in the BDMV/STREAM folder. I specifically chose from 5.00 to 7.00 seconds. That is where you see the image being transitioned in the background video, so you know it has to be rendered. Only the frame 6.01 and 6.03 is not shown in the capture.

I did a check on my original captured m2ts video and there were no frame not accessed in 2 seconds and have a screen recording of that.

BTW: My videos are captured as M2TS and not MTS. The difference is that the software used will write the start time in the name of the clips and they will be sorted chronologically. This makes it easy to sort if shooting is over a long period of time on different SD cards from different sources.

Filename	20190321225113.mp4	Download
Description	Cpu produced m2ts with frames 6.01 and 6.03 not accessed.
Filesize	5045 Kbytes
Downloaded:	597 time(s)

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 22, 2019 01:39

Quote I did check out my 1080/p60 m2ts original and m2ts cpu rendered PD17 clips in the BDMV/STREAM folder. I specifically chose from 5.00 to 7.00 seconds. That is where you see the image being transitioned in the background video, so you know it has to be rendered. Only the frame 6.01 and 6.03 is not shown in the capture.

I did a check on my original captured m2ts video and there were no frame not accessed in 2 seconds and have a screen recording of that.

This seems to be slightly different as it looks like PD's frame counter actually skips those two frames. That may be part of the 59.94/60 conversion. With my tests, the timeline counter shows every single frame number, but the content of the video as shown on the preview doesn't match what is actually recorded in the clip.

What happens if you try out any of the RTX or GTX clips in my shared folder? The clips are only 17MB each and you don't need to download all of them.

YouTube/optodata

DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors

Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°

tomasc

Senior Contributor Joined: Aug 25, 2011 12:33 Messages: 6464 Offline

Mar 22, 2019 11:57

I think that there is an answer for displaying the same frame while the frame counter advances. You may not like it. Downloaded the PD17 h.264 m2ts test clip GTX411.m2ts. See the same results you placed for the first second. My PD17 would go to undocked full screen while advancing the frame counter. Hit Esc key. Redock the preview screen. My pc does that on all PD versions if I hit frame advance too many times. Hit play and it plays with a blank black screen. Must hit the stop button to get the preview to display correctly again. See the screen recording.

Advanced the frame counter(fc) 1 frame at a time and see the numbered frame(f). Here are the discrepancies:
Fc 13-25 f13
Fc 29 f27
Fc 31 f32
Fc 34 f35
Fc 37 f38
Fc 40 – 51 f39
Fc 53 – 1:04 f52
Fc 1:06 f1:05 screenshot screen for saving snapshot capture popped up. Cancelled it. See the screenshot with the frame counter (fc) at 1:06 and the blank black screen.

The discrepancies of the frame counter(fc) matches the frames in your excel spreadsheet.

Filename	20190322094720.mp4	Download
Description	PD undocks the preview screen and does not recover. Must hit STOP to get correct preview again.
Filesize	3166 Kbytes
Downloaded:	621 time(s)


Filename	Opto screenshot.jpg	Download
Description	Frame counter at 1:06. Had to cancel the snapshot save screen that popped up.
Filesize	286 Kbytes
Downloaded:	23 time(s)

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 22, 2019 13:11

Quote I think that there is an answer for displaying the same frame while the frame counter advances. You may not like it. Downloaded the PD17 h.264 m2ts test clip GTX411.m2ts. See the same results you placed for the first second. My PD17 would go to undocked full screen while advancing the frame counter. Hit Esc key. Redock the preview screen. My pc does that on all PD versions if I hit frame advance too many times. Hit play and it plays with a blank black screen. Must hit the stop button to get the preview to display correctly again. See the screen recording.

...

The discrepancies of the frame counter(fc) matches the frames in your excel spreadsheet.

Thanks very much for confriming the missing frames. I see you're running Win7, so that helps rule out the possibility that this only happened with PD17 on Win10.

Your attached screen recording clearly shows the blank preview issue, but not the preview undocking. I had to limit my clicks to about 1 per second to avoid the screen undocking. There's another screen capture vid in the media library of your screen shot (...4720.mp4). Is that the one that has the undocking in it?

Also, seeing the issue in the fully undocked preview window shows exactly why my editing has been so difficult. Just try to find the right frame when PD does this:

The video also shows that PD is actually able to access all the frames when playing back and producing, so the issue is in the random access/decode algorithm needed for editing, That code is subject to differences in how the source clip was produced as shown by the significant differences in it's behavior when working with PD17-produced clips vs those produced with the same settings in PD14, and I believe this issue got worse with the 2514 update.

Since I have the subscription version, I can't roll back to an earlier version. It might be helpful to see if anyone who's able to run v2314 could test a few clips and see if PD17 can randomly/indivdually access more frames than my spreadsheet shows. Same goes for PD15 and 16 to get a better sense of when this whole mess really started.

This is a big deal

tomasc

Senior Contributor Joined: Aug 25, 2011 12:33 Messages: 6464 Offline

Mar 22, 2019 13:34

It does seem that the Nvidia 411 driver gave the worst results on PD17 according to the spreadsheet. The 417-driver performed much better. Years ago, I read that a manufacturer can write a driver to perform the best in any magazine graphics card test when they know the test beforehand. The tests were published online. Some users believe that once you find the best driver then don’t update.

The Intel graphics driver was once criticized for fuzzy ouput. Others got blamed for caching the test. Hardware acceleration and rendering from what I remember can be like reduce the number of screen refresh like if the screen is static then there is no need to refresh. This is one component of hardware acceleration. Very little change then don’t refresh. In mpeg – encoding the moving part of video gets encoded on every frame while the background is encoded only every other frame. Tests show how the human eyes perceive things and advanced codec are developed based on this. Graphics card drivers try to give users the best gaming experience.

Your line 49-52 iCPU meaning cpu encoding possibly with and without hardware decoding produced the best results along with Quick sync.

Maybe this is the reason some users like to use smart rendering and cpu rendering. I have not seen an article on how avchd encoding works before posting just now.

tomasc

Senior Contributor Joined: Aug 25, 2011 12:33 Messages: 6464 Offline

Mar 22, 2019 14:05

You replied while I was putting together the next response. The screen recording is made after the preview screen was redocked. It is a good challenge.

Need to have other users involved in this issue.

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 22, 2019 15:58

Quote Your line 49-52 iCPU meaning cpu encoding possibly with and without hardware decoding produced the best results along with Quick sync.

The funny thing is that if I produce to H.264 MP4 rather than M2TS, there are either no issues or maybe only one missing frame. Same with H.265 M2TS and MP4, so the simple workaround is to avoid producing to H.264 M2TS.

Obviously this is an issue that Cyberlink should address, and I'll file a support ticket if another member or two could post their findings with a couple of the sample clips.

YouTube/optodata

DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors

Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°

Roland vl

Newbie Joined: Nov 23, 2017 03:44 Messages: 39 Offline

Mar 24, 2019 06:00

Quote

...

Since I have the subscription version, I can't roll back to an earlier version. It might be helpful to see if anyone who's able to run v2314 could test a few clips and see if PD17 can randomly/indivdually access more frames than my spreadsheet shows. Same goes for PD15 and 16 to get a better sense of when this whole mess really started...

I have PD16 and I have tested with your files "iCPU" and "GTX411". The findings from your sheet and your description of the problem: I can confirm this also for PD16 (no access of several frames in preview).

BUT. When I produce with PD16 I have access to all my frames of my produced file (only CPU-rendering - no hardware decoding, no svrt, no hardware acceleration.) This means a degradation of PD17 vs PD16.

Using another editor VideoStudioX10, when I load your files "iCPU" and "GTX411" I have access to all frames.

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 24, 2019 12:15

Quote I have PD16 and I have tested with your files "iCPU" and "GTX411". The findings from your sheet and your description of the problem: I can confirm this also for PD16 (no access of several frames in preview).

BUT. When I produce with PD16 I have access to all my frames of my produced file (only CPU-rendering - no hardware decoding, no svrt, no hardware acceleration.) This means a degradation of PD17 vs PD16.

Using another editor VideoStudioX10, when I load your files "iCPU" and "GTX411" I have access to all frames.

Thanks very much for testing this out!

That matches what I've seen, namely that the clips themselves are ok and that the issue really looks to be in PD17's "random access" algorithm. That critical code is needed to pick out an individual frame and fully reconstruct it from the encoded data so it can be worked with on the timeline, and clearly some significant issues have crept in in the most recent version(s) of it.

I have filed support ticket CS002004733 for this issue.

YouTube/optodata

DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors

Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Mar 26, 2019 00:02

See if this meets your expectations.

Jeff

EDIT, upload to forum had failed, shared link and file I had provided was removed as it already served it's purpose.

This message was edited 1 time. Last update was at Mar 26. 2019 07:48

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 26, 2019 02:55

Quote See if this meets your expectations.
https://drive.google.com/drive/folders/1U1KpIfwGJfoB7WEFW6vBmWeIonnxmrfF?usp=sharing

Jeff

Thanks for producing and sharing this. The frame behavior of this clip doesn't match anything in my spreadsheet: PD17 can access all frames up until 4:42, where the previous frame (:41) is displayed. The same thing happens at the final two frames, where 4:57 is shown rather than :58 and :59.

Is that what you see on your system, and what profile and produce settings did you use to make the clip? Maybe an AMD GPU?

I should be able to work with this clip without encountering any of the more frustrating issues, so this is definitely an improvement.

Also, at Cyberlink's request I have added a 2nd tab at the bottom of the spreadsheet that decodes the shorthand jargon of my test clip names, and also details the produce settings for each. Under Preferences > Produce, Open CL, Reduce video noise and the WMV Speed Mode options were always disabled. The only setting I changed during testing was SVRT/IDR and that had only a minor effect.

This message was edited 1 time. Last update was at Mar 26. 2019 03:04

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Mar 26, 2019 07:45

Quote Thanks for producing and sharing this. The frame behavior of this clip doesn't match anything in my spreadsheet: PD17 can access all frames up until 4:42, where the previous frame (:41) is displayed. The same thing happens at the final two frames, where 4:57 is shown rather than :58 and :59.

Sounds like you are maybe judging this by timeline playback, try media library playback and see if the 4:42 issue is not actually ok. If more content, like 6min continuous clip, the 4:42 issue would have been ok too I believe for timeline playback.

Quote Is that what you see on your system, and what profile and produce settings did you use to make the clip? Maybe an AMD GPU?

CPU and/or Nvidia hardware encode, both yield the same that I could see. I did not try a AMD GPU. Issue is with PD, not Nvidia hardware encode as is so often blamed on this forum. In my view, it's an age-old PD issue that I've seen resurface a few times, hence I knew how to create a custom profile based on PD's "Profile Analyzer" to correct. I'm not sure what CL uses for version control, it's like some student picks up some old code or something for his project and it gets implemented. How this stuff gets through a QA system I'll never understand. I'm sure CL support will correct in a future patch, for at least a bit again.

Jeff

SoNic67

Senior Contributor Joined: Sep 27, 2014 14:14 Messages: 1308 Offline

Mar 26, 2019 10:35

Quote I'm not sure what CL uses for version control, it's like some student picks up some old code or something for his project and it gets implemented. How this stuff gets through a QA system I'll never understand. I'm sure CL support will correct in a future patch, for at least a bit again.
Jeff

Sadly that's the aproach of more and more companies. Cuts in the QA department, since they get the QA done in forums...

optodata

Senior Contributor Location: California, USA Joined: Sep 16, 2011 16:04 Messages: 8630 Offline

Mar 26, 2019 12:27

Quote Sounds like you are maybe judging this by timeline playback, try media library playback and see if the 4:42 issue is not actually ok. If more content, like 6min continuous clip, the 4:42 issue would have been ok too I believe for timeline playback.

In the media library, the frame @ 4:42 is accessible and so is the frame @ 4:58, with only the final frame (4:59) displaying as a duplicate of 4:58. That's either a fairly subtle change (299/300 accurat and accessible frames vs. 297/300); or a huge improvement (2/3 of the timeline's inaccessable frames are now accessible). Either way, it is clearly different.

I just tried previewing my PD14 GTX (411) test clip from the media library and found these differences compared to the timeline:

So it would appear that the media library previewer is distinct from the one used on the timeline, and it is able to access more frames. Maybe that awareness will help the people at Cyberlink as they work on resolving this, but having better previewing performance in the media library isn't of any benefit when trying to edit affected clips.

Quote Sadly that's the aproach of more and more companies. Cuts in the QA department, since they get the QA done in forums...

I hope CL would at least look into interning or hiring an MSQA person and giving them a chance to look at all the disparate modules, uncountable band-aid fixes and myriad internal inconsistencies and simply prioritize them.

Then get the biggest bang for the buck by focussing on the most significant productivity killers, make the product more reliable and thrill your customer base while adding legions of new customers. That's what smart business investment looks like.

YouTube/optodata

DS365 | Win11 Pro | Ryzen 9 3950X | RTX 4070 Ti | 32GB RAM | 10TB SSDs | 5K+4K HDR monitors

Canon Vixia GX10 (4K 60p) | HF G30 (HD 60p) | Yi Action+ 4K | 360Fly 4K 360°

JL_JL

Senior Contributor Location: Arizona, USA Joined: Oct 01, 2006 20:01 Messages: 6091 Offline

Mar 26, 2019 14:05

Quote
So it would appear that the media library previewer is distinct from the one used on the timeline, and it is able to access more frames.

That's been known for ages over any PD release I've had, back to 2.x. It's part of the reason why users are confronted with media playing in the library but then get a black screen in the timeline and often get sent on a codec mission. Or another classic observance was hardware decoding working with media library playback but not timeline playback.

You can see the different playback was even referenced here from a beta tester, release 9 I believe. https://forum.cyberlink.com/forum/posts/list/14828.page#post_box_71971

Quote I hope CL would at least look into interning or hiring an MSQA person and giving them a chance to look at all the disparate modules, uncountable band-aid fixes and myriad internal inconsistencies and simply prioritize them.

Then get the biggest bang for the buck by focussing on the most significant productivity killers, make the product more reliable and thrill your customer base while adding legions of new customers. That's what smart business investment looks like.

One would hope, but that's been a user request on virtually every front when posts were made for improvement ideas over many years, I guess it must not make sense vs new bright red lipstick.

Jeff

PowerDirector Moderator

Senior Contributor Location: New Taipei City, Taiwan Joined: Oct 18, 2016 00:25 Messages: 2104 Offline

Apr 03, 2019 04:01

Quote I've been working with HD MTS clips since 2011 and have almost always produced to M2TS, since the quality is very good and YouTube happily accepts them as is. Over the years I've had many kinds of frustrating experiences due to the technical kinds of edits I tend to make, as I often need to place a keyframe, cut or trim on exactly the right frame.

Things got noticeably worse when the 2514 patch came out for PD17, but I had a difficult time finding any common ground between various problems I kept seeing - even though they kept coming up again and again. Now, after some hunches and a ton of forensic work, I have a big part of the answer.

It's a frame access problem with PD17 (specifically 17.0.2514.2) and H.264 M2TS clips.

Optodata,

A hotfix has been made available to fix this issue and posted here: https://forum.cyberlink.com/forum/posts/list/78931.page

Cheers,
PowerDirector Moderator

For customer support related issues, please contact:
- Customer service: https://membership.cyberlink.com/support/customer-services.do
- Technical support: https://membership.cyberlink.com/support/service/technical-support.do