[MSE] Video glitches when switching from lower quality to higher quality content

[filipe-norte]

During BBC test suite run (e.g. ACT-1070), video glitches / small temporary freezes can be occasionally observed. The glitches happen when a switch from lower quality content (e.g. bitrate, resolution) to higher quality content occurs, but not the other way around.

The high level sequence of events is the following:

• Playback start is triggered
• Low quality data segments are provided covering a sufficiently long period, e.g. [0, 40] seconds
• After a few seconds, e.g. at 10 seconds media time offset, the high quality data init segment is provided
• After a few seconds, e.g. at 15 seconds media time offset, a few segments of high quality data are provided starting from the time the init segment was provided (e.g. [10, 25] seconds)
• As soon as high quality data is provided for the currently playing position (e.g. 18 seconds media time offset), we switch to the high quality data playback
• video glitch is observed

The attached test case reproduces this behavior with wpe-webkit using custom streams (not bbc original). Same test case works without glitches on Firefox.

bbc-tests-video-glitch-custom-stream.20260407.1.zip

The video glitches can be seen on both wpe-2.46 and wpe-2.38, although the sequence here described and reproduction with the test app was confirmed only with wpe-2.46.

The analysis using original streams shows the following:

Lower quality data pushed with 25 fps, leading to total buffered range from 149.76 to 180.48:

0:02:43.678611755    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xa5b3f690, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 425862, offset none, offset_end none, flags 0x0    
0:02:43.684073705    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5b66aa0, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:43.690383611    27    0x66e80 DEBUG                qtdemux qtdemux.c:9331:gst_qtdemux_guess_framerate:<qtdemux0> Calculating framerate, timescale 600 gave fps_n 25 fps_d 1 

0:02:43.787276919    27    0x4a920 INFO       webkitmediaplayer GStreamerCommon.cpp:613:didLogMessage: SourceBuffer::sourceBufferPrivateAppendComplete(86330001) buffered = [{149760000/1000000 = 149.76}..{180480000/1000000 = 180.48}] , totalBufferSize: 3075193

Init segment is pushed with higher quality data and 50 fps:

0:02:44.051806894    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xa5b445e8, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 687, offset none, offset_end none, flags 0x0
0:02:44.052169706    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5faac98, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:44.060935478    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1
0:02:44.061535769    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1

0:02:44.067008460    27    0x4a920 INFO       webkitmediaplayer GStreamerCommon.cpp:613:didLogMessage: SourceBuffer::sourceBufferPrivateAppendComplete(86330001) buffered = [{149760000/1000000 = 149.76}..{180480000/1000000 = 180.48}] , totalBufferSize: 3075193  

Actual higher quality data follows around 3 seconds later for range 165.12 to 168.94 sec (= 2min48sec.940ms)!

0:02:47.433335464    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:773:pushNewBuffer:<append-pipeline-video-mp4-0> pushing data buffer buffer: 0xaa9a35e8, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 1336346, offset none, offset_end none, flags 0x0

0:02:47.449566243    27    0x66e80 DEBUG                qtdemux qtdemux.c:9331:gst_qtdemux_guess_framerate:<qtdemux0> Calculating framerate, timescale 600 gave fps_n 50 fps_d 1

0:02:47.451445745    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:794:pushNewBuffer:<append-pipeline-video-mp4-0> pushing end-of-append buffer buffer: 0xa5b30738, pts 99:99:99.999999999, dts 99:99:99.999999999, dur 99:99:99.999999999, size 0, offset none, offset_end none, flags 0x0

0:02:47.451654891    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1
0:02:47.451779149    27    0x4a920 DEBUG       webkitvideotrack VideoTrackPrivateGStreamer.cpp:131:updateConfigurationFromCaps:<webkitappsinkwithworkarounds1:sink> Updating video configuration from video/x-h264, stream-format=(string)avc3, alignment=(string)au, level=(string)3.1, profile=(string)high, codec_data=(buffer)0164001fffe000, width=(int)960, height=(int)540, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1

0:02:47.455585744    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:542:appsinkNewSample:<append-pipeline-video-mp4-0> append: trackId=1 PTS={165120000/1000000 = 165.12} DTS={165120000/1000000 = 165.12} DUR={20000/1000000 = 0.02} presentationSize=960x540

...

0:02:47.586452523    27    0x4a920 TRACE              webkitmse AppendPipeline.cpp:542:appsinkNewSample:<append-pipeline-video-mp4-0> append: trackId=1 PTS={168920000/1000000 = 168.92} DTS={168940000/1000000 = 168.94} DUR={20000/1000000 = 0.02} presentationSize=960x540

0:02:47.605508054    27    0x4a920 DEBUG         webkitmediasrc WebKitMediaSourceGStreamer.cpp:686:webKitMediaSrcStreamFlush:<source:src_1> Sending FLUSH_START downstream.

This is the time where a video glitch / small freeze is seen, which is coincident with the flush being triggered.

[vivienne-w]

Thanks for the analysis and the test case!

I can see the same issue here, also in upstream, however it doesn't seem to be specific to increasing quality. Decreasing quality or even just appending the same samples again reproduces the stutters as long as the new bitrate is high enough - the new samples will need some time to be decoded and displayed, and until then we have nothing to display.

We should already be handling this by buffering, I'll investigate why this isn't happening here.

[vivienne-w]

@filipe-norte-red in the end, this looks like a hardware limitation.
As you noticed, forcing an immediate quality change like this will cause a flush -
this is actually triggered by multiplatform WebKit code, before enqueueing the new samples, right here:
https://github.com/WebKit/WebKit/blob/8ef14b8fb84787970de037cc722cd6598f60ff19/Source/WebCore/platform/graphics/SourceBufferPrivate.cpp#L458

Like I mentioned above, for us this means we no longer have samples to display, and the decoder has to catch up.
As such, there also isn't anything we can do to work around this issue. I'd suggest avoiding this situation and only
changing quality in newly appended ranges where possible.

Regarding firefox, what version did you test this on? I currently have 147 installed, where I can clearly see
the same stuttering, same on Mac in Safari.

[filipe-norte]

Thanks for the feedback and update, @vivienne-w .

I'm using Firefox148.0.2 (64-bit) on Ubuntu 24.04 and confirmed that I don't see any video glitch in my case, just a clean transition between qualities. Also tried on Chrome with Version 146.0.7680.80 (Official Build) 64-bit (although had to patch the test to enable controls to enable playback).
The test makes one transition to higher quality and then one to lower quality a few seconds after (these are the relevant ones). It will eventually stop for buffering a few seconds after the last transition just because the test is not adding more data (this was just for the purpose of showing the main issue).

I haven't analyzed the sample replacement code in detail, but if we replace samples only from the beginning of the next segment for which data was not provided to the decoding pipeline yet, I'd expect that the flush operation would be avoided in such case, is that correct? If the new quality data happens mid-segment and the full segment data was not provided for the original quality, then in this case the flush would be required.

Consider the examples below where LQ = Low Quality, HQ = High Quality, s1 = segment 1, s2 = segment 2, etc.:

No flush expected use case 1

• LQ s1 appended
• LQ s2 appended
• LQ s3 appended
• LQ s1 fully sent to decoding pipeline
• HQ s2 appended, replacing LQ s2
• HQ s2 to decoding pipeline

The transition is clean from LQ to HQ from s1 to s2

No flush expected use case 2

• LQ s1 appended
• LQ s2 appended
• LQ s3 appended
• LQ s1 fully sent to decoding pipeline
• LQ s2 partially sent to decoding pipeline
• HQ s2 appended, does not replace any LQ s2 samples, as we have the full segment in original quality
• HQ s3 appended, replaces LQ s3
• LQ s2 remaining samples sent to decoding pipeline
• HQ s3 to decoding pipeline

The transition is clean from LQ to HQ from s2 to s3

Flush expected use case 1

• LQ s1 appended
• LQ s2 partially appended
• LQ s1 to decoding pipeline
• LQ s2 partially sent to decoding pipeline
• HQ s2 appended, adds remaining samples for the segment range not yet sent to the decoding pipeline

A flush is required during s2 as the app did not provide full segment data before the switch

Thanks

[vivienne-w]

@filipe-norte-red To elaborate on quality increases/decreases: the test case does indeed change quality twice, however the second change, which reverts to the original quality, does so because we stop appending samples in the higher quality. This means we just use the original samples, which have plenty of time to be enqueued normally, and flow through the playback pipeline without the pressure on the decoder the first quality change causes.
In your test case, you change from an initial 400k stream to a 4_000k one, if you instead start at a higher quality and change to a lower quality (I've used 12_000k -> 4_000k for my testing), you can see this issue occur also for a reduction in quality.

Regarding flushes, it sadly isn't that simple. WebKit will enqueue as many samples as it can, until the implementation tells it it no longer isReadyForMoreSamples() 1. In our case, we maintain between 2 and 5 seconds of samples 2, plus whatever buffers the elements in the playback pipeline hold (mainly queue elements, but also the decoder itself for example). As soon as a sample is enqueued, there is no way to stop or replace it, except by flushing and re-enqueueing.

That aside, the BBC test you mentioned is using dash.js, which we have contributed an option to which would avoid causing this situation in the first place 3. There is a possibility that even with that fix enabled, the size of our buffers might exceed whatever threshold dash.js would be applying, but I'm still investigating if that is what is happening.

[filipe-norte]

Thanks for the prompt reply and clarifications, @vivienne-w .

Just to clarify the flow I was referring to (and I understand it may unfortunately not be as "simple" as the description may seem to suggest :) ), we wouldn't need to stop or try to undo what was sent do the decoding pipeline (without a flush). We would continue sending the samples in the original quality (if available) till the next segment boundary (the flow control via isReadyForMoreSamples() would not affect this). The sample replacement algorithm would need to know from which point samples in the new quality could be safely added (replace original quality ones), which would be the start of a segment not yet sent to the decoding pipeline (which the trackbuffer decode queue may tell us based on the code here). If samples till the next segment boundary are not available (in the original quality), the flush is unavoidable.

Note: I refer to segments, but the boundary is really the next available keyframe (to have a clean transition point), so consider for the sake of simplicity of the explanation that the term "segment" I used refers actually to a GOP (which may not be the case in practice).

I was also able to reproduce the video glitch on my computer using Firefox as you observed by using the 12Mbps stream. I still observed it only when going from low to high quality, but I understand the issue should be the same the other way around.

Thanks

[vivienne-w]

Thanks for the clarification, I understand better what you were suggesting now. Delaying the quality change until a later keyframe is presumably also what Chromium is doing to allow for smooth playback, but that is a tradeoff rather than a solution. For example, we cannot predict how much data will be buffered in the player, or how far keyframes will be apart from one another, so it would take an arbitrary amount of time for the new quality to actually display. This would create a new problem - imagine the user of the application manually selecting a new resolution, which they can only see 10 seconds later.

However, I've looked into the ACT-1070 test you mentioned, as well as the specific version of dash.js 1 it uses. I'm able to reproduce these stutters with a small test using the same fastSwitchingEnabled option that the BBC test also uses, and enabling avoidCurrentTimeRangePruning does consistently fix it. The BBC test doesn't seem to use this, so if it's possible to have that option enabled, that looks like it would be the best solution. I'll also see if I can find a convenient way to set this option on-the-fly in the test, so you can also try it on your end.

Alternatively, we can also try reducing buffer sizes, so the window of time around the playback position in which appends cause a flush and re-enqueue becomes smaller.

[filipe-norte]

That is a good point, @vivienne-w ! If the app just appends new quality data as a consequence of an explicit user request, then such delay would be potentially misleading to the user and not allow a good user experience. The flow I described would work only for non-user initiated quality changes, such as network speed change, etc. I imagine there isn't really a guarantee that the app goes through a flow of calls that would let webkit know there is an intentional action where a video glitch might be expected.
Let me come back to you regarding the option you suggested.

[filipe-norte]

@vivienne-w , unfortunately requesting BBC to make changes to their apps' dash config is not a viable option and an alternative would be required.
Thank you

[vivienne-w]

@filipe-norte thanks for the update, I'm now working on delaying the quality change until a later keyframe where possible. I'll also see how much we can reduce buffer sizes, which would reduce how often this scenario would occur, and should also minimize how long we have to delay the change when it does happen.