Reject VCL OBUs with mlayer_id/tlayer_id > max

[y-guyon]

In handle_sequence_header(), validate that a VCL OBU's mlayer_id and tlayer_id do not exceed the active sequence header's max_mlayer_id and max_tlayer_id before using the active sequence header on non-keyframes. Previously, those bounds were only checked when activating a new sequence header on a keyframe, causing an assertion failure in is_tlayer_scalable_and_dependent() when decoding a subsequent non-keyframe VCL OBU with an out-of-range mlayer_id.

To reproduce:

build/test_libavm --gtest_filter="*InvalidMlayerIdInNonKeyframeObu*"

Without the patch:

test_libavm: /usr/local/google/home/yguyon/git/avm/av2/common/av2_common_int.h:3245: is_tlayer_scalable_and_dependent: Assertion 'seq->max_mlayer_id >= curr_mlayer_id' failed. Aborted (core dumped) build/test_libavm --gtest_filter="*InvalidMlayerIdInNonKeyframeObu*"

With the patch:

[ OK ] NoAssertionFailure.InvalidMlayerIdInNonKeyframeObu (12 ms)

This bug was found with libavif's fuzz targets.

[wantehchang]

Yannis: This if statement becomes dead code after you added keyframe_unit_in_tu && to the condition at line 7629. Please double check whether this if statement should be deleted or should be moved to a different place, such as inside the new if statement at line 7860.

Note: I didn't really review this PR. I only did some pattern matching.

[wantehchang]

See my comment at line 7644. I wonder if we should move the are_seq_headers_consistent() check here so that this if statement looks like the following:

  if (!keyframe_unit_in_tu) {
    if (!are_seq_headers_consistent(&cm->seq_params, seq_from_uch)) {
      avm_internal_error(&cm->error, AVM_CODEC_CORRUPT_FRAME,
                         "Sequence Header changed at %s",
                         avm_obu_type_to_string(obu_type));
    }
    return;
  }