Add NEON for av2_highbd_cwp_convolve_2d - #5425
jjustiss-apple wants to merge 1 commit into
Conversation
|
I'll work on adding NEON for the other |
|
@jjustiss-apple : just want to confirm: did you meant to target |
@urvangjoshi Good question--the If you see a better solution to reduce churn, I'd be happy to rework this PR. |
Ah, got it. That makes sense. I'll let @jianj-g review. |
| const int do_average = conv_params->do_average; | ||
| const int use_wtd_comp_avg = is_uneven_wtd_comp_avg(conv_params); | ||
|
|
||
| int16_t im_block[(MAX_SB_SIZE + MAX_FILTER_TAP) * MAX_SB_SIZE]; |
There was a problem hiding this comment.
it'd be better to use DECLARE_ALIGNED(16, ...) for this array
| if (do_average) { | ||
| uint16x8_t r = cwp_compound_avg_clip_8( | ||
| lo, hi, round1_shift, d16, fwd_offset, bck_offset, use_wtd_comp_avg, | ||
| sub_const, rb_shift, max_val); | ||
| vst1q_u16(d, r); | ||
| } else { | ||
| cwp_store_dst16_8(lo, hi, round1_shift, d16); | ||
| } |
There was a problem hiding this comment.
For all vertical loop functions, do_average here is evaluated 4 times in each inner loop and they are invariant for this function. I was trying to find a way to move it out of the inner loop, and AI (Gemini) came up with this idea (Please verify, the example is for 8wide_neon):
Define a macro for this if/else but evaluate !do_average first (and combine do_average and use_wtd_comp_avg into one single MODE):
// Compile-time constant MODE:
// 0 = store to dst16 (!do_average)
// 1 = simple compound avg (do_average && !use_wtd_comp_avg)
// 2 = weighted compound avg (do_average && use_wtd_comp_avg)
#define CWP_FINISH_ROW_8(lo, hi, MODE) \
do { \
if ((MODE) == 0) { \
cwp_store_dst16_8((lo), (hi), round1_shift, d16); \
d16 += dst16_stride; \
} else { \
uint16x8_t r = cwp_compound_avg_clip_8( \
(lo), (hi), round1_shift, d16, fwd_offset, bck_offset, \
(MODE) == 2, sub_const, rb_shift, max_val); \
vst1q_u16(d, r); \
d += dst_stride; \
d16 += dst16_stride; \
} \
} while (0)
And then for each vertical loop function, define another macro for the inner loop:
#define VERT8_4ROWS(MODE) \
do { \
int16x8_t s7 = vld1q_s16(s); s += src_stride; \
int16x8_t s8 = vld1q_s16(s); s += src_stride; \
int16x8_t s9 = vld1q_s16(s); s += src_stride; \
int16x8_t s10 = vld1q_s16(s); s += src_stride; \
\
int32x4_t lo, hi; \
lo = cwp_highbd_convolve8_v_lo(s0, s1, s2, s3, s4, s5, s6, s7, f_lo, f_hi, \
vert_offset); \
hi = cwp_highbd_convolve8_v_hi(s0, s1, s2, s3, s4, s5, s6, s7, f_lo, f_hi, \
vert_offset); \
CWP_FINISH_ROW_8(lo, hi, MODE); \
\
lo = cwp_highbd_convolve8_v_lo(s1, s2, s3, s4, s5, s6, s7, s8, f_lo, f_hi, \
vert_offset); \
hi = cwp_highbd_convolve8_v_hi(s1, s2, s3, s4, s5, s6, s7, s8, f_lo, f_hi, \
vert_offset); \
CWP_FINISH_ROW_8(lo, hi, MODE); \
\
lo = cwp_highbd_convolve8_v_lo(s2, s3, s4, s5, s6, s7, s8, s9, f_lo, f_hi, \
vert_offset); \
hi = cwp_highbd_convolve8_v_hi(s2, s3, s4, s5, s6, s7, s8, s9, f_lo, f_hi, \
vert_offset); \
CWP_FINISH_ROW_8(lo, hi, MODE); \
\
lo = cwp_highbd_convolve8_v_lo(s3, s4, s5, s6, s7, s8, s9, s10, f_lo, \
f_hi, vert_offset); \
hi = cwp_highbd_convolve8_v_hi(s3, s4, s5, s6, s7, s8, s9, s10, f_lo, \
f_hi, vert_offset); \
CWP_FINISH_ROW_8(lo, hi, MODE); \
\
s0 = s4; s1 = s5; s2 = s6; s3 = s7; s4 = s8; s5 = s9; s6 = s10; \
height -= 4; \
} while (height > 0)
Then in the vertical loop function, you can use:
if (!do_average) {
VERT8_4ROWS(0);
} else if (use_wtd_comp_avg) {
VERT8_4ROWS(2);
} else {
VERT8_4ROWS(1);
}
I think it may worth the effort. MODE is a compile-time constant this way. Let me know what you think!
There was a problem hiding this comment.
Will do! Let me benchmark this out and I'll share the A/B results.
There was a problem hiding this comment.
@jianj-g thank you for the excellent suggestion! Your suggestion yielded another 6-8% speed-gain on the micro benchmarks. More importantly, it provided great structure and moved the bottlenecks to uncover some more optimization opportunities for my optimizer to explore.
On-top of your suggestion, it looks like converting the runtime shifts to consolidate in SQRSHRUN and adding symmetric vertical folding, and halving-adds were all improvements.
Thank you again, let me know what you think of the update.
p.s. the updated ARM CTC run is nearly complete, I'll update the PR-body with the results once its done. I primarily want to confirm these are lossless changes.
5a7c3a4 to
f2b5c20
Compare
f2b5c20 to
0977268
Compare
|
There's a mismatch in the latest version of NEON that needs to be addressed. I'll update the unit tests to cover the gap. |
4 participants
Add ARM NEON intrinsics for av2_highbd_cwp_convolve_2d (2D compound
weighted prediction convolution, high bit-depth).
Micro-benchmarks (Apple M2 P-core, single-thread, EIGHTTAP_REGULAR subpel=8):
CTC Results (RA, cpu-used=1, 33 frames, A5, ARM):
Pending