Repository navigation
Commit 7f719e3
committed
fix: make kl_divergence and chi_square audits run on DuckDB and Trino
kl_divergence failed on every engine:
- table_a_with_q joined USING (@source_column) against a CTE that only has
@target_column, so different columns failed ("missing from right side of join")
- with the same column on both sides, num_rows became ambiguous
- the final SELECT grouped by the column values, so the divergence was
compared per value instead of summed over the distribution
- single-argument LOG() is log10 on DuckDB and does not exist on Trino
Rename the value column to x in both distributions, join them on it, sum
(p - q) * LN(p / q) over the values present in both (the same terms the old
query kept), and compare the single result with the threshold.
chi_square referenced the chi_square select alias in HAVING, which Trino
rejects ("Column 'chi_square' cannot be resolved"). Compute the statistic in a
CTE and filter it with WHERE.
kl_divergence also used HAVING on its alias; the rewrite filters with WHERE.
Signed-off-by: shinichitazawa <shinichitazawa.aws@gmail.com>1 parent d69c262 commit 7f719e3
2 files changed
Lines changed: 36 additions & 57 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
532 | 532 | | |
533 | 533 | | |
534 | 534 | | |
535 | | - | |
| 535 | + | |
536 | 536 | | |
537 | 537 | | |
538 | 538 | | |
539 | 539 | | |
540 | 540 | | |
541 | 541 | | |
542 | 542 | | |
543 | | - | |
| 543 | + | |
544 | 544 | | |
545 | 545 | | |
546 | 546 | | |
547 | 547 | | |
548 | 548 | | |
549 | 549 | | |
550 | 550 | | |
551 | | - | |
552 | | - | |
| 551 | + | |
553 | 552 | | |
554 | 553 | | |
555 | 554 | | |
556 | 555 | | |
557 | 556 | | |
558 | | - | |
559 | | - | |
| 557 | + | |
560 | 558 | | |
561 | 559 | | |
562 | 560 | | |
563 | | - | |
| 561 | + | |
| 562 | + | |
564 | 563 | | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
| 564 | + | |
569 | 565 | | |
570 | | - | |
571 | | - | |
572 | | - | |
573 | | - | |
574 | | - | |
575 | | - | |
576 | | - | |
577 | | - | |
578 | | - | |
579 | | - | |
580 | | - | |
581 | | - | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
586 | | - | |
587 | | - | |
588 | | - | |
589 | | - | |
590 | | - | |
591 | | - | |
592 | | - | |
593 | | - | |
594 | | - | |
595 | | - | |
596 | | - | |
597 | | - | |
598 | | - | |
599 | | - | |
600 | | - | |
601 | | - | |
602 | | - | |
603 | | - | |
604 | | - | |
| 566 | + | |
| 567 | + | |
605 | 568 | | |
606 | 569 | | |
607 | | - | |
608 | | - | |
609 | | - | |
610 | | - | |
611 | | - | |
612 | | - | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
613 | 573 | | |
614 | 574 | | |
615 | 575 | | |
| |||
646 | 606 | | |
647 | 607 | | |
648 | 608 | | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
649 | 615 | | |
650 | 616 | | |
651 | 617 | | |
652 | | - | |
653 | | - | |
654 | | - | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
655 | 621 | | |
656 | 622 | | |
657 | 623 | | |
658 | 624 | | |
659 | | - | |
| 625 | + | |
660 | 626 | | |
661 | 627 | | |
662 | 628 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
651 | 651 | | |
652 | 652 | | |
653 | 653 | | |
654 | | - | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
655 | 668 | | |
656 | 669 | | |
657 | 670 | | |
| |||
0 commit comments