Commit edfb21d
Add source filter and use indexed hash prefix in cert tag batch query
The certification tag batch query (TagUsageDAO.getCertTagsInternalBatch)
was hitting ~12 seconds per call on instances with deep classification
hierarchies — fired ~5,800 times per Data Insights run, contributing
~19 hrs of cumulative DB time per DI run.
Two missing index-friendly predicates caused the slowness:
1. No `source = ?` filter — couldn't use idx_tag_usage_target_exact
(source, targetFQNHash, state) INCLUDE (tagFQN, labelType) whose
covering INCLUDE has tagFQN.
2. `tagFQN LIKE 'Certification.%'` on the raw column — there's no
LIKE-friendly index on raw tagFQN, only on tagfqn_lower text_pattern_ops
and tagFQNHash. The LIKE always ran as a post-filter on every row the
IN clause returned.
Fix:
- Add `source = :source` filter (Certifications are always Classification
source = 0).
- Switch `tagFQN LIKE :tagFQNPrefix` → `tagFQNHash LIKE :tagFQNHashPrefix`,
with the hash prefix pre-computed via FullyQualifiedName.buildHash so the
query hits the indexed hash column.
Same SQL on MySQL and Postgres — no @ConnectionAwareSqlQuery split needed.
Also a correctness improvement: the `source = 0` filter excludes glossary
terms (source = 1) that happen to have FQNs starting with "Certification.".
Previously such glossary terms could be incorrectly returned as
certifications; now they're excluded as expected.
Test:
- Added test_certBatch_bulkFetchReturnsCorrectCertsPerEntity in
TagResourceIT — exercises the bulk fetch path with three schemas
(cert-tagged / untagged / non-cert-tagged) and asserts each gets
the right certification (or null) in the listed response. Locks in
source-filter correctness and prevents future regressions where a
non-cert tag could leak into the certification field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 048a15e commit edfb21d
3 files changed
Lines changed: 122 additions & 5 deletions
File tree
- openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests
- openmetadata-service/src/main/java/org/openmetadata/service/jdbi3
Lines changed: 108 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1589 | 1589 | | |
1590 | 1590 | | |
1591 | 1591 | | |
| 1592 | + | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
| 1619 | + | |
| 1620 | + | |
| 1621 | + | |
| 1622 | + | |
| 1623 | + | |
| 1624 | + | |
| 1625 | + | |
| 1626 | + | |
| 1627 | + | |
| 1628 | + | |
| 1629 | + | |
| 1630 | + | |
| 1631 | + | |
| 1632 | + | |
| 1633 | + | |
| 1634 | + | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
| 1696 | + | |
| 1697 | + | |
| 1698 | + | |
| 1699 | + | |
1592 | 1700 | | |
1593 | 1701 | | |
1594 | 1702 | | |
| |||
Lines changed: 5 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6476 | 6476 | | |
6477 | 6477 | | |
6478 | 6478 | | |
6479 | | - | |
6480 | | - | |
| 6479 | + | |
| 6480 | + | |
| 6481 | + | |
6481 | 6482 | | |
6482 | 6483 | | |
6483 | 6484 | | |
| 6485 | + | |
6484 | 6486 | | |
6485 | | - | |
| 6487 | + | |
6486 | 6488 | | |
6487 | 6489 | | |
6488 | 6490 | | |
| |||
Lines changed: 9 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5266 | 5266 | | |
5267 | 5267 | | |
5268 | 5268 | | |
5269 | | - | |
| 5269 | + | |
| 5270 | + | |
| 5271 | + | |
5270 | 5272 | | |
5271 | 5273 | | |
5272 | 5274 | | |
| |||
9944 | 9946 | | |
9945 | 9947 | | |
9946 | 9948 | | |
9947 | | - | |
| 9949 | + | |
| 9950 | + | |
| 9951 | + | |
| 9952 | + | |
| 9953 | + | |
| 9954 | + | |
9948 | 9955 | | |
9949 | 9956 | | |
9950 | 9957 | | |
| |||
0 commit comments