fix(ssrs): streamline test connection, stream reports, retry transient failures

[ulixius9]

Summary

• Test connection no longer paginates all reports. The GetDashboards step was calling client.get_reports() which walked every page of /Reports — on a large SSRS instance that's a huge, unnecessary cost. It now issues a single $top=1 probe (client.test_get_reports), mirroring the existing CheckAccess probe against /Folders.
• SsrsClient.get_reports now streams. Converted from list-accumulating to an Iterator[SsrsReport] that yield froms each page as it's fetched, and SsrsSource.get_dashboards_list yields non-hidden reports one at a time. Ingestion memory stays flat regardless of report count.
• Transient failures are now retried. Session mounts an HTTPAdapter with a urllib3 Retry (2 retries, exponential backoff, covers connect errors, read timeouts, 5xx). Motivated by a production trace where a single 30s ReadTimeout against /Reports aborted the fetch and left the pipeline reporting success with 0 records.

Test plan

• Unit tests (tests/unit/topology/dashboard/test_ssrs.py) — 22 pass, includes new test_get_reports_streams_lazily asserting the generator only fetches the first page until the consumer advances.
• Integration tests (tests/integration/ssrs/, tests/integration/connections/test_ssrs_connection.py) — 10 pass, includes new test_get_connection_test_get_reports, test_connection_bad_host_get_reports, and test_get_reports_retries_transient_failures (flaky mock server returns 503 twice then 200; asserts exactly 3 requests).
• End-to-end against a real SSRS instance — not feasible without a Windows host; manual verification recommended.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR improves SSRS ingestion reliability and scalability by making the SSRS “reports” listing lazy/streamed, reducing test-connection cost, and adding automatic retries for transient HTTP failures.

Changes:

• Replace SSRS connection test “GetDashboards” step from a full /Reports pagination to a $top=1 probe via SsrsClient.test_get_reports.
• Convert SsrsClient.get_reports() to stream results page-by-page (iterator) and update SsrsSource.get_dashboards_list() to yield visible reports lazily.
• Add requests session retry behavior (urllib3 Retry via HTTPAdapter) and extend unit/integration tests accordingly.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ingestion/src/metadata/ingestion/source/dashboard/ssrs/client.py	Adds session-level retries and changes get_reports() to a streaming iterator; introduces test_get_reports() probe.
ingestion/src/metadata/ingestion/source/dashboard/ssrs/metadata.py	Changes dashboard listing to yield non-hidden reports lazily from the client iterator.
ingestion/src/metadata/ingestion/source/dashboard/ssrs/connection.py	Updates test-connection step mapping to use the lightweight test_get_reports() probe.
ingestion/tests/unit/topology/dashboard/test_ssrs.py	Updates tests for iterator-based APIs and adds a laziness test verifying only the first page is fetched initially.
ingestion/tests/integration/ssrs/test_metadata.py	Updates integration tests to materialize the iterator with list(...).
ingestion/tests/integration/connections/test_ssrs_connection.py	Adds integration coverage for test_get_reports() and for retry behavior using a flaky HTTP handler.

[harshach]

Are we sure retries is hte reason we are getting timed out?

[gitar-bot]

Code Review ✅ Approved

Streamlines SSRS test connections, implements report streaming, and adds retry logic for transient failures. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

CONNECT_TIMEOUT/READ_TIMEOUT combined with MAX_RETRIES can exceed the 3-minute test-connection timeout used by SSRS (THREE_MIN=180s). Worst case for a single GET is roughly (CONNECT_TIMEOUT+READ_TIMEOUT)*(MAX_RETRIES+1) plus backoff, which can cause test_get_reports / connection testing to time out even though retries are still in progress. Consider lowering the probe timeouts (or making _get accept an override timeout for test probes), reducing retries for test probes, or increasing the SSRS test-connection timeout to keep the worst-case under the configured limit.

[sonarqubecloud]

Quality Gate failed for 'open-metadata-ingestion'

Failed conditions
E Security Review Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

[github-actions]

🟡 Playwright Results — all passed (10 flaky)

✅ 3701 passed · ❌ 0 failed · 🟡 10 flaky · ⏭️ 89 skipped

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	481	0	0	4
🟡 Shard 2	655	0	1	7
🟡 Shard 3	665	0	1	1
🟡 Shard 4	646	0	2	27
🟡 Shard 5	609	0	2	42
🟡 Shard 6	645	0	4	8

🟡 10 flaky test(s) (passed on retry)

• Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
• Flow/PersonaFlow.spec.ts › Set default persona for team should work properly (shard 3, 1 retry)
• Pages/Customproperties-part2.spec.ts › entityReferenceList shows item count, scrollable list, no expand toggle (shard 4, 1 retry)
• Pages/DataContractsSemanticRules.spec.ts › Validate Description Rule Is_Not_Set (shard 4, 1 retry)
• Pages/EntityDataSteward.spec.ts › User as Owner Add, Update and Remove (shard 5, 1 retry)
• Pages/Glossary.spec.ts › Add and Remove Assets (shard 5, 2 retries)
• Pages/Lineage/DataAssetLineage.spec.ts › verify create lineage for entity - Search Index (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace