Skip to content

Increase BackingStore existence check timeout to 30 minutes - #195

Merged
treddy08 merged 1 commit into
mainfrom
fix/backingstore-wait-30min
Oct 29, 2026
Merged

treddy08 merged 1 commit into
mainfrom
fix/backingstore-wait-30min

Conversation

@treddy08

Copy link
Copy Markdown
Contributor

Extend the timeout for waiting for NooBaa BackingStore creation from 5 minutes to 30 minutes to accommodate NooBaa's automatic AWS fallback behavior.

Background:
NooBaa operator has built-in platform detection that attempts to create AWS S3-backed storage when running on AWS. When AWS credentials are not available (common in sandbox environments), NooBaa has a hardcoded 10-minute timeout before automatically falling back to pv-pool backed storage.

Timeline observed in production:

  • NooBaa created at T+0
  • Attempts AWS S3 BackingStore creation (waits for credentials)
  • At T+10min: Automatic fallback to pv-pool BackingStore
  • At T+13min: BackingStore fully OPTIMAL

Changes:

  • Increase retries from 30 to 180 (180 retries * 10 seconds = 30 minutes)
  • Keeps delay at 10 seconds for reasonable check frequency
  • Total wait time: 1800 seconds (30 minutes) vs previous 300 seconds (5 minutes)

This timeout accounts for:

  • NooBaa's 10-minute AWS credential wait
  • BackingStore pod creation and initialization (~3 minutes)
  • 15+ minute buffer for slower clusters

Tested-by: Production deployment observation on cluster cvmdm.sandbox2624.opentlc.com
Resolves: Ansible timeout failures during ODF deployment on AWS without credentials

Extend the timeout for waiting for NooBaa BackingStore creation from
5 minutes to 30 minutes to accommodate NooBaa's automatic AWS fallback behavior.

Background:
NooBaa operator has built-in platform detection that attempts to create
AWS S3-backed storage when running on AWS. When AWS credentials are not
available (common in sandbox environments), NooBaa has a hardcoded 10-minute
timeout before automatically falling back to pv-pool backed storage.

Timeline observed in production:
- NooBaa created at T+0
- Attempts AWS S3 BackingStore creation (waits for credentials)
- At T+10min: Automatic fallback to pv-pool BackingStore
- At T+13min: BackingStore fully OPTIMAL

Changes:
- Increase retries from 30 to 180 (180 retries * 10 seconds = 30 minutes)
- Keeps delay at 10 seconds for reasonable check frequency
- Total wait time: 1800 seconds (30 minutes) vs previous 300 seconds (5 minutes)

This timeout accounts for:
- NooBaa's 10-minute AWS credential wait
- BackingStore pod creation and initialization (~3 minutes)
- 15+ minute buffer for slower clusters

Tested-by: Production deployment observation on cluster cvmdm.sandbox2624.opentlc.com
Resolves: Ansible timeout failures during ODF deployment on AWS without credentials
@treddy08
treddy08 merged commit 296104a into main Oct 29, 2026
1 check passed
@treddy08
treddy08 deleted the fix/backingstore-wait-30min branch October 29, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants