Skip to content

Fix two causes of flaky nightly builds - #2311

Merged
kounelisagis merged 2 commits into
mainfrom
kounelisagis/nightly-flakes
Sep 25, 2026
Merged

kounelisagis merged 2 commits into
mainfrom
kounelisagis/nightly-flakes

Conversation

@kounelisagis

@kounelisagis kounelisagis commented Sep 25, 2026 •

Copy link
Copy Markdown
Member
  • The bytes round-trip test no longer fails when a slow runner takes over 2s; it only checks the data read back.
  • The TileDB release list download is retried three times, and a download that still fails says so instead of the unrelated STRIP requires two arguments error.
  • These are the two causes behind the recent The nightly build with earliest supported numpy job failed on Thursday (2026-03-26) #2306 failures; a healthy run builds and tests the same as before.

The test failed whenever its body took over 2s, which a shared CI runner does now and then, while the round trip itself passed. It was left in to debug #1194; the test now checks only what it reads back.
…be fetched

file(DOWNLOAD) without STATUS leaves an empty file on failure and no error, so a dropped download surfaced later as "string sub-command STRIP requires two arguments". It is now retried three times and then reported with the URL and reason.

@teo-tsirpanis teo-tsirpanis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reluctant approve; not a big fan of this approach, but I don't see anything very wrong.

Related to this, it would make more sense to me for nightly builds to build TileDB from source; either the dev branch or the latest release from vcpkg, which would feature an updated set of dependencies.

Comment on lines +52 to +69
foreach(ATTEMPT RANGE 1 ${ATTEMPTS})
file(DOWNLOAD
${UPSTREAM_URL}/${VERSION}/releases.csv
${RELLIST_URL}
${CMAKE_CURRENT_BINARY_DIR}/releases.csv
SHOW_PROGRESS
${HASH_ARGS}
STATUS DOWNLOAD_STATUS
)
list(GET DOWNLOAD_STATUS 0 DOWNLOAD_CODE)
list(GET DOWNLOAD_STATUS 1 DOWNLOAD_MESSAGE)
if(DOWNLOAD_CODE EQUAL 0)
break()
endif()
message(WARNING "Downloading ${RELLIST_URL} failed (attempt ${ATTEMPT} of ${ATTEMPTS}): ${DOWNLOAD_MESSAGE}")
if(ATTEMPT LESS ATTEMPTS)
execute_process(COMMAND ${CMAKE_COMMAND} -E sleep 5)
endif()
endforeach()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of adding retry logic in our own build system code. I wish CMake supported that itself.

@kounelisagis
kounelisagis merged commit e07bd04 into main Sep 25, 2026
26 checks passed
@kounelisagis
kounelisagis deleted the kounelisagis/nightly-flakes branch September 25, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants