Skip to content

fix(MemcachedFactory): Enable TCP_NODELAY and adjust default timeouts to be reasonable - #62670

Merged
DerDreschner merged 1 commit into
masterfrom
fix/flaky-memcached-tests
Jul 29, 2026
Merged

fix(MemcachedFactory): Enable TCP_NODELAY and adjust default timeouts to be reasonable#62670
DerDreschner merged 1 commit into
masterfrom
fix/flaky-memcached-tests

Conversation

@DerDreschner

@DerDreschner DerDreschner commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

The CI tests against memcached fail sporadically. After some research, it's because the default timeouts are way to low for high load scenarios. The ones being used at the moment have no reasoning behind them at all (we use these ones according to the commit message here). This is being mentioned in #14995 and can be observed when comparing our values with those of other PHP applications as well:

  • MediaWiki: Uses 500ms for everything by default (see those config on line 55 onward together with the option setters on line 100 onward)
  • Symfony and Laravel: None specific timeouts being set by default, which means the library defaults from 5 seconds for POLL_TIMEOUT and 4 seconds for CONNECT_TIMEOUT apply

For our CI tests, the POLL_TIMEOUT is the one that makes the tests fail. As we're not enabling TCP_NODELAY at the moment, the bare minimum ACK time for small packages (like a HIT MISS with binary protocol) is 40ms with the default Linux kernel settings. Dangerously close to the 50ms timeout we currently use. Under high load scenarios, the kernel can increase the ACK time to up to 200ms and the RFC even allows up to 500ms.

That's why this PR enabled TCP_NODELAY to speed up HIT MISS scenarios (same as Symfony does, which seem reasonable as well) and adjusts the default timeouts to those being used by MediaWiki as they seem pretty reasonable.

Checklist

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

… to be reasonable

Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
@DerDreschner DerDreschner self-assigned this Jul 29, 2026
@DerDreschner
DerDreschner requested a review from a team as a code owner July 29, 2026 14:23
@DerDreschner DerDreschner added 3. to review Waiting for reviews feature: caching Related to our caching system: scssCacher, jsCombiner... CI labels Jul 29, 2026
@DerDreschner
DerDreschner requested review from icewind1991, nfebe, salmart-dev and sorbaugh and removed request for a team July 29, 2026 14:23
@DerDreschner DerDreschner added the tests Related to tests label Jul 29, 2026
@DerDreschner
DerDreschner enabled auto-merge July 29, 2026 14:24
@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable34

@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable33

@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable32

@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable31

@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable30

@DerDreschner

Copy link
Copy Markdown
Contributor Author

/backport to stable29

@DerDreschner

Copy link
Copy Markdown
Contributor Author

Backport down to stable29 happens to stabilize CI runs on GitHub. Previous branches don't have the memcached PHPUnit tests using GitHub Actions.

@kesselb kesselb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up that ancient ticket, seems reasonable to adjust the limits 👍

nit: Using the class name as scope is not ideal. The scope is meant to describe the area/module/subsystem affected. Hence I would suggest cache or memcache. Which files are changed are visible from the diff already, no need to repeat that in the commit message.

@kesselb kesselb added this to the Nextcloud 35 milestone Jul 29, 2026

@susnux susnux left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense

@DerDreschner
DerDreschner merged commit f84fe75 into master Jul 29, 2026
274 of 285 checks passed
@DerDreschner
DerDreschner deleted the fix/flaky-memcached-tests branch July 29, 2026 22:53
@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable29 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable29
git pull origin stable29

# Create the new backport branch
git checkout -b backport/62670/stable29

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable29

Error: Failed to check for changes with origin/stable29: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable30 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable30
git pull origin stable30

# Create the new backport branch
git checkout -b backport/62670/stable30

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable30

Error: Failed to check for changes with origin/stable30: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable31 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable31
git pull origin stable31

# Create the new backport branch
git checkout -b backport/62670/stable31

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable31

Error: Failed to check for changes with origin/stable31: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable32 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable32
git pull origin stable32

# Create the new backport branch
git checkout -b backport/62670/stable32

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable32

Error: Failed to check for changes with origin/stable32: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable33 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable33
git pull origin stable33

# Create the new backport branch
git checkout -b backport/62670/stable33

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable33

Error: Failed to check for changes with origin/stable33: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@backportbot

backportbot Bot commented Jul 29, 2026

Copy link
Copy Markdown

The backport to stable34 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable34
git pull origin stable34

# Create the new backport branch
git checkout -b backport/62670/stable34

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick 308240aa

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/62670/stable34

Error: Failed to check for changes with origin/stable34: No changes found in backport branch


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review Waiting for reviews backport-request CI feature: caching Related to our caching system: scssCacher, jsCombiner... performance 🚀 tests Related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change default memcached timeout values

3 participants