Skip to content

KeyLock: add per-job configurable heartbeat timeout in RunOpts #16

Description

@sotashimozono

Problem

src/KeyLock.jl has a heartbeat stale threshold, but it is a global constant. For HPC workflows:

  • Fast jobs (N=20 DMRG): 60s stale threshold is fine
  • Slow jobs (N=100 TPQMPS, chi=128, beta=10): runs 30–90 minutes; a 5-minute stale threshold causes incorrect lock reclamation by other workers

Proposed fix

Add configurable timeouts to RunOpts:

@kwdef struct RunOpts
    heartbeat_interval::Float64 = 30.0   # seconds between heartbeat touches
    stale_threshold::Float64 = 300.0     # seconds before lock is considered stale
    max_retries::Int = 3
    # ... existing fields
end

Pass opts.stale_threshold to KeyLock.is_stale(lock; threshold=...).

Use case

FiniteTemperature.jl Phase 1 TPQ runs at chi=128, N=100, beta=10 can take 30–90 minutes per sample. The current default 5-minute stale threshold causes premature lock reclamation on multi-node HPC runs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature新機能・既存アルゴリズムの拡張

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions