Cache private keys, reuse remote clients, and batch DAG status queries

[romer8]

The problem

Tethys polls job status through condorpy on every UI refresh. Each poll was far more
expensive than it needed to be. Instrumenting a single status poll of a 4-node DAG
against an HTCondor 8.8.2 pool:

Cost	Time	Share
RSA private key decryption (5x)	2.27 s	70%
condor_q / condor_history (14 calls)	0.58 s	29%
SSH handshakes (5x)	0.37 s	8%
Total	2.03 s

Under concurrent load this starves the calling application: 8 concurrent viewers of 10
running workflows drove poll latency to 20 s and cut throughput 9x, with the scheduler
itself idle at 0.2% CPU.

Why it was slow

The private key was re-decrypted on every property access. set_scheduler() built a new
RemoteClient each time it was called, and RemoteClient.__init__ calls
paramiko.RSAKey.from_private_key_file(), which runs a KDF (~0.47 s) for a
passphrase-protected key. Callers invoke set_scheduler() from a property, so this happened
several times per poll. RemoteClient already caches its SSH transport, but replacing the
client object discarded that cache, forcing a fresh handshake too.

Node statuses were queried one at a time. _update_statuses() looped over node_set
calling job.status, and each of those runs condor_q ... && condor_history .... A
DAGManJobID constraint already returns every node of the DAG in one query, so the per-node
round trips were redundant. On a 14-node DAG this is ~40 remote commands per poll.

Reading node_set performed a remote query. The property called update_node_ids() on
every access, so merely iterating the nodes cost a round trip.

The changes

1. load_private_key() caches decrypted keys on (path, passphrase, mtime). The mtime
   means a rotated key file is picked up rather than served stale. Guarded by a lock, since
   callers may be threaded.
2. set_scheduler() reuses the existing RemoteClient when the connection parameters are
   unchanged, via RemoteClient.matches(), so the cached transport survives. _remote_id
   is preserved on the reuse path -- it names the job's remote working directory, so minting
   a new one would orphan the remote workspace.
3. node_statuses_by_cluster_id() fetches every node's status in one query;
   _update_statuses() uses it and falls back to per-node queries if it fails.
4. node_set resolves node ids once per object instead of on every read. add_node()
   clears the flag so a newly added node is still resolved.

Results

Same live DAG, before and after: 2.03 s -> 0.51 s per poll (4x), with identical
correctness -- same job status, same node count, same per-node statuses, and the same 14
remote commands issued, confirming no work was skipped. Key decryptions per poll went 5 -> 0
(warm) and clients 5 -> 1.

The stronger evidence for change 3 is that Workflow.statuses becomes independent of DAG
size. Measured against 26-node DAGs with 24 nodes expanded, on a caller pinned to 0.5 CPU,
with L concurrent viewers each monitoring their own live workflow:

Concurrent viewers	Workflow.statuses path	per-node job.status path
2	490 ms	2700 ms
4	510 ms	2800 ms
6	700 ms	3300 ms
8	890 ms	4000 ms

The batched path stays flat at ~0.5 s and degrades gracefully under concurrency instead of
collapsing. Zero failures at every level.

What this does not fix

The right-hand column above is the honest limit of this PR. A caller that asks each node for
its status individually -- for node in workflow.node_set: node.job.status -- still pays one
condor_q/condor_history pair per node, measured at ~115 ms per expanded node. condorpy
cannot batch that from the inside, because the request arrives one node at a time.

node_statuses_by_cluster_id() is the primitive such callers should move to; changes 1, 2
and 4 reduce the cost of each individual call but not their number. Tethys has a companion
change (tethysplatform/tethys#1294) that consumes the batched method and persists the result,
which is what removes the per-node pattern on its side.

Reproducing the measurements

One caveat for anyone verifying this: a DAG node that DAGMan has not yet submitted has no
cluster_id, and condorpy reports Unexpanded for it without any remote call. If the root
node of a test DAG runs for the duration of the test, the fan-out is never submitted and poll
cost looks flat regardless of DAG size. Give the root node a short runtime and the leaves a
long one, and confirm expansion with
condor_q -constraint "DAGManJobID==<cluster>" before trusting a number.

Tests

tests/test_remote_reuse.py adds 9 tests covering key caching (including reload on file
change), client reuse and replacement when the host changes, _remote_id stability across
reuse, and node id resolution behavior.

Test count goes from 72 to 81 with python -m unittest discover. The two pre-existing
errors (test_remote, test_integraiton.test_submit) are unchanged -- they require a local
condor pool and ~/.ssh/id_rsa, which CI provides.

Verified against HTCondor 8.8.2 and 25.x.