Avoid KeyError when the Axon is not ready (SYN-11748)

[vEpiphyte]

SYN-11748: Avoid KeyError when the Axon is not ready

Problem

Cortex.axoninfo is an empty dict until the Axon connects (the embedded Axon finishing its boot, or a
remote Axon's onlink firing). Several Storm code paths indexed axoninfo['synapse']['version'] directly.
If one of them ran before the Axon was connected, Storm raised a bare KeyError.

Every axoninfo read under synapse/:

Site	Read	Waited on axready first?
stormtypes.resolveAxonProxyArg()	['synapse']['version']	Only through its callers
$lib.axon.wget() with ssl_opts	['synapse']['version']	Yes, with no timeout
$lib.axon.wput() with ssl_opts	['synapse']['version']	Yes, with no timeout
$lib.inet.http with sha256 fields (proxy + ssl_opts)	['synapse']['version']	No
$lib.axon.unpack()	.get('features', {})	No, so it reported FeatureNotSupported before connect

There was also no way for an operator to see whether the Cortex's Axon was ready.

Changes

• Added stormtypes.getAxonSynapseVersion(runt). It waits for the Axon, reads the version with get(), and raises
  FeatureNotSupported if the version is missing. Every site that indexed axoninfo['synapse']['version']
  now uses it.
• Cortex.getAxon() now takes timeout=s_const.AXON_READY_TIMEOUT (300 seconds, the same as the default
  $lib.inet.http request timeout) and raises s_exc.TimeOut if the Axon is not ready in time.
  timeout=None waits indefinitely as before. It returns the Axon iden from the Axon's reported cell
  info, or None if the Axon did not report one. Previously it returned self.axon.iden, which was a
  telepath Method object rather than an iden when the Axon was remote. If the Axon is already ready, it
  returns immediately. This is a behavior change for Python callers, which previously waited indefinitely.
• Behavior change: every $lib.axon method (wget, wput, urlfile, put, has, size, read,
  unpack, readlines, jsonlines, csvrows, list, upload, hashset, metrics, del and dels) now raises
  TimeOut after 300 seconds instead of blocking indefinitely. Each one checks its permission before it
  waits.
• $lib.axon.unpack() now waits for the Axon before it checks the Axon's unpack feature, so it no longer
  reports FeatureNotSupported just because the Axon has not connected yet.
• The Cortex getCellInfo() output now includes axon:ready in its cell section. The Axon's version is
  deliberately not included, because Telepath getCellInfo() is open to any authenticated user.
• The deprecated $lib.bytes APIs (put, has, size, hashset and upload), the IMAP server fetch()
  method, the delnode --delbytes command, and Cortex.feedFromAxon() (Telepath and $lib.feed.fromAxon())
  get the same bounded wait, after their permission checks. Cortex.exportStormToAxon()
  ($lib.export.toaxon()), which has no permission check of its own, waits before it starts the export.
• The Telepath APIs CoreApi.getAxonUpload() and CoreApi.getAxonBytes() used to wait indefinitely on
  axready. They now raise TimeOut after 300 seconds.
• The Cortex Axon HTTP endpoints (/api/v1/axon/files/...) used to wait for the Axon indefinitely, before
  checking authentication or permissions. They now check permissions first, then wait up to 300 seconds and
  return an HTTP 503 with a TimeOut error if the Axon is still not ready. An unauthenticated or
  unauthorized request is refused immediately, whatever the Axon's state.
• The Axon HTTP upload handler (/api/v1/axon/files/put) no longer starts an upload in the Axon after it
  denies a request. This fix also applies to the Axon service itself.

Tests

• Clearing core.axoninfo raises FeatureNotSupported instead of KeyError: 'synapse'.
• With the Axon down, every Axon-backed Storm, Telepath, and HTTP API times out, and a user without permission
  is denied immediately.
• getAxon(), axon:ready, and the denied-upload fix have direct tests.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.84%. Comparing base (c5cf39d) to head (8845a41).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5019      +/-   ##
==========================================
- Coverage   97.93%   97.84%   -0.09%     
==========================================
  Files         308      308              
  Lines       65763    65792      +29     
==========================================
- Hits        64405    64377      -28     
- Misses       1358     1415      +57     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.