Add opt-in API call logging for LANforge JSON requests

[Sidartha-CT]

---

Add opt-in API call logging for LANforge JSON requests

Summary

Adds a lightweight, opt-in CSV logger for every json_get / json_post / json_put / json_delete call made through either of LANforge's two client paths:

• py-json/LANforge/lfcli_base.py → Realm
• lanforge_client/lanforge_api.py → LFJsonCommand / LFJsonQuery

When enabled via:

lf_webpage.py --save_api

each call's method, URL, payload, HTTP status, and error/diagnostics get appended to a CSV (default ~/lf_api_calls.csv), which is copied into the run's report folder.

This is meant for debugging a specific test run, not for permanent instrumentation.

---

Why a new lanforge_client/api_logger.py module

We want to record specific fields per call (method, url, payload, status, error) and post-process them later as CSV, so the recording logic needed to live inside json_get() / json_post() / json_put() / json_delete() themselves rather than being bolted on from outside.

There are two independent call paths that implement those four methods:

• LFCliBase (py-json, used by Realm and most scripts)
• BaseLFJsonRequest (lanforge_client/lanforge_api.py, not currently used by interop)

Rather than duplicating logging logic in both, the common pieces (CSV schema, enable/disable state, the actual record() call) live in one new module in lanforge_client/, and each call path just calls into it.

lanforge_client is the natural home since it's already a shared dependency of both.

---

Why global (module-level) state instead of a constructor parameter

The enable flag is deliberately process-wide global state in api_logger.py, not a parameter threaded through Realm / LFCliBase / profile constructors.

Reason:

Realm gets instantiated directly all over the py-scripts tree, and profile classes build their own nested instances.

For example:

• lf_webpage.py holds a Realm
• It also imports l4_cxprofile, which itself instantiates its own Realm

To pass an "enable logging" flag through the constructor, it would need to be threaded through every one of those call sites, and there's no guarantee the import/instantiation chain doesn't go a level or two deeper in other scripts.

A module-level configure() called once near the top of a script's main() avoids that — every LFCliBase / Realm or LFJsonCommand / LFJsonQuery created afterward in the same process just picks it up.

---

What's covered / not covered

Covered

• LFRequest.py now captures last_response_code and a one-line last_diagnostics summary (reason + X-Error-* headers) on every request, win or fail, so callers have something worth logging.
• lfcli_base.py (LFCliBase, used by Realm) and lanforge_api.py (BaseLFJsonRequest) both call api_logger.record(...) after every GET / POST / PUT / DELETE, using those diagnostics.

Not covered

• A small number of scripts don't go through json_get() / etc. at all — those aren't covered yet and will be migrated separately.

---

Pause / resume

monitor_for_runtime_csv() in lf_webpage.py polls in a tight loop and would otherwise flood the CSV with repetitive, rarely-useful entries.

Added:

• api_logger.pause()
• api_logger.resume()

so state (enabled + filename) is preserved but recording is suppressed during that loop by default.

--log_monitor_api_calls opts back in if someone actually wants those calls logged too.

---

Usage

lf_webpage.py ... \
    --save_api \
    [--api_log_file_name /path/to/log.csv] \
    [--log_monitor_api_calls] (this is somthing change wrt script ) 

---

Commits

• LFRequest.py — capture response code + diagnostics summary per request
• api_logger.py — new process-wide CSV logger (configure / record / is_enabled)
• lfcli_base.py — wire _log_api_call into json_get / json_post / json_put / json_delete
• lanforge_api.py — same, for the lanforge_client request path
• lf_webpage.py — add --save_api / --api_log_file_name flags, configure logger, copy CSV into report dir
• api_logger.py — add pause() / resume()
• lf_webpage.py — pause logging during the runtime-CSV monitor loop by default, add --log_monitor_api_calls override

---

---

Future Scope

The current changes improve the existing API logging by capturing additional request information and storing it in a structured format for easier debugging. Potential future enhancements include:

• Extend API call capture to scripts and code paths that do not currently use the common json_get(), json_post(), json_put(), and json_delete() interfaces.

• Record REST request latency alongside the existing response code and diagnostics to help identify slow responses.

• Capture additional system state when communication issues are detected, such as:

  • Device presence in /ports/list
  • Endpoint update timestamps
  • /resources/ response status
  • TX/RX byte counters and other activity metrics

• Add configurable troubleshooting indicators that detect missed report intervals and automatically record additional diagnostic information.

• Include high-level runtime summaries, such as:

  • Running, waiting, and stopped connection counts
  • Station state summary (associated, unassociated, with IP, down, phantom)
  • Distribution of endpoint update intervals
  • Average REST response time during the test run

• Improve the generated API logs with optional summaries or filtering to simplify correlating API activity with existing LANforge logs and test failures.

[Sidartha-CT]

Here is the log csv that generated out of this when we ran lf_webpage.py :
lf_api_calls.csv

[memnochproxy]

Might we consider:
from pathlib import Path
home_dir = Path.home()

[memnochproxy]

I would appreciate an alternative to using globals.

[memnochproxy]

I would appreciate an alternative to using globals. Can we turn these into class members?

[memnochproxy]

Is there any reason we cannot design these methods into an ApiLogger class?

[memnochproxy]

It looks like all of these changes will be lost on the next API generation. These changes need to live in btbits/client/candela/lanforge/json_api.py

[Sidartha-CT]

Thanks for pointing that out. Is this something that needs to be updated on the btbits side first? I don't have access to that repository, so I'm not very familiar with how the API generation flow works. If these changes need to be made there before they're generated into this repo, please let me know.

[memnochproxy]

"save_api" is a misleading term, please consider "enable_api_logging"

[smileyrekiere]

This is not the time the message is sent, it refers to the time that the csv is recording this data

[Sidartha-CT]

@memnochproxy ,

Why global (module-level) state instead of a constructor parameter?

The logging enable flag is intentionally kept as process-wide module state in api_logger.py instead of being passed through constructors.

Realm is instantiated in many places across the py-scripts tree, and profile classes often create their own nested Realm instances. Passing a logging flag through constructors would require updating every instantiation path, including nested ones, making the change invasive and harder to maintain.

By calling configure() once near the beginning of the script, every LFCliBase/Realm and LFJsonCommand/LFJsonQuery created afterward automatically uses the same configuration within the process.

For now, this approach achieves the required functionality with minimal changes. Once this PR is approved, I'll revisit the implementation and, if feasible, migrate it to a class-based design while preserving the same behavior.
++ @smileyrekiere