feat: Fetch logs through the CLI

python/docs/cli/index.rst

@@ -59,6 +59,7 @@ Common Workflows
 
 * Develop interactively with :doc:`studio`.
 * Submit and inspect training or batch work with :doc:`job` and :doc:`mmt`.
+* Tail, follow, and search the logs of any of them with :doc:`logs`.
 * Build and operate inference services with :doc:`deployment` and :doc:`model`.
 * Move data and artifacts with :doc:`file`, :doc:`folder`, :doc:`container`, and :doc:`cp`.
 * Configure accounts, organizations, teamspaces, cloud accounts, and SSH with
@@ -79,6 +80,7 @@ reference.
    mmt
    machine
    deployment
+   logs
    container
    model
    api-key

python/docs/cli/logs.rst

@@ -0,0 +1,8 @@
+Logs
+++++
+
+.. lightning-reference:: logs
+   :root-label: lightning logs
+   :anchor-prefix: logs
+
+   from lightning_sdk.cli.logs import logs

python/docs/sdk/examples.rst

@@ -29,4 +29,5 @@ CLI examples
    examples/mmts-cli
    examples/teamspaces-cli
    examples/sandboxes-cli
+   examples/logs-cli
    examples/api-cli

python/docs/sdk/examples/logs-cli.rst

@@ -0,0 +1 @@
+.. include:: ../../../examples/logs_cli.rst

python/examples/README.md

@@ -69,6 +69,7 @@ export LIGHTNING_SANDBOX_API_KEY="..."
 | [`mmts_cli.rst`](mmts_cli.rst)             | Launch and inspect multi-machine training runs from the CLI                         |
 | [`teamspaces_cli.rst`](teamspaces_cli.rst) | Set CLI context and pass explicit teamspace scope to resource commands              |
 | [`sandboxes_cli.rst`](sandboxes_cli.rst)   | Create sandboxes, run commands, inspect logs, stop, resume, and delete from the CLI |
+| [`logs_cli.rst`](logs_cli.rst)             | Tail, follow, search, and page logs for jobs, MMTs, deployments, and sandboxes      |
 | [`api_cli.rst`](api_cli.rst)               | Call authenticated Lightning API endpoints directly from shell scripts              |
 
 # Running examples

python/examples/jobs.rst

@@ -59,7 +59,13 @@ Operational notes
 
 - ``Job.run`` creates a new job; ``Job("name", teamspace=...)`` fetches an
   existing one.
-- ``job.logs`` is available after the job reaches a terminal state.
+- ``job.logs`` reads the logs saved so far, whether the job is running or
+  finished. Call it to pass options: ``job.logs(follow=True)`` streams new lines
+  from a running job, and ``tail``, ``timestamps``, ``query`` (only lines
+  containing every whitespace-separated term) and ``severity`` (``error``,
+  ``warning``, ``info`` or ``debug`` and above) narrow what comes back.
+- ``mmt.logs`` reads every machine of a multi-machine job, merged into one
+  timeline and labelled with the machine each line came from.
 - Studio-backed jobs must run in the same teamspace and cloud account as the
   Studio.
 - Container-backed jobs cannot also pass ``studio=``.

python/examples/logs_cli.rst

@@ -0,0 +1,150 @@
+Logs CLI examples
+=================
+
+``lightning logs`` reads logs for anything that runs in a teamspace — jobs,
+multi-machine jobs, deployments, and sandboxes — with one set of flags. Use it
+to tail a run from a terminal, search a finished run for an error, or page
+through a long history in a script.
+
+Prerequisites
+-------------
+
+.. code-block:: console
+
+   $ pip install lightning-sdk -U
+   $ lightning login
+   $ lightning config set teamspace owner/teamspace
+
+Every example below also accepts ``--teamspace owner/teamspace`` explicitly; pass
+it when you work across several teamspaces or run in CI.
+
+Pick what to read
+-----------------
+
+Name or id, one resource at a time:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job
+   $ lightning logs --job-id job-1234
+   $ lightning logs --mmt-name sdk-tutorial-mmt
+   $ lightning logs --deployment-name sdk-tutorial-api
+   $ lightning logs --sandbox-id sbx-42
+
+``--job-id`` can be repeated to merge several jobs into one timeline:
+
+.. code-block:: console
+
+   $ lightning logs --job-id job-1234 --job-id job-5678
+
+Multi-machine jobs and multi-replica deployments are merged the same way, and
+each line is labelled with the machine or replica it came from.
+
+Tail and follow
+---------------
+
+``--tail`` prints the last N lines; ``--follow`` keeps the stream open until the
+resource finishes or you press Ctrl-C:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job --tail 100
+   $ lightning logs --deployment-name sdk-tutorial-api --follow
+   $ lightning logs --mmt-name sdk-tutorial-mmt --tail 50 --follow
+
+Search and narrow
+-----------------
+
+``--query`` matches text (matches are highlighted), ``--severity`` sets the
+minimum level (``error`` > ``warning`` > ``info`` > ``debug``), and
+``--since``/``--until`` bound the time range. Time bounds take a duration —
+``30s``, ``5m``, ``2h``, ``3d``, ``1w`` — or an RFC3339 timestamp:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job --query "CUDA out of memory"
+   $ lightning logs --deployment-name sdk-tutorial-api --severity warning --since 2h
+   $ lightning logs --mmt-name sdk-tutorial-mmt -q error --since 3d --until 1d
+   $ lightning logs --job-name sdk-tutorial-job --since 2026-01-01T00:00:00Z
+
+Filters combine with ``--tail`` and ``--follow``, so you can watch only what
+matters:
+
+.. code-block:: console
+
+   $ lightning logs --deployment-name sdk-tutorial-api --severity error --follow
+
+Sandboxes
+---------
+
+A sandbox reads by id or name, and ``--sandbox-command-id`` narrows to a single
+detached command:
+
+.. code-block:: console
+
+   $ lightning logs --sandbox-id sbx-42
+   $ lightning logs --sandbox-id sbx-42 --sandbox-command-id cmd-abc123 --no-timestamps
+
+Looking a sandbox up by ``--sandbox-name`` lists sandboxes, which needs a
+teamspace- or org-scoped API key:
+
+.. code-block:: console
+
+   $ export LIGHTNING_SANDBOX_API_KEY="..."
+   $ lightning logs --sandbox-name sdk-tutorial-sandbox --tail 20
+
+``--sandbox-id`` works with a normal ``lightning login``.
+
+Page through a long history
+---------------------------
+
+Without ``--tail``/``--follow``, results come back oldest-first in pages.
+``--limit`` sets the page size; the command prints the next-page command on
+stderr, so piping stdout stays clean:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job --limit 500
+   $ lightning logs --job-name sdk-tutorial-job --limit 500 --page-token <token>
+
+``--tail`` and ``--limit`` are mutually exclusive (one reads backwards from the
+end, the other forwards from the start), and ``--page-token`` reads one fixed
+page, so it cannot be combined with ``--follow``.
+
+Scripting
+---------
+
+``--json`` emits entries plus the next page token, and ``--no-timestamps`` drops
+the timestamp prefix for grep-friendly output:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job --limit 200 --json > logs.json
+   $ lightning logs --job-name sdk-tutorial-job --json | jq -r '.entries[].message'
+   $ lightning logs --job-name sdk-tutorial-job --no-timestamps | grep -c Traceback
+
+Fail a CI step when a run logged an error:
+
+.. code-block:: console
+
+   $ lightning logs --job-name sdk-tutorial-job --severity error --limit 1 --json \
+       | jq -e '.entries | length == 0'
+
+Shortcuts
+---------
+
+``lightning job logs`` and ``lightning deployment logs`` resolve the resource for
+you and print through the same reader:
+
+.. code-block:: console
+
+   $ lightning job logs sdk-tutorial-job --follow --timestamps
+   $ lightning job logs sdk-tutorial-job --query error --severity error
+   $ lightning deployment logs sdk-tutorial-api --tail 100
+
+See also
+--------
+
+- ``jobs_cli.rst`` for submitting and inspecting the jobs these logs come from.
+- ``sandboxes_cli.rst`` for running detached sandbox commands.
+- ``lightning logs --help`` for the full option reference.

python/lightning_sdk/api/deployment_api.py

@@ -7,12 +7,17 @@
 from pathlib import Path, PurePosixPath
 from tempfile import TemporaryDirectory
 from time import sleep
-from typing import Any, Dict, List, Literal, Optional, Sequence, TextIO, Tuple, Union
+from typing import Any, Dict, Iterator, List, Literal, Optional, Sequence, TextIO, Tuple, Union
+from urllib.parse import urlparse
 
 import requests
 
 from lightning_sdk.api import lightning_storage_upload as lightning_storage_upload_api
-from lightning_sdk.api.utils import _BlobUploader, _machine_to_compute_name, resolve_path_mappings
+from lightning_sdk.api.logs_api import LogEntry, parse_log_entries
+from lightning_sdk.api.utils import _BlobUploader, _get_cloud_url, _machine_to_compute_name, resolve_path_mappings
+
+# `Auth` is the deployment endpoint auth type in this module, so the login client is aliased.
+from lightning_sdk.lightning_cloud.login import Auth as _LoginAuth
 from lightning_sdk.lightning_cloud.openapi import (
     JobsServiceCreateDeploymentBody,
     JobsServiceReloadDeploymentWeightsBody,
@@ -621,6 +626,54 @@ def get_job_logs(
         kwargs = {k: v for k, v in kwargs.items() if v is not None}
         return self._client.jobs_service_get_job_logs(project_id=teamspace_id, id=job_id, **kwargs)
 
+    def iter_job_log_entries(
+        self,
+        teamspace_id: str,
+        job_id: str,
+        *,
+        deployment_id: Optional[str] = None,
+        since: Optional[str] = None,
+        until: Optional[str] = None,
+        rank: Optional[int] = None,
+    ) -> Iterator[LogEntry]:
+        """Yield a job's saved log lines from the legacy per-job log pages.
+
+        :meth:`get_job_logs` returns page metadata only, so each page is downloaded from its
+        signed URL and parsed here. Prefer :class:`~lightning_sdk.api.logs_api.LogsApi`, which
+        returns lines inline; this path exists for reads the merged logs API cannot serve yet
+        (currently only ``rank``).
+
+        Args:
+            teamspace_id: The teamspace that owns the job.
+            job_id: The job ID.
+            deployment_id: Optional deployment ID filter.
+            since: Optional start timestamp.
+            until: Optional end timestamp.
+            rank: Optional distributed job rank.
+
+        Yields:
+            LogEntry: The saved log lines, page by page.
+        """
+        logs = self.get_job_logs(
+            teamspace_id,
+            job_id,
+            deployment_id=deployment_id,
+            since=since,
+            until=until,
+            rank=rank,
+        )
+        session = requests.Session()
+        session.headers.update({"Authorization": _LoginAuth().authenticate()})
+        for page in logs.pages or []:
+            url = getattr(page, "url", None)
+            if not url:
+                continue
+            if not urlparse(url).netloc:
+                url = f"{_get_cloud_url().rstrip('/')}/{url.lstrip('/')}"
+            response = session.get(url, timeout=60)
+            response.raise_for_status()
+            yield from parse_log_entries(response.text)
+
     def stop(self, deployment: V1Deployment) -> V1Deployment:
         """Scale a deployment to zero replicas and wait until all replicas have stopped.