Skip to content

The jalpaca pty harness lives outside the repo, where #197 already established measurements should not #247

Description

@xoloki

The pty/pyte harness that drives jalpaca -- jalpaca_harness.py, its
utf8_test.py, smoke_test.py and rss_guard_probe.py -- lives in an agent's
private project directory under ~/.claude/projects/. It should be in
tests/harness/.

The argument is #197's, and it already came true once

#197 moved tools/perf into the tree because "the performance measurements
live outside the repo and are unreviewed; one was wrong by 15x." This harness
has the same property and a worked example: #185 was filed saying jalpaca drew
a streamed emoji as four pieces of garbage, and it does not -- ncurses
accumulates the partial sequence itself. Disproving that took the harness, and
the evidence cannot be reproduced by anyone reading the branch, because the
thing that produced it is not in the branch.

The harness also encodes two rules that were each paid for by a failure:

  • Assert on a rendered screen, never on the byte stream. A pty capture is a
    byte stream; stripping escapes discards ESC[K, so characters the terminal
    erased linger in the flattened text -- which produced both false failures and
    a false pass before the emulator went in.
  • Kill, then reap. close() SIGKILLs before waitpid, because a harness
    that only waited left the orphan holding 10.26 GB in the uninterruptible wait
    of jalpaca held 10.26 GB in an uninterruptible wait during a watchdog panic; does not reproduce #171. It returns False when the process could not be reaped, which is what
    a TH_UNINT wait looks like from outside.

Plus an RSS guard that raises above 3 GB, with a comment recording that the
limit was once raised to fit a test and put back. Those are exactly the notes
that belong where a reviewer reads them.

And there are two agents on jalpaca now. A harness in one of their home
directories diverges the way cuda::NeuralNetwork did in #138 -- silently,
because nothing builds it.

What it costs, which is the part to decide

pyte is not in the system python and PEP 668 blocks pip there, so it
needs a venv. That is a new kind of dependency for this tree, and it is the
only real objection.

The shape that fits what is already here: configure probes for python3 with
pyte importable and sets an AM_CONDITIONAL, the way HAVE_MAGICK and
HAVE_METAL gate their tests. Without it the test is not built; with it, the
test still exits 77 when the model file is absent, like the four
*_live_test programs that want a Dovecot or an nginx. Nobody's make check
breaks for want of a python package.

The tree is not hostile to this: jlib/ai/unicode.py is committed so the
tables can be re-derived, and tests/ already carries in-process harnesses --
httpserver.hh, mailserver.hh, audio_test.hh, certificate.hh.

What moves and what does not

Moves: jalpaca_harness.py, README.md, utf8_test.py, smoke_test.py,
rss_guard_probe.py, and the reply-path and interrupted-reply tests written
for #185.

Does not: the pr-*.txt drafts that have accumulated in the same directory,
and its perf/ copy, which tools/perf superseded in #197. The venv is built,
not committed.

One thing to fix on the way in: the README's paths still name the pre-move
project directory, so the setup instructions as written do not work.

What it would let make check say

That a byte-fallback character reaches the screen as a character, that an
interrupted reply does not eat the line break after it (#185), that Ctrl-A and
the other editing keys do what the help says, and that a run does not grow past
its guard. All of it needs a GGUF, so all of it skips where there is not one --
which is the same bargain #236 describes for jserve, and the same answer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions