Some task_cleanroom Docker images do not contain all documentation

[kolasleep]

I looked at several Docker images for task_cleanroom, and I only found executable files but no doc files. Why is that?
For example: https://hub.docker.com/layers/programbench/zk-org_1776_zk.10d93d5/task_cleanroom/images/sha256-74335dee5a7c847cd3485cebdb753559991c02471117a68da6aab20f54e0bf9d

[klieret]

Hi @kolasleep , appreciate the report, I looked into this a bit. It seems like sometimes the LM generated clean.sh script we used to remove source and implementation details got somewhat overzealous and removed the entire docs/. README.md and LICENSE.md are still there though.

[kolasleep]

Hi @kolasleep , appreciate the report, I looked into this a bit. It seems like sometimes the LM generated clean.sh script we used to remove source and implementation details got somewhat overzealous and removed the entire docs/. README.md and LICENSE.md are still there though.

Hi, thanks for looking into this and explaining what happened. That makes sense. Hope the overzealous cleanup issue can be fixed soon. Thanks again for the update!

[klieret]

@WXDawaken in #15 reports this with ffmpeg, adding this here

It confused me that there are no domain specific documents like codec/format spec provided in the ffmpeg cleanroom, there are only .flac and .wav assets, and the README.md is basically nonsense for agent.
So the agent has to generate the assets itself, and try to analyze the outcome from original executable to extract the original decoding algo like h264?
This seems related to #7 . In the ffmpeg cleanroom, doc/ appears to be removed, while the remaining README only points to online docs that agents cannot access. Since the tests cover many codec/container behaviors, could you clarify whether ffmpeg docs/specs were intentionally excluded, or whether this is an over-cleaning bug?

[klieret]

In the meantime I analyzed all 200 instances and it seems like this overzealous cleaning happened quite a few times. Planning to address this very soon.

[WXDawaken]

@klieret , Thanks for looking into the over-cleaning issue. One thing I’m still unclear about is the intended cleaning/documentation policy.

I let codex do some exploration. Some repositories have doc/ directories that mix very different kinds of material. For example, FFmpeg’s doc/ contains user-facing .texi manuals, but also doc/examples/*.c and doc/print_options.c. zstd similarly has doc/zstd_compression_format.md, which seems like valid format documentation, but also doc/educational_decoder/zstd_decompress.c, which looks like a reference implementation.

So deleting all of doc/ is too aggressive, but keeping all of doc/ may leak implementation details.

Could you clarify the intended policy for ProgramBench cleanrooms?

1. Do cleanrooms use any per-repository documentation allowlist or manual review to decide which files are valid documentation?
2. If not, would it make sense to document the policy and expose the generated clean.sh scripts / file manifests for review?
3. Should cleanrooms include only documentation present in the original repository?
4. Should external domain specs be included when behavioral tests rely on external formats/protocols/codecs?
5. If only repo docs are restored, should tasks like FFmpeg still be considered valid under the “binary + documentation, no internet” setting?

This matters because tasks like FFmpeg may remain much more dependent on prior domain knowledge than ordinary missing-doc cases, even after the repo docs are restored.

Anyway, the concept of ProgramBench is great still.