Skip to content

Commit 696cce3

Browse files
donislawdevclaude
andcommitted
format: archives can hold their files in directories
Two settings, shared by zip and tar.gz through internal/format/archive so the two containers cannot drift: depth says how many directories deep the files sit, and directory_entries says whether the archive also lists the directories themselves. Those are separate questions, because extractors differ - some create a directory when they meet a path that needs one, and some create only what the archive names. The default is flat, so no existing archive changes by a byte and no version needs bumping. Asking for directory_entries without a depth is refused naming both settings rather than quietly doing nothing. The ceiling of 50 is measured rather than picked. tar.gz pins USTAR, which carries a path in a 155 byte prefix and a 100 byte name split on a slash, so whether a path fits depends on where its slashes fall. Measured against archive/tar: depth 61 with a 12 byte name is taken at 256 bytes and depth 62 is refused at 260, which puts the real limit at 59 for the longest name this build makes. A guard asks archive/tar about every registered format rather than trusting that arithmetic. The size stays exact. zip counts by writing the container to a counting writer, so path lengths were already counted. tar.gz counts by formula, and a USTAR header is 512 bytes at every depth it accepts, so the formula needed no length term - only 512 per directory entry. Directories are not children. A child's seed is FileSeed(seed, index) over a running index, so a directory in that list would shift the seed of every file after it and rewrite its contents. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent e1a63a5 commit 696cce3

14 files changed

Lines changed: 825 additions & 37 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,30 @@ because it turns other people's test suites red.
7070

7171
### Added
7272

73+
- **An archive can hold its files in directories.** `--set depth=3` puts every
74+
file three levels down, and `--set directory_entries=true` also makes the
75+
archive list the directories themselves. Both work on `zip` and on `targz`.
76+
77+
Two settings rather than one, because they are two questions. Depth is about
78+
the paths inside. Directory entries are about whether the archive names the
79+
directories at all - and extractors differ there: some create a directory
80+
when they meet a path that needs one, and some create only what the archive
81+
names. An archive is the one format where you can test both.
82+
83+
The default is flat, which is what archives from this tool have always been,
84+
so **no existing file changes by a byte**. Asking for `directory_entries`
85+
without a depth is refused rather than quietly ignored: a flat archive has no
86+
directories to name, and the message says so and names both settings.
87+
88+
Depth goes up to 50. The limit is measured rather than picked: a `.tar.gz`
89+
writes USTAR headers, which carry a path in a 155 byte prefix and a 100 byte
90+
name split on a slash, and past a certain length no split works. Directories
91+
cost 512 bytes each in a `.tar.gz` and about 76 plus the path in a `.zip`.
92+
The size you order is still the size you get, to the byte.
93+
94+
The padding entry stays at the top of the archive rather than moving into the
95+
directories, so you can always tell it apart from the files you asked for.
96+
7397
- **A zip can be locked with ZipCrypto, the old scheme.** `--set encryption=zipcrypto`.
7498

7599
It is here for what it does to a reader rather than for what it protects. Measured: .NET's own `ZipFile` opens one of these, reports the entry at its true length, hands back a stream and fills it with the ENCRYPTED bytes - and never says the entry was encrypted at all. An application built on that library processes noise and calls it data. AES fails loudly in the same library, which is the safer defect and the less interesting one.

‎internal/format/archive/archive.go‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,19 @@ func mustSize(s string) int64 {
113113
// somebody kept them so by hand, and the comment saying so was the whole
114114
// mechanism.
115115
var axes = map[string]format.Property{
116+
Depth: {
117+
Name: Depth, Kind: format.PropertyInt,
118+
Min: 0, Max: maxDepth,
119+
Default: strconv.Itoa(defaultDepth),
120+
Detail: "How many directories deep the files inside sit. 0 puts them all at the top.",
121+
},
122+
DirectoryEntries: {
123+
Name: DirectoryEntries, Kind: format.PropertyBool,
124+
Default: "false",
125+
Detail: "Whether the archive also lists the directories themselves. " +
126+
"Most file browsers show the same folders either way, so the difference is in the entry list rather than on screen. " +
127+
"It matters for readers that only create a directory the archive names.",
128+
},
116129
Entries: {
117130
Name: Entries, Kind: format.PropertyInt,
118131
Min: 0, Max: maxEntries,

‎internal/format/archive/layout.go‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
package archive
2+
3+
import (
4+
"fmt"
5+
"strconv"
6+
"strings"
7+
8+
"github.com/donislawdev/TestingFilesGenerator/internal/format"
9+
)
10+
11+
// Where the files inside an archive sit, and whether the archive lists the
12+
// directories themselves.
13+
//
14+
// Both containers read this one description rather than each growing its own,
15+
// for the reason the package exists: zip and tar held six constants with
16+
// identical values and two identical readers before it, and the only thing
17+
// keeping them equal was a comment.
18+
//
19+
// Two settings rather than one, and the pairing is deliberate. A tester asking
20+
// "does the tool under test cope with nested paths" wants depth. A tester
21+
// asking "does it cope with an archive that does NOT list its directories"
22+
// wants the other, because extractors differ: some create a directory when
23+
// they meet a path that needs one, and some only create what the archive
24+
// names. An archive is the one format where those are separate questions.
25+
const (
26+
Depth = "depth"
27+
DirectoryEntries = "directory_entries"
28+
)
29+
30+
const (
31+
// defaultDepth is flat, and it has to stay flat. Every archive this tool
32+
// has written so far holds its files at the top, so any other default
33+
// would move the bytes of all of them - untouchable rule 3, and the reason
34+
// this change needs no version bump at all.
35+
defaultDepth = 0
36+
37+
// maxDepth is measured rather than chosen, and the measurement is about
38+
// tar rather than about taste.
39+
//
40+
// targz pins tar.FormatUSTAR (size.go), which carries a path in two
41+
// fields: a 155 byte prefix and a 100 byte name, split ON A SLASH. So a
42+
// path is writable when some slash leaves at most 155 before it and at
43+
// most 100 after it, which is a rule about where the slashes fall and not
44+
// about length. With the segments below, slashes sit every 4 bytes, the
45+
// last usable one is at 155, and the path therefore has to come to 256
46+
// bytes or fewer: 4*depth + len(entry name) <= 256.
47+
//
48+
// Measured 2026-09-01 against Go's own archive/tar with USTAR pinned:
49+
// depth 61 with a 12 byte name is taken at 256 bytes and depth 62 is
50+
// REFUSED at 260. The size is flat the whole way - 1536 B at every depth
51+
// up to the refusal, no hidden step - so the tar arithmetic needs no
52+
// length term at all.
53+
//
54+
// 61 is therefore the ceiling for a 12 byte name and NOT the ceiling to
55+
// declare, because the entry name is not always 12 bytes. The longest one
56+
// this build can produce is targz_0001.tar.gz at 17, which lands the limit
57+
// at 59. Fifty leaves room for a name of 56 bytes, which is far past
58+
// anything the registry holds, and a guard proves it for every registered
59+
// format rather than trusting this paragraph.
60+
maxDepth = 50
61+
62+
// dirSegment numbers the levels so a path reads as what it is. Two digits
63+
// because maxDepth is two digits, and a fixed width so every segment is
64+
// the same size and the arithmetic above stays a multiplication.
65+
dirSegment = "d%02d/"
66+
67+
// dirSegmentBytes is what one segment comes to once rendered - "d00/" is
68+
// four bytes where the format string above is six. Written out rather than
69+
// taken as len(dirSegment), which is the bug the depth guard caught the
70+
// first time it ran: the arithmetic said every path was 2 bytes per level
71+
// longer than it is, which would have understated the ceiling rather than
72+
// overstating it, so nothing would have failed until somebody widened the
73+
// segment. A guard compares this against a really rendered path.
74+
dirSegmentBytes = 4
75+
)
76+
77+
// Layout is what the two settings come to once read.
78+
type Layout struct {
79+
// Depth is how many directories deep the files sit. Zero is flat.
80+
Depth int
81+
// DirEntries says whether the archive also names the directories.
82+
DirEntries bool
83+
}
84+
85+
// Path is where an entry called name sits under this layout.
86+
//
87+
// The empty name gives the directory chain itself with its trailing slash,
88+
// which is what both containers want a directory entry to be called.
89+
func (l Layout) Path(name string) string {
90+
if l.Depth <= 0 {
91+
return name
92+
}
93+
var b strings.Builder
94+
b.Grow(l.Depth*len(dirSegment) + len(name))
95+
for i := 0; i < l.Depth; i++ {
96+
fmt.Fprintf(&b, dirSegment, i)
97+
}
98+
b.WriteString(name)
99+
return b.String()
100+
}
101+
102+
// Directories is every directory this layout creates, outermost first.
103+
//
104+
// Outermost first because that is the order an extractor wants to meet them
105+
// in: a reader that creates directories as it goes cannot make d00/d01 before
106+
// it has made d00. It is empty when nothing was asked for, so a caller can
107+
// range over it without asking whether the setting is on.
108+
func (l Layout) Directories() []string {
109+
if !l.DirEntries || l.Depth <= 0 {
110+
return nil
111+
}
112+
out := make([]string, 0, l.Depth)
113+
for i := 1; i <= l.Depth; i++ {
114+
out = append(out, Layout{Depth: i}.Path(""))
115+
}
116+
return out
117+
}
118+
119+
// LongestPath is the longest path this layout can produce for an entry name of
120+
// the given length. It exists for the guard that proves maxDepth is safe.
121+
func LongestPath(depth, nameLen int) int {
122+
return depth*dirSegmentBytes + nameLen
123+
}
124+
125+
// MaxDepth is the deepest nesting this build offers, for the guard that checks
126+
// the declaration against what tar will actually take.
127+
func MaxDepth() int { return maxDepth }
128+
129+
// ReadLayout works out where the files go, and refuses a pair that cannot mean
130+
// anything.
131+
//
132+
// directory_entries with a flat archive is the pair, and it is a refusal
133+
// rather than a setting quietly doing nothing. There are no directories in a
134+
// flat archive, so the answer would be the same whichever way it was set - and
135+
// rule 6 forbids exactly that silence. The message names BOTH halves, because
136+
// a reader who set one of them cannot tell from "directory_entries is not
137+
// allowed" which one to change.
138+
//
139+
// It is reachable from the window as well as from a recipe, which is why it
140+
// has to be a good message rather than an internal check: the control is a
141+
// checkbox, a checkbox always sends its value, and somebody can tick it while
142+
// depth is still nought.
143+
func ReadLayout(id string, r format.Request) (Layout, error) {
144+
depth, err := intProperty(id, r.Properties, Depth, defaultDepth, 0, maxDepth)
145+
if err != nil {
146+
return Layout{}, err
147+
}
148+
dirs, err := boolProperty(id, r.Properties, DirectoryEntries, false)
149+
if err != nil {
150+
return Layout{}, err
151+
}
152+
if dirs && depth == 0 {
153+
return Layout{}, &format.PropertyValueError{
154+
Format: id,
155+
Key: DirectoryEntries,
156+
Value: "true",
157+
Reason: "a flat archive has no directories to list, and " + Depth + " is 0",
158+
Remedy: "Ask for " + Depth + " of 1 or more, or leave " + DirectoryEntries + " off.",
159+
}
160+
}
161+
return Layout{Depth: depth, DirEntries: dirs}, nil
162+
}
163+
164+
// boolProperty reads a true or false setting.
165+
//
166+
// The registry has already refused anything that is not true or false by the
167+
// time a generator runs, since the declaration says the kind. This repeats the
168+
// check for the same reason intProperty repeats its range: a caller reaching
169+
// the generator directly is not going through the registry.
170+
func boolProperty(id string, props map[string]string, key string, fallback bool) (bool, error) {
171+
raw, ok := props[key]
172+
if !ok || raw == "" {
173+
return fallback, nil
174+
}
175+
v, err := strconv.ParseBool(strings.ToLower(raw))
176+
if err != nil {
177+
return false, &format.PropertyValueError{
178+
Format: id,
179+
Key: key,
180+
Value: raw,
181+
Reason: "it takes true or false",
182+
Remedy: "Write " + key + ": true or " + key + ": false.",
183+
}
184+
}
185+
return v, nil
186+
}

‎internal/format/targz/size.go‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,16 @@ func roundUpBlock(n int64) int64 {
5959
// tarLength is the length of the tar stream before it reaches gzip.
6060
func tarLength(m memo) int64 {
6161
total := int64(2 * tarBlock) // the end of archive marker
62+
// A directory is a header and nothing else, so it costs exactly one block.
63+
// Measured 2026-09-01 against archive/tar rather than read off the format:
64+
// a tar holding one directory and nothing else comes to 1536 B, of which
65+
// 1024 is the end of archive marker. A guard holds that number.
66+
//
67+
// The path length does NOT appear here, and that is measured too. USTAR
68+
// splits a path across a 155 byte prefix and a 100 byte name, so a header
69+
// is the same 512 bytes at every depth it accepts - flat all the way to
70+
// the refusal, with no step. maxDepth is what keeps it on the near side.
71+
total += int64(len(m.layout.Directories())) * tarBlock
6272
for _, c := range m.children {
6373
total += tarBlock + roundUpBlock(c.plan.Bytes)
6474
}
@@ -279,6 +289,20 @@ func build(ctx context.Context, w io.Writer, m memo) error {
279289
}
280290
tw := tar.NewWriter(zw)
281291

292+
// Directories first, outermost first, and only when asked for. They are
293+
// not in m.children on purpose: a child's seed is FileSeed(seed, index)
294+
// over a running index, so a directory in that list would shift the seed
295+
// of every file after it and rewrite its contents. That is untouchable
296+
// rule 2 - an edit in one place moving the bytes in another.
297+
for _, dir := range m.layout.Directories() {
298+
if err := ctx.Err(); err != nil {
299+
return err
300+
}
301+
if err := writeDirectory(tw, dir, m.own); err != nil {
302+
return fmt.Errorf("targz: the directory %q could not be named: %w", dir, err)
303+
}
304+
}
305+
282306
for _, c := range m.children {
283307
if err := ctx.Err(); err != nil {
284308
return err
@@ -334,6 +358,34 @@ func writeEntry(ctx context.Context, tw *tar.Writer, e tarEntry, body func(io.Wr
334358
return body(tw)
335359
}
336360

361+
// writeDirectory names one directory in the tar.
362+
//
363+
// The mode is 0755 rather than own.Mode, and that is a decision rather than an
364+
// oversight. entry_mode is declared as "the permissions recorded for each file
365+
// inside", and a directory is not a file - recording 644 on one would produce
366+
// an archive that extracts into directories nothing can be written into, which
367+
// is a surprise nobody asked this setting for. The owner fields DO follow
368+
// entry_owner, so an archive that says everything belongs to root says it about
369+
// the directories too.
370+
//
371+
// It costs exactly one block and no more, which is the whole of what the
372+
// arithmetic above had to learn about it. Measured rather than read off the
373+
// format, and a guard holds the number.
374+
func writeDirectory(tw *tar.Writer, name string, own archive.Ownership) error {
375+
return tw.WriteHeader(&tar.Header{
376+
Name: name,
377+
Size: 0,
378+
Mode: 0o755,
379+
Uid: own.Uid,
380+
Gid: own.Gid,
381+
Uname: own.Uname,
382+
Gname: own.Gname,
383+
ModTime: fixedTime,
384+
Typeflag: tar.TypeDir,
385+
Format: tar.FormatUSTAR,
386+
})
387+
}
388+
337389
// tarEntry is one entry's header, as both the measuring pass and the
338390
// writing pass describe it.
339391
//

‎internal/format/targz/targz.go‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ func init() {
116116
// package. Listed rather than received whole, so a format takes only
117117
// the axes it can actually carry.
118118
Properties: archive.Axes(archive.Entries, archive.EntryFormat, archive.EntrySize,
119+
archive.Depth, archive.DirectoryEntries,
119120
archive.EntryMode, archive.EntryOwner),
120121

121122
// Neither half of this format has anywhere to put a password, and
@@ -164,6 +165,11 @@ type memo struct {
164165
// The zero value is not the default - ReadOwnership fills it, because
165166
// the mode this format has always written is 644 rather than 0.
166167
own archive.Ownership
168+
// layout is where the files inside sit. It has to be here rather than an
169+
// argument because tarLength counts from this struct and build writes
170+
// from it: a layout the two disagreed about would promise one size and
171+
// write another.
172+
layout archive.Layout
167173
// withExtra says whether the header carries a gzip extra field at all,
168174
// and extraLen says how many bytes it holds. Two fields rather than one
169175
// with a sentinel, matching withFiller beside them, because an EMPTY
@@ -185,8 +191,13 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
185191
return format.Plan{}, err
186192
}
187193

188-
m := memo{seed: r.Seed, own: own}
189-
if m.children, err = planChildren(r, groups); err != nil {
194+
layout, err := archive.ReadLayout("targz", r)
195+
if err != nil {
196+
return format.Plan{}, err
197+
}
198+
199+
m := memo{seed: r.Seed, own: own, layout: layout}
200+
if m.children, err = planChildren(r, groups, layout); err != nil {
190201
return format.Plan{}, err
191202
}
192203

@@ -209,7 +220,7 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
209220
// Members are numbered across the whole archive rather than per group, so the
210221
// seed of a member does not move when a group above it changes count. That is
211222
// untouchable rule 2 applied one level down.
212-
func planChildren(r format.Request, groups []format.Content) ([]child, error) {
223+
func planChildren(r format.Request, groups []format.Content, layout archive.Layout) ([]child, error) {
213224
var out []child
214225
index := 0
215226
// Numbering runs per format rather than per group, so two groups of the
@@ -231,7 +242,7 @@ func planChildren(r format.Request, groups []format.Content) ([]child, error) {
231242
}
232243
numbered[g.Format]++
233244
out = append(out, child{
234-
name: fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension),
245+
name: layout.Path(fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension)),
235246
desc: desc,
236247
plan: cp,
237248
})
@@ -290,7 +301,12 @@ func describe(target int64, label string, m memo, groups []format.Content) forma
290301
// Stored rather than deflated, which is what makes the size exact
291302
// in one pass. Stated here so a test can assert on it rather than
292303
// infer it from how well the file compresses.
293-
"compression": "none",
304+
"compression": "none",
305+
// Where the files sit, and whether the directories are named -
306+
// written every time rather than only when nested, so a harness
307+
// never has to read a missing key as flat.
308+
archive.Depth: m.layout.Depth,
309+
archive.DirectoryEntries: m.layout.DirEntries,
294310
format.PropertyLabelEmbedded: label != "",
295311
},
296312
}

0 commit comments

Comments
 (0)