Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
# Linux PGO builds link with rustup's rust-lld and fold identical functions (--icf=all). The Linux
# and macOS tsc ship without their symbol tables (stripped after the checks); the artifact
# unstripped-tsc-<platform> keeps the unstripped tsc for perf and profiles.
# The linux-x64 PGO tsc links lust's memfast.c in place of musl's memcpy, memmove, memcmp, bcmp and
# memchr (crates/ts_goport/scripts/memfast.sh): musl's memcmp and bcmp are byte loops.
# Windows (x86_64-pc-windows-msvc, aarch64-pc-windows-msvc) builds in Git Bash, with the C runtime
# linked in (+crt-static), so tsc.exe needs no Visual C++ redistributable. jemalloc does not build
# there: it uses the system allocator. Each PGO tsc builds natively, so its training runs on its
Expand All @@ -45,6 +47,7 @@ on:
- "scripts/goport/pgo-train.sh"
- "crates/ts_goport/scripts/copy-libs.sh"
- "crates/ts_goport/scripts/build-pgo.sh"
- "crates/ts_goport/scripts/memfast.*"
- "LICENSE"
- "NOTICE.md"
- "licenses/**"
Expand Down Expand Up @@ -179,6 +182,13 @@ jobs:
# jemalloc stops at start when the kernel page is bigger than its build page. arm64 kernels
# use 4, 16 or 64 KiB pages; 2^16 covers all three.
[[ $target != aarch64-* ]] || echo "JEMALLOC_SYS_WITH_LG_PAGE=16" >> "$GITHUB_ENV"
# linux-x64: lust's memfast.c in place of musl's memcpy, memmove, memcmp, bcmp and memchr
# (memfast.sh), in the PGO build only. The plain tsc keeps musl's, so the PGO check below
# also checks memfast against them.
if [[ $target == x86_64-unknown-linux-musl ]]; then
flags=$(crates/ts_goport/scripts/memfast.sh "$RUNNER_TEMP/memfast-wrap.o")
echo "PGO_RUSTFLAGS=$flags" | tee -a "$GITHUB_ENV"
fi
# The plain tsc: the reference of the PGO check below, or the shipped tsc with no-pgo. PGO_LINK is
# the link mode of both builds (build-pgo.sh): static on Linux, crt-static on Windows.
- name: Build
Expand Down Expand Up @@ -211,7 +221,7 @@ jobs:
if [[ $RUNNER_OS == Linux ]]; then
gccld="$(rustc --print sysroot)/lib/rustlib/$(rustc -vV | sed -n 's/^host: //p')/bin/gcc-ld"
[[ -x $gccld/ld.lld ]] || { echo "::error::no $gccld/ld.lld"; exit 1; }
export PGO_RUSTFLAGS="-Clink-arg=-fuse-ld=lld -Clink-arg=-B$gccld -Clink-arg=-Wl,--icf=all"
export PGO_RUSTFLAGS="${PGO_RUSTFLAGS:+$PGO_RUSTFLAGS }-Clink-arg=-fuse-ld=lld -Clink-arg=-B$gccld -Clink-arg=-Wl,--icf=all"
fi
PGO_TRAIN="scripts/goport/pgo-train.sh run $RUNNER_TEMP/train" crates/ts_goport/scripts/build-pgo.sh "$RUNNER_TEMP/pgo"
mkdir -p "$RUNNER_TEMP/out"
Expand Down Expand Up @@ -259,6 +269,12 @@ jobs:
grep -q "^error: cannot read the de-DE messages" check-missing.txt
if [[ $RUNNER_OS == Linux ]]; then
file "$dir/tsc" | tee /dev/stderr | grep -q "statically linked"
# memfast (linux-x64): the five __wrap_ functions, and none of musl's.
if [[ ${PGO_RUSTFLAGS:-} == *--wrap=* ]]; then
syms=$(nm "$dir/tsc" | grep -E ' [Tt] (__wrap_)?(memcpy|memmove|memcmp|bcmp|memchr)$' | awk '{print $3}' | sort | xargs)
echo "mem functions: $syms"
[[ $syms == "__wrap_bcmp __wrap_memchr __wrap_memcmp __wrap_memcpy __wrap_memmove" ]]
fi
# Static: it starts on musl (Alpine) too.
docker run --rm -v "$dir:/t:ro" alpine:3 /t/tsc --version
fi
Expand Down
11 changes: 11 additions & 0 deletions NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,14 @@ The collation and normalization tables in `crates/ts_goport/src/gostd/data` and
packages above.

License: [licenses/Unicode-LICENSE.txt](licenses/Unicode-LICENSE.txt).

## memfast

Copyright (c) 2026 Theo Browne. Licensed under the MIT License.

`crates/ts_goport/scripts/memfast.c` is `src/runtime/memfast.c` of
[t3dotgg/lust-lang](https://github.com/t3dotgg/lust-lang) (commit 6ced2b86fd), without change
below its header. The linux-x64 tsc links it in place of musl's memcpy, memmove, memcmp, bcmp and
memchr.

License: [licenses/memfast-LICENSE.txt](licenses/memfast-LICENSE.txt).
3 changes: 2 additions & 1 deletion crates/ts_goport/scripts/build-pgo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,8 @@
# PGO_RUSTFLAGS more RUSTFLAGS for both builds. The script sets
# RUSTFLAGS itself, so the caller's RUSTFLAGS do not
# reach cargo. release.yml passes the lld link args
# here (identical code folding on Linux).
# here (identical code folding on Linux, and memfast
# on linux-x64; memfast.sh).
#
# Choosing PGO_LINK (integrator, once a round): build the default and
# PGO_LINK=static. On cup2 (under its lock) time 'tsgo --version' and query
Expand Down
223 changes: 223 additions & 0 deletions crates/ts_goport/scripts/memfast.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
/* memfast.c from lust-lang (https://github.com/t3dotgg/lust-lang), the file
src/runtime/memfast.c at commit 6ced2b86fd66280cf262b6c8d6b0ff61c625d26b. The text
after this header is that file without change; its comment describes lust's build.

Copyright (c) 2026 Theo Browne
SPDX-License-Identifier: MIT
License: licenses/memfast-LICENSE.txt (NOTICE.md, "memfast").

ts-rust links it into the linux-x64 tsc (static musl) in place of musl's memcpy,
memmove, memcmp, bcmp and memchr: crates/ts_goport/scripts/memfast.sh. */
/* Fast memcpy, memmove, memcmp, bcmp and memchr for musl release builds (src/cc.rs
`memfast_unit`; the release driver links this unit next to mimalloc). musl's memcmp
and memchr are byte loops, and zig's compiler_rt memcpy and memmove (which a zig musl
link uses) move a word at a time: a release tsgo spent 2.3 G instructions in them on
the TypeORM check, where glibc's vector forms do the same work in about a tenth.
These use 16-byte vectors on x86_64 (SSE2, which every x86_64 has) with unaligned head
and tail blocks, and 8-byte words elsewhere. Results are libc's (memcmp returns the
difference of the first two bytes that differ), and no function reads a byte outside
the ranges it is given. The C compiler must not turn a loop here back into a call of
the same function: `MF_FN` asks clang for no builtins and gcc for no loop patterns.
The unit has no includes (a release unit names its own types). */
typedef __SIZE_TYPE__ mf_size;
typedef __UINTPTR_TYPE__ mf_uptr;
typedef __UINT64_TYPE__ mf_u64;
typedef __UINT32_TYPE__ mf_u32;
typedef unsigned char mf_u8;
/* Unaligned word access through types of alignment 1 (units build with
-fno-strict-aliasing). */
typedef mf_u64 mf_u64u __attribute__((aligned(1)));
typedef mf_u32 mf_u32u __attribute__((aligned(1)));
#define MF_LD8(p) (*(const mf_u64u *)(const void *)(p))
#define MF_ST8(p, v) (*(mf_u64u *)(void *)(p) = (v))
#define MF_LD4(p) (*(const mf_u32u *)(const void *)(p))
#define MF_ST4(p, v) (*(mf_u32u *)(void *)(p) = (v))

#if defined(__clang__)
#define MF_FN __attribute__((no_builtin))
#elif defined(__GNUC__)
#define MF_FN __attribute__((optimize("no-tree-loop-distribute-patterns")))
#else
#define MF_FN
#endif

#if defined(__x86_64__) && defined(__SSE2__) && (defined(__clang__) || defined(__GNUC__)) && !defined(MF_NO_V16)
#define MF_V16 1
typedef char mf_v16 __attribute__((vector_size(16)));
typedef char mf_v16u __attribute__((vector_size(16), aligned(1)));
#define MF_LDV(p) (*(const mf_v16u *)(const void *)(p))
#define MF_STV(p, v) (*(mf_v16u *)(void *)(p) = (v))
/* Bit i set when byte i of `v` (a comparison result, bytes 0 or -1) is set. */
#define MF_MASK(v) ((unsigned)__builtin_ia32_pmovmskb128((v)))
#endif

#if defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#define MF_LE 1
#endif

/* Up to 64 bytes: every load before any store, so memmove takes it for any overlap. */
MF_FN static inline __attribute__((always_inline)) void mf_copy_small(mf_u8 *d, const mf_u8 *s, mf_size n) {
if (n <= 16) {
if (n >= 8) {
mf_u64 a = MF_LD8(s), b = MF_LD8(s + n - 8);
MF_ST8(d, a);
MF_ST8(d + n - 8, b);
} else if (n >= 4) {
mf_u32 a = MF_LD4(s), b = MF_LD4(s + n - 4);
MF_ST4(d, a);
MF_ST4(d + n - 4, b);
} else if (n) {
mf_u8 a = s[0], b = s[n >> 1], c = s[n - 1];
d[0] = a;
d[n >> 1] = b;
d[n - 1] = c;
}
return;
}
#ifdef MF_V16
if (n <= 32) {
mf_v16 a = MF_LDV(s), b = MF_LDV(s + n - 16);
MF_STV(d, a);
MF_STV(d + n - 16, b);
return;
}
mf_v16 a = MF_LDV(s), b = MF_LDV(s + 16), c = MF_LDV(s + n - 32), e = MF_LDV(s + n - 16);
MF_STV(d, a);
MF_STV(d + 16, b);
MF_STV(d + n - 32, c);
MF_STV(d + n - 16, e);
#else
mf_u64 w[8];
mf_size k = 0;
for (; k + 8 <= n; k += 8) w[k >> 3] = MF_LD8(s + k);
mf_u64 t = MF_LD8(s + n - 8);
for (mf_size j = 0; j < k; j += 8) MF_ST8(d + j, w[j >> 3]);
MF_ST8(d + n - 8, t);
#endif
}

/* More than 64 bytes, front to back: right for disjoint ranges and for d below s. The
last 64 bytes are read first, as the loop may overwrite them when d is below s. */
MF_FN static inline __attribute__((always_inline)) void mf_copy_fwd(mf_u8 *d, const mf_u8 *s, mf_size n) {
#ifdef MF_V16
mf_v16 t0 = MF_LDV(s + n - 64), t1 = MF_LDV(s + n - 48), t2 = MF_LDV(s + n - 32), t3 = MF_LDV(s + n - 16);
for (mf_size i = 0; n - i > 64; i += 64) {
mf_v16 a = MF_LDV(s + i), b = MF_LDV(s + i + 16), c = MF_LDV(s + i + 32), e = MF_LDV(s + i + 48);
MF_STV(d + i, a);
MF_STV(d + i + 16, b);
MF_STV(d + i + 32, c);
MF_STV(d + i + 48, e);
}
MF_STV(d + n - 64, t0);
MF_STV(d + n - 48, t1);
MF_STV(d + n - 32, t2);
MF_STV(d + n - 16, t3);
#else
mf_u64 t = MF_LD8(s + n - 8);
for (mf_size i = 0; n - i > 8; i += 8) MF_ST8(d + i, MF_LD8(s + i));
MF_ST8(d + n - 8, t);
#endif
}

/* More than 64 bytes, back to front, for d above s: the first 64 bytes are read first. */
MF_FN static inline __attribute__((always_inline)) void mf_copy_bwd(mf_u8 *d, const mf_u8 *s, mf_size n) {
#ifdef MF_V16
mf_v16 h0 = MF_LDV(s), h1 = MF_LDV(s + 16), h2 = MF_LDV(s + 32), h3 = MF_LDV(s + 48);
for (mf_size i = n; i > 64; i -= 64) {
mf_v16 a = MF_LDV(s + i - 64), b = MF_LDV(s + i - 48), c = MF_LDV(s + i - 32), e = MF_LDV(s + i - 16);
MF_STV(d + i - 64, a);
MF_STV(d + i - 48, b);
MF_STV(d + i - 32, c);
MF_STV(d + i - 16, e);
}
MF_STV(d, h0);
MF_STV(d + 16, h1);
MF_STV(d + 32, h2);
MF_STV(d + 48, h3);
#else
mf_u64 h = MF_LD8(s);
for (mf_size i = n; i > 8; i -= 8) MF_ST8(d + i - 8, MF_LD8(s + i - 8));
MF_ST8(d, h);
#endif
}

MF_FN void *memcpy(void *d, const void *s, mf_size n) {
if (n <= 64) mf_copy_small((mf_u8 *)d, (const mf_u8 *)s, n);
else mf_copy_fwd((mf_u8 *)d, (const mf_u8 *)s, n);
return d;
}

MF_FN void *memmove(void *d, const void *s, mf_size n) {
if (n <= 64) mf_copy_small((mf_u8 *)d, (const mf_u8 *)s, n);
else if ((mf_uptr)d - (mf_uptr)s >= (mf_uptr)n) mf_copy_fwd((mf_u8 *)d, (const mf_u8 *)s, n);
else mf_copy_bwd((mf_u8 *)d, (const mf_u8 *)s, n);
return d;
}

/* The first index below n where a and b differ, from index i on, or n. */
MF_FN static inline __attribute__((always_inline)) mf_size mf_diff(const mf_u8 *a, const mf_u8 *b, mf_size i, mf_size n) {
#ifdef MF_V16
for (; n - i >= 16; i += 16) {
unsigned m = MF_MASK(MF_LDV(a + i) == MF_LDV(b + i)) ^ 0xFFFFu;
if (m) return i + (mf_size)__builtin_ctz(m);
}
if (i < n && n >= 16) {
/* The last 16 bytes; the ones below i are equal. */
mf_size t = n - 16;
unsigned m = MF_MASK(MF_LDV(a + t) == MF_LDV(b + t)) ^ 0xFFFFu;
return m ? t + (mf_size)__builtin_ctz(m) : n;
}
#endif
#ifdef MF_LE
for (; n - i >= 8; i += 8) {
mf_u64 x = MF_LD8(a + i) ^ MF_LD8(b + i);
if (x) return i + (mf_size)(__builtin_ctzll(x) >> 3);
}
#endif
for (; i < n; i++)
if (a[i] != b[i]) return i;
return n;
}

MF_FN int memcmp(const void *a, const void *b, mf_size n) {
const mf_u8 *x = (const mf_u8 *)a, *y = (const mf_u8 *)b;
mf_size i = mf_diff(x, y, 0, n);
return i < n ? (int)x[i] - (int)y[i] : 0;
}

/* bcmp(a, b, n): 0 when equal, else not 0 (the C compiler emits it for memcmp() == 0). */
MF_FN int bcmp(const void *a, const void *b, mf_size n) {
return mf_diff((const mf_u8 *)a, (const mf_u8 *)b, 0, n) < n;
}

MF_FN void *memchr(const void *s, int c, mf_size n) {
const mf_u8 *p = (const mf_u8 *)s;
mf_u8 b = (mf_u8)c;
mf_size i = 0;
#ifdef MF_V16
if (n >= 16) {
mf_v16 v = (mf_v16){0} + (char)b;
for (; n - i >= 64; i += 64) {
mf_v16 e0 = MF_LDV(p + i) == v, e1 = MF_LDV(p + i + 16) == v, e2 = MF_LDV(p + i + 32) == v, e3 = MF_LDV(p + i + 48) == v;
if (MF_MASK(e0 | e1 | e2 | e3)) {
mf_u64 m = (mf_u64)MF_MASK(e0) | (mf_u64)MF_MASK(e1) << 16 | (mf_u64)MF_MASK(e2) << 32 | (mf_u64)MF_MASK(e3) << 48;
return (void *)(p + i + (mf_size)__builtin_ctzll(m));
}
}
for (; n - i >= 16; i += 16) {
unsigned m = MF_MASK(MF_LDV(p + i) == v);
if (m) return (void *)(p + i + (mf_size)__builtin_ctz(m));
}
if (i < n) {
/* The last 16 bytes, without the ones below i (already checked). */
mf_size t = n - 16;
unsigned m = MF_MASK(MF_LDV(p + t) == v) & (0xFFFFu << (unsigned)(i - t));
if (m) return (void *)(p + t + (mf_size)__builtin_ctz(m));
}
return 0;
}
#endif
for (; i < n; i++)
if (p[i] == b) return (void *)(p + i);
return 0;
}
50 changes: 50 additions & 0 deletions crates/ts_goport/scripts/memfast.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
# Compiles memfast.c (lust's memcpy, memmove, memcmp, bcmp and memchr; MIT, see its header) for
# the static musl linux-x64 tsc, checks the object, and prints the rustc flags that link it in
# place of musl's functions. release.yml passes them to build-pgo.sh (PGO_RUSTFLAGS).
#
# Usage: memfast.sh <out.o>
# CC the C compiler (default musl-gcc)
#
# Why: musl's memcmp and bcmp are byte loops and its memcpy and memmove use rep movs. They took
# 4.4% of the cycles of a VS Code check and 9.2% on TypeORM; with memfast 2.0% and 3.3% (relmusl1,
# cycles:u --singleThreaded). Results are libc's: the tsc output does not change.
# The functions get the names __wrap_<name>, and the link wraps the five symbols
# (-Wl,--wrap=<name>). Plain names would clash with musl's in libc.a, because rustc puts
# -C link-arg after it. --wrap sends every call to these, musl's own calls too. memset stays
# musl's (asm).
# Checks: no undefined symbol (a loop that the compiler turns back into a memcpy call would call
# itself through the wrap), and no AVX or BMI code (the tsc runs on any x86-64).
set -euo pipefail

case "${1:-}" in
"" | help | -h | --help)
sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed -e '$d' -e 's/^# \{0,1\}//'
exit 2
;;
esac
out=$1
src="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/memfast.c"
names=(memcpy memmove memcmp bcmp memchr)
defs=() wraps=""
for n in "${names[@]}"; do
defs+=("-D$n=__wrap_$n")
wraps+="${wraps:+,}--wrap=$n"
done
# lust builds the unit with -fno-strict-aliasing (see the comment in memfast.c).
"${CC:-musl-gcc}" -O2 -fno-strict-aliasing "${defs[@]}" -c "$src" -o "$out"
undefined="$(nm -u "$out")"
if [[ -n $undefined ]]; then
echo "error: $out has undefined symbols: $undefined" >&2
exit 1
fi
# grep -c, not -q: with pipefail, a grep that stops early can fail the pipe (SIGPIPE).
if [[ $(objdump -d --no-show-raw-insn "$out" |
grep -cE $'\t(v[a-z0-9]+|andn|bextr|blsi|blsmsk|blsr|bzhi|pdep|pext|mulx|rorx|sarx|shlx|shrx) ') != 0 ]]; then
echo "error: $out has AVX or BMI code" >&2
exit 1
fi
for n in "${names[@]}"; do
[[ $(nm "$out" | grep -cE " T __wrap_$n\$") == 1 ]] || { echo "error: $out has no __wrap_$n" >&2; exit 1; }
done
echo "-Clink-arg=-Wl,$wraps -Clink-arg=$out"
21 changes: 21 additions & 0 deletions licenses/memfast-LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Theo Browne

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
4 changes: 3 additions & 1 deletion npm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,9 @@ package.

The CI builds (but darwin-x64 and win32-arm64) are PGO builds of the shipped profile (fat LTO):
`build-pgo.sh` trains each platform's tsc on the set of `scripts/goport/pgo-train.sh`, and the job
fails unless the PGO tsc gives the same output as a plain build on that set. They have no BOLT: BOLT
fails unless the PGO tsc gives the same output as a plain build on that set. The Linux x64 PGO tsc
links `crates/ts_goport/scripts/memfast.c` in place of musl's memcpy, memmove, memcmp, bcmp and
memchr (`memfast.sh`); the plain build keeps musl's. They have no BOLT: BOLT
refuses the static Linux bin, macOS has no perf branch sampling for its profile, and BOLT has no PE
(Windows) support. To make the Linux x64 tsc by hand on zbook, use `RELEASE_FEATURES=noembed
RELEASE_LIBC=musl RELEASE_PIE=0 crates/ts_goport/scripts/build-release.sh <out>/linux-x64` and pack
Expand Down
1 change: 1 addition & 0 deletions npm/pack.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ function portNotice() {
...part("TypeScript third-party notices", tsNoticeFile),
...part("Go license", path.join(repoRoot, "licenses", "Go-LICENSE.txt")),
...part("Unicode license", path.join(repoRoot, "licenses", "Unicode-LICENSE.txt")),
...part("memfast license", path.join(repoRoot, "licenses", "memfast-LICENSE.txt")),
].join("\n");
}
function writeLicense(dir) {
Expand Down
Loading