Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package sk.ainet.buildlogic.npm

/**
* Which Yarn root — and so which committed lockfile — a pin applies to.
*
* Most pins want both, which is the default. Scope a pin when the package exists in
* only one of the two dependency graphs, because Yarn writes a lockfile entry for
* **every** `resolutions` key whether or not anything in that graph requests the
* package. An unscoped `webpack` pin therefore adds webpack and its ~76 transitive
* packages to `kotlin-js-store/wasm/yarn.lock`, which bundles nothing with webpack —
* they would be downloaded and installed by the wasm build for no reason, and
* Dependabot would start reporting them against a lockfile that never uses them.
*
* Check which graph actually holds a package before scoping a pin:
*
* ```
* grep -c '^webpack@' kotlin-js-store/yarn.lock kotlin-js-store/wasm/yarn.lock
* ```
*/
enum class NpmPinTarget {

/** `kotlin-js-store/yarn.lock`, driven by the `js` target's Yarn root. */
JS,

/** `kotlin-js-store/wasm/yarn.lock`, driven by the `wasmJs` target's Yarn root. */
WASM,

;

companion object {

/** The default scope of [NpmPinsExtension.pin]: force the version everywhere. */
val ALL: Set<NpmPinTarget> = entries.toSet()
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -31,30 +31,48 @@ import org.gradle.api.provider.Provider
* pin("socket.io", libs.versions.npm.socketio)
* pin("@types/node", libs.versions.npm.typesNode)
* ```
*
* ## Scoping a pin to one lockfile
*
* A pin applies to both Yarn roots unless told otherwise. Pass [NpmPinTarget] values
* when the package lives in only one graph, so the other lockfile does not acquire an
* entry — and a whole transitive tree — for a package it never resolves:
*
* ```kotlin
* pin("webpack", libs.versions.npm.webpack, NpmPinTarget.JS)
* ```
*/
abstract class NpmPinsExtension {

/**
* Package name -> exact version, as declared by [pin]. Consumed by
* Package name -> exact version for `kotlin-js-store/yarn.lock`. Consumed by
* [NpmPinsPlugin] and [VerifyNpmPinsTask]; declare pins through [pin] rather than
* mutating this directly, which skips validation and duplicate detection.
*/
abstract val pins: MapProperty<String, String>
abstract val jsPins: MapProperty<String, String>

/** Package name -> exact version for `kotlin-js-store/wasm/yarn.lock`. See [jsPins]. */
abstract val wasmPins: MapProperty<String, String>

/**
* Whether `verifyNpmPins` fails when a pinned package is absent from every
* lockfile. Defaults to `false`: a pin that outlives its package (because the
* Kotlin toolchain dropped the dependency) is stale rather than broken, and
* Whether `verifyNpmPins` fails when a pinned package is absent from the lockfile
* it is scoped to. Defaults to `false`: a pin that outlives its package (because
* the Kotlin toolchain dropped the dependency) is stale rather than broken, and
* should be reported without breaking the build.
*/
abstract val failOnMissingPackage: Property<Boolean>

private val declared = mutableSetOf<String>()

/** Pins [packageName] to a version held in the version catalog. */
fun pin(packageName: String, version: Provider<String>) {
/**
* Pins [packageName] to a version held in the version catalog.
*
* With no [targets] the pin applies to every lockfile; name them to restrict it.
*/
fun pin(packageName: String, version: Provider<String>, vararg targets: NpmPinTarget) {
val name = validatePackageName(packageName)
pins.put(name, version.map { validateVersion(name, it) })
val checked = version.map { validateVersion(name, it) }
scopeOf(targets).forEach { target -> mapFor(target).put(name, checked) }
}

/**
Expand All @@ -63,9 +81,18 @@ abstract class NpmPinsExtension {
* Prefer the [Provider] overload — a number in `libs.versions.toml` is visible to
* dependency-update tooling, a number in the build script is not.
*/
fun pin(packageName: String, version: String) {
fun pin(packageName: String, version: String, vararg targets: NpmPinTarget) {
val name = validatePackageName(packageName)
pins.put(name, validateVersion(name, version))
val checked = validateVersion(name, version)
scopeOf(targets).forEach { target -> mapFor(target).put(name, checked) }
}

private fun scopeOf(targets: Array<out NpmPinTarget>): Set<NpmPinTarget> =
if (targets.isEmpty()) NpmPinTarget.ALL else targets.toSet()

private fun mapFor(target: NpmPinTarget): MapProperty<String, String> = when (target) {
NpmPinTarget.JS -> jsPins
NpmPinTarget.WASM -> wasmPins
}

private fun validatePackageName(packageName: String): String {
Expand All @@ -75,8 +102,9 @@ abstract class NpmPinsExtension {
"[npm-pins] Package name '$packageName' must not contain whitespace"
}
require(declared.add(name)) {
"[npm-pins] '$name' is pinned twice — a Yarn resolution is global, so the " +
"second declaration would silently win"
"[npm-pins] '$name' is pinned twice — a Yarn resolution is global within a " +
"Yarn root, so the second declaration would silently win. Declare it " +
"once and pass both targets, or scope each pin to a different target."
}
return name
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ import sk.ainet.buildlogic.root.SkainetRootExtension
*
* There are two such extensions — one for JS, one for Wasm — each with its own
* lockfile, so a pin has to be applied twice. This plugin does that from a single
* declaration.
* declaration, and lets a pin opt out of one of them via [NpmPinTarget] when the
* package exists in only one graph.
*
* ## Declaring a pin
*
Expand All @@ -42,6 +43,9 @@ import sk.ainet.buildlogic.root.SkainetRootExtension
* skainet {
* npmPins {
* pin("ws", libs.versions.npm.ws)
* // Only the JS graph bundles with webpack; scoping keeps webpack's ~76
* // transitive packages out of kotlin-js-store/wasm/yarn.lock.
* pin("webpack", libs.versions.npm.webpack, NpmPinTarget.JS)
* }
* }
* ```
Expand Down Expand Up @@ -75,30 +79,31 @@ class NpmPinsPlugin : Plugin<Project> {

// Resolved lazily: pins are declared in the root script body, which runs after
// the plugins block that applies this plugin.
val pinsProvider: Provider<Map<String, String>> = extension.pins
val jsPins: Provider<Map<String, String>> = extension.jsPins
val wasmPins: Provider<Map<String, String>> = extension.wasmPins

// The Yarn plugins are applied by KGP only once a js/wasmJs target is configured,
// which happens well after this plugin is applied — hence the reactive hooks.
// Each root gets only the pins scoped to it; see NpmPinTarget for why that matters.
project.plugins.withType(YarnPlugin::class.java) {
project.extensions.getByType(YarnRootExtension::class.java).applyPins(pinsProvider.get())
project.extensions.getByType(YarnRootExtension::class.java).applyPins(jsPins.get())
}
project.plugins.withType(WasmYarnPlugin::class.java) {
// getByName rather than WasmYarnRootExtension[project]: the latter applies
// WasmYarnPlugin as a side effect, dragging wasm Yarn setup into js-only builds.
val wasmYarn = project.extensions.getByName(WasmYarnRootExtension.YARN) as WasmYarnRootExtension
wasmYarn.applyPins(pinsProvider.get())
wasmYarn.applyPins(wasmPins.get())
}

val verify = project.tasks.register("verifyNpmPins", VerifyNpmPinsTask::class.java) {
group = "verification"
description = "Check the committed Yarn lockfiles against the pins declared in skainet { npmPins { } }"
pins.set(pinsProvider)
this.jsPins.set(jsPins)
this.wasmPins.set(wasmPins)
failOnMissingPackage.set(extension.failOnMissingPackage)
rootDirectory.set(project.layout.projectDirectory)
lockFiles.from(
project.layout.projectDirectory.file("kotlin-js-store/yarn.lock"),
project.layout.projectDirectory.file("kotlin-js-store/wasm/yarn.lock"),
)
jsLockFile.from(project.layout.projectDirectory.file("kotlin-js-store/yarn.lock"))
wasmLockFile.from(project.layout.projectDirectory.file("kotlin-js-store/wasm/yarn.lock"))
// The lockfiles are outputs of the store tasks. Order after them rather than
// depending on them, so `verifyNpmPins` stays a cheap file check on its own but
// still sees the current state when a full build refreshes the lockfiles.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,60 +13,91 @@ import org.gradle.api.tasks.PathSensitive
import org.gradle.api.tasks.PathSensitivity
import org.gradle.api.tasks.TaskAction
import org.gradle.work.DisableCachingByDefault
import java.io.File

/**
* Fails when a committed Yarn lockfile disagrees with a declared npm pin.
*
* Yarn `resolutions` make the pin take effect, but only the next time the lockfile
* is regenerated. Without this check a stale or hand-edited lockfile silently wins
* — which is exactly how PR #894's `ws` bump ended up as a zero-line diff.
*
* Each lockfile is checked against its own pin map, so a pin scoped to one
* [NpmPinTarget] is never reported as missing from the other lockfile.
*/
@DisableCachingByDefault(because = "Reads two small lockfiles; caching costs more than it saves")
abstract class VerifyNpmPinsTask : DefaultTask() {

/** Package name -> pinned version, sourced from the `npm-*` catalog aliases. */
/** Package name -> pinned version for the Kotlin/JS lockfile. */
@get:Input
abstract val jsPins: MapProperty<String, String>

/** Package name -> pinned version for the Kotlin/Wasm lockfile. */
@get:Input
abstract val pins: MapProperty<String, String>
abstract val wasmPins: MapProperty<String, String>

@get:Input
abstract val failOnMissingPackage: Property<Boolean>

/** The committed lockfiles under `kotlin-js-store/`. Missing files are skipped. */
/** `kotlin-js-store/yarn.lock`. Missing file is skipped. */
@get:InputFiles
@get:PathSensitive(PathSensitivity.RELATIVE)
abstract val jsLockFile: ConfigurableFileCollection

/** `kotlin-js-store/wasm/yarn.lock`. Missing file is skipped. */
@get:InputFiles
@get:PathSensitive(PathSensitivity.RELATIVE)
abstract val lockFiles: ConfigurableFileCollection
abstract val wasmLockFile: ConfigurableFileCollection

/** Only used to render lockfile paths relative to the repository root in messages. */
@get:Internal
abstract val rootDirectory: DirectoryProperty

@TaskAction
fun verify() {
val expected = pins.get()
if (expected.isEmpty()) {
val lanes = listOf(
Lane("Kotlin/JS", jsPins.get(), jsLockFile.files),
Lane("Kotlin/Wasm", wasmPins.get(), wasmLockFile.files),
)

val pinnedPackages = lanes.flatMap { it.pins.keys }.toSortedSet()
if (pinnedPackages.isEmpty()) {
logger.lifecycle("[npm-pins] No npm pins declared; nothing to verify.")
return
}

val mismatches = mutableListOf<String>()
val seen = mutableSetOf<String>()
val rootDir = rootDirectory.get().asFile
val mismatches = mutableListOf<String>()
val missing = mutableListOf<String>()
var checkedLockFiles = 0

for (lane in lanes) {
if (lane.pins.isEmpty()) continue
val lockFile = lane.lockFiles.firstOrNull { it.isFile } ?: continue
checkedLockFiles++

lockFiles.files.filter { it.isFile }.sortedBy { it.path }.forEach { lockFile ->
val relative = lockFile.relativeToOrSelf(rootDir).path
val seen = mutableSetOf<String>()
YarnLockParser.parse(lockFile.readText()).forEach { (packageName, version) ->
val pinned = expected[packageName] ?: return@forEach
val pinned = lane.pins[packageName] ?: return@forEach
seen += packageName
if (version != pinned) {
mismatches += "$relative: $packageName resolved to $version, pinned to $pinned"
}
}
(lane.pins.keys - seen).sorted().forEach { missing += "$relative (${lane.label}): $it" }
}

val missing = expected.keys - seen
if (missing.isNotEmpty()) {
val message = "[npm-pins] Pinned but absent from every lockfile: ${missing.sorted().joinToString(", ")}. " +
"The pin may be stale — drop it from gradle/libs.versions.toml if the package is gone for good."
val message = buildString {
appendLine("[npm-pins] Pinned but absent from the lockfile the pin is scoped to:")
missing.sorted().forEach { appendLine(" - $it") }
append(
"The pin may be stale — drop it from gradle/libs.versions.toml if the " +
"package is gone for good, or narrow its NpmPinTarget if it only ever " +
"existed in the other graph."
)
}
if (failOnMissingPackage.get()) throw GradleException(message)
logger.warn(message)
}
Expand All @@ -84,6 +115,10 @@ abstract class VerifyNpmPinsTask : DefaultTask() {
)
}

logger.lifecycle("[npm-pins] ${expected.size} pin(s) verified against ${lockFiles.files.count { it.isFile }} lockfile(s).")
logger.lifecycle(
"[npm-pins] ${pinnedPackages.size} pin(s) verified against $checkedLockFiles lockfile(s)."
)
}

private data class Lane(val label: String, val pins: Map<String, String>, val lockFiles: Set<File>)
}
22 changes: 16 additions & 6 deletions build.gradle.kts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import sk.ainet.buildlogic.npm.NpmPinTarget

plugins {
alias(libs.plugins.androidLibrary) apply false
alias(libs.plugins.kotlinMultiplatform) apply false
Expand All @@ -19,16 +21,24 @@ allprojects {
version = providers.gradleProperty("VERSION_NAME").getOrElse("unspecified")
}

// Root-project SKaiNET conventions. npm pins are forced onto both kotlin-js-store
// Root-project SKaiNET conventions. npm pins are forced onto the kotlin-js-store
// lockfiles via Yarn resolutions; see docs "Pinning npm Packages".
//
// `ws` is the only pinned package present in both dependency graphs, so it is the only
// unscoped pin. Everything else lives solely in the Kotlin/JS graph and is scoped to it:
// Yarn writes a lockfile entry for every resolutions key whether or not that graph
// requests the package, so an unscoped pin would add the package — and, for webpack, its
// ~76 transitive dependencies — to kotlin-js-store/wasm/yarn.lock for nothing.
skainet {
npmPins {
pin("ws", libs.versions.npm.ws)
pin("js-yaml", libs.versions.npm.js.yaml)
pin("socket.io-parser", libs.versions.npm.socketio.parser)
pin("fast-uri", libs.versions.npm.fast.uri)
pin("serialize-javascript", libs.versions.npm.serialize.javascript)
pin("brace-expansion", libs.versions.npm.brace.expansion)
pin("js-yaml", libs.versions.npm.js.yaml, NpmPinTarget.JS)
pin("socket.io-parser", libs.versions.npm.socketio.parser, NpmPinTarget.JS)
pin("fast-uri", libs.versions.npm.fast.uri, NpmPinTarget.JS)
pin("serialize-javascript", libs.versions.npm.serialize.javascript, NpmPinTarget.JS)
pin("brace-expansion", libs.versions.npm.brace.expansion, NpmPinTarget.JS)
pin("diff", libs.versions.npm.diff, NpmPinTarget.JS)
pin("webpack", libs.versions.npm.webpack, NpmPinTarget.JS)
}
}

Expand Down
29 changes: 29 additions & 0 deletions docs/modules/ROOT/pages/contributing/build-from-source.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,35 @@ skainet {
./gradlew kotlinUpgradeYarnLock kotlinWasmUpgradeYarnLock
----

==== Scoping a pin to one lockfile

Yarn writes a lockfile entry for *every* `resolutions` key, whether or not that dependency graph actually requests the package. An unscoped pin for a package that only the JS graph uses therefore adds it — and everything it depends on — to `kotlin-js-store/wasm/yarn.lock`, where nothing imports it. For `webpack` that is roughly 76 extra packages the Wasm build would download and install for nothing, and which Dependabot would then report against a lockfile that never uses them.

Pass an `NpmPinTarget` when a package lives in only one graph:

[source,kotlin]
----
import sk.ainet.buildlogic.npm.NpmPinTarget

skainet {
npmPins {
pin("ws", libs.versions.npm.ws) // <1>
pin("webpack", libs.versions.npm.webpack, NpmPinTarget.JS) // <2>
}
}
----
<1> No target: applies to both lockfiles. Correct for `ws`, which both graphs resolve.
<2> `NpmPinTarget.JS`: only `kotlin-js-store/yarn.lock` gets the resolution.

Check which graph holds a package before deciding:

[source,bash]
----
grep -c '^webpack@' kotlin-js-store/yarn.lock kotlin-js-store/wasm/yarn.lock
----

`verifyNpmPins` checks each lockfile against its own pin map, so a scoped pin is never reported as missing from the lockfile it was never meant to reach.

`verifyNpmPins` re-reads the committed lockfiles and fails if any pinned package resolved elsewhere. It is wired into `check` and runs on the `js-wasm` leg of `.github/workflows/build.yml`.

The root plugin is not optional for web modules: `sk.ainet.multiplatform` fails at configuration time if a module builds `js`/`wasmJs` while the root project does not apply `sk.ainet.npm-pins`. Without it no `resolutions` are written *and* `verifyNpmPins` does not exist to notice — a silent security regression rather than a build error.
Expand Down
Loading
Loading