Add subgraph mode: run unmodified subgraph projects on HyperIndex - #1538
Add subgraph mode: run unmodified subgraph projects on HyperIndex#1538DZakh wants to merge 57 commits into
Conversation
Maps The Graph's manifest, schema, and graph-ts mappings API (specVersion 0.0.4-1.3.0, apiVersion 0.0.5-0.0.9) onto HyperIndex's current surface, and sketches the shim design: AsyncLocalStorage entity-write scoping, a register pass for dataSource.create, and a sync-over-async bridge for store.get and eth_calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…re errors The worker/codemod bridge is dropped in favor of try-sync reads against the in-memory state (hasInMemory/getUnsafeInMemory already exist per load group), throwing an envio-owned suspend error on miss while the load is scheduled, then rerunning the handler once it resolves. Preload batches the warmup so the execute pass replays zero times in the common case. Unsupported subgraph features now fail with a uniform loud error instead of no-ops: callHandlers, block filter:call, graft, nonFatalErrors, subgraph composition, and event.receipt.logs at runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…, implementation plan Interfaces and timeseries/aggregations now fail with the uniform unsupported-feature error. Suspending marks the per-event context as aborted so a caught suspend error stops the handler on its next context access. The doc is restructured around a short how-it-works summary, condensed tables, and a phased implementation plan (core sync API, CLI subgraph mode, graph-ts shim, e2e scenario). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…ime home, ens, replay cap isResolved and the abort flag merge into one contextStatus (Active | Aborted(exn) | Resolved) shared by reference with entity sub-proxies. The sync/suspend API stays internal-only for the subgraph runtime, which ships inside the envio package as an envio/subgraph subpath so versions stay lock-step. ens.nameByHash becomes a best-effort cached effect returning null on miss. Replay termination is a progress check plus a fixed high round cap, not configurable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
… config, drop replay cap Block handlers expose only block.number; other ethereum.Block fields, and event.transactionLogIndex, throw the unsupported-feature error on access. The parsed manifest passes through the resolved public config JSON under a subgraph field instead of a normalized copy in .envio/. The replay termination guard is dropped for the first iteration. Int8->BigInt rationale noted (64-bit vs 32-bit Int). Testing adds an Envio Subgraph tool to the open-indexer-benchmark repo running its Subgraph case on HyperIndex. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…dation tests Unknown manifest fields/kinds/versions, schema directives/arguments/types, and runtime graph-ts API accesses are refused with an 'unknown' error (distinct from 'unsupported') that still points at updating envio or opening an issue. SubgraphValidation_test.res in envio-tests, patterned on UserApiValidation_test.res, asserts exact messages for every unsupported and unknown case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…oxy, throwing getters Namespace objects get full Proxies (unenumerable future APIs, cold path); hot classes get a Proxy at the bottom of the prototype chain so known members never pay trap cost while unknown names still throw; known-but- refused fields use plain throwing getters with the unsupported error. Traps allowlist symbols and thenable/inspect names, and ESM link errors for missing named imports are rewrapped into the unknown-error template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
HandlerLoader activates it when the public config carries the subgraph field; nothing outside the envio package imports it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…ction Verified against graph-tooling's codegen: generated entity classes, contract bindings, event and template classes are implemented purely in terms of graph-ts APIs (Entity/Value/TypedMap/store, ethereum.SmartContract, DataSource.create). So generated/ needs no shim - the real graph codegen output executes on top of the graph-ts shim, the editor keeps seeing the real package types, and conformance is tested via golden generated/ fixtures across graph-cli versions plus a type-level satisfies check against real graph-ts declarations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
ENVIO_API_TOKEN via env or .env, ENVIO_RPC_URL shared by sync fallback and contract-call effects (comma-separated fallback list), and an optional envio.yaml overlay for the full envio surface the manifest can't express. Contract calls without an RPC fail lazily with a setup error; translation pins address_format lowercase for graph-ts parity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
Value is url | JSON object | JSON array of either, parsed with the same schema and deny-unknown rules as config.yaml's rpc field, injected verbatim into the generated chain config (bare URLs default to fallback) and reused in order by the shim's call effects. HyperRPC doesn't support eth_call, so contract calls without the var fail with a setup error naming it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
ENVIO_API_TOKEN and ENVIO_SUBGRAPH_RPC are the whole configuration surface in subgraph mode; every other knob stays at envio defaults. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFGUiVt8rwYfXziK55GbAt
…1536) * Resolve review findings in the subgraph spec - block.timestamp via internal batched HyperSync effect; other block fields stay errors (post-hoc fetch isn't reorg-consistent) - translator owns schema strictness (envio's parser ignores unknown directives); immutable: true documented as accepted divergence - Timestamp scalar kept as envio Timestamp with store-boundary conversion - receipt scalars flagged HyperSync-only; RPC interaction inherits envio behavior - declared eth_calls make the missing-RPC error eager at startup - try_ distinguishes contract reverts from transport failures - getInBlock checkpoint-filtered against same-batch earlier-block hits - topic filters on dynamic-typed indexed params -> unsupported error - drop stale 'three channels' text from the removed envio.yaml revision Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VL2v3tc1g6XP98w2gyC8q2 * Make the block timestamp effect uncached Read once per block by that block's own handler, so a persisted row per indexed block is bloat with no reuse; in-memory memoization still covers replay rounds and the preload to execute transition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VL2v3tc1g6XP98w2gyC8q2 --------- Co-authored-by: Claude <noreply@anthropic.com>
The subgraph runtime needs graph-ts' synchronous host ops on top of envio's async ones. This lands the core half of that bridge: - `contextStatus` (Active | Aborted(exn) | Resolved) replaces `isResolved`, held by reference so the entity sub-proxies share the handler context's lifecycle — and their traps and op closures now enforce it. - `getSync` / `getWhereSync` / `getInBlockSync` / `effectSync` read the in-memory state and, on a miss, schedule the async op and suspend. - `runSync` awaits what a round scheduled and replays the body from the top, capped at 10k rounds. All internal: not in index.d.ts, reserved for the subgraph runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Manifest parsing walks the YAML by hand instead of deriving serde structs: deny-unknown has to report every unknown key with its path, and a project should see all of its problems in one run rather than one per attempt. The schema translator owns strictness for the same reason — envio's parser ignores unrecognised directives and doesn't require @entity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
The graph-ts shim and the runtime that drives it live inside the envio package with no subpath export: nothing outside imports them, and HandlerLoader activates them when the resolved config carries a translated manifest. Mappings resolve @graphprotocol/graph-ts to the shim at runtime only, so types keep resolving to the real package. Each manifest handler becomes an onEvent/onBlock wrapper running the mapping inside runSync, with dataSource.create captured by a contractRegister pass that reruns the same mapping in register mode. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
parse_from_project_files is the one door dev, start and codegen all go through, so keying subgraph mode off a subgraph.yaml with no config.yaml beside it is enough to make all three work inside an unmodified project. A mapping importing the project's generated/ is the only thing that makes graph codegen necessary, and the failed import is exactly where that shows up — so the runtime builds it there, with the project's own graph-cli. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Factory + template + eth_call + block handler, indexed straight off subgraph.yaml with no envio config anywhere in the project. Running it surfaced a gap: the register pass has no effect caller, so a mapping that calls a contract crashed instead of reporting. An effect reached before any create() could have decided which address to register, so it is refused with the §7 error; once registration has happened the rest of the mapping only feeds writes that are no-ops there, so it is skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
The scenario's mappings now import the project's generated/ the way a real subgraph's do, and that output is real `graph codegen` from a pinned graph-cli, checked in so the mappings run against exactly those bytes. Running it found three things the hand-written mappings never reached: - `new ethereum.CallResult()` is how a generated binding signals a revert, so the constructor has to default to reverted. - Generated code asserts on `Value.displayKind()`, which was missing. - `changetype<Pair | null>(store.get(...))` erases its type argument, so a loaded entity reaches the mapping without the generated prototype that carries `pair.token0`. The entity prototype tail now falls through to the stored field, typed by the Value kind — which is what the generated getter would have returned. A name the entity doesn't hold is still refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
A manifest names an event by a signature with no parameter names. Envio reads the parameters out of the ABI itself, so a unique name is all it needs — but an overloaded name isn't unique, and envio's routing rejects two events sharing one. Those get the spelled-out human-readable signature and a suffixed name, which the runtime then registers under. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
envio hands a block handler only the block number. Every handler invocation in a batch asks for its timestamp within the same microtask, so the requests are collected and answered by a single HyperSync range query instead of a round trip per block — which is also what keeps a block handler from needing an RPC endpoint at all. RPC covers whatever HyperSync couldn't answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Running a third-party subgraph project surfaced both of these. A subgraph project's package.json carries no `type`, so tsx loads its mappings and its generated/ as CommonJS — and reaching the shim, a real ES module, through require() is what Node refuses inside a cycle. The resolve hook now declares files under the project root as modules, which is what they are. Codegen also has to run before the first mapping import rather than in response to it: Node caches a failed module resolution for the life of the process, so generating afterwards and retrying resolves to the same failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Running the open-indexer-benchmark's Subgraph case found this: the fold that keeps a chain open-ended until every source on it stops was also applied to the first source, which always folded its endBlock away against the None the chain was created with. A manifest with one bounded data source indexed past its end block forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
A subgraph's mappings are compiled by asc against @graphprotocol/graph-ts and then run by the shim, and nothing made those two agree — a gap showed up as green types and a runtime error. tsc cannot read graph-ts directly: it is AssemblyScript, and u64/i32/usize are not TypeScript types. Rewriting the primitives and dropping the AS decorators makes the signatures readable, and emitting declarations with --noCheck keeps the bodies — which mix integer widths in ways AS allows and TS does not — out of it. The generated declarations are committed alongside the graph-ts version they came from. conformance.ts then asserts every export is assignable to the real one. The 18 gaps it found are recorded as @ts-expect-error, which cleans itself up: closing a gap makes the directive unused, so the comment goes with the fix. TypedMap.mustGetEntry is the first one closed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
non-EVM subgraphs The 18 recorded gaps are down to 6, and those 6 are one difference reached six ways: graph-ts' BigInt extends Uint8Array and is constructed from a byte length, while this one is constructed from the value it holds. Mappings never call `new BigInt(...)` — they go through `BigInt.from*` — and the class is otherwise the same shape, bytes included. `envio dev` now runs the project's own `graph build`, which is the asc type check a subgraph already has, so a type error reads the same here as on Graph Node. Stamped against the inputs it reads, so it costs nothing on a restart that changed nothing. A NEAR or Cosmos subgraph now hears that Envio Subgraph indexes EVM chains, rather than that its network is unknown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Found running the benchmark's state-aggregation case, whose mapping does the usual load, mutate, save. `changetype` erases the generated prototype, and with it the `save()` that knows which table to write back to — so the entity prototype tail remembers the type the row was loaded from and rebuilds it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
A CPU profile of the benchmark's event-stream case put ~7% of indexing time in building each BigInt's two's-complement representation — work added for type conformance that no mapping reads: graph codegen never indexes a BigInt, and the conversions that would are overridden here. The class still extends Uint8Array, so the type contract is unchanged; the bytes are just not filled. envio runs each handler twice over one payload, to preload and to execute, and the graph-ts conversion was repeated both times. Also drops ENVIO_SUBGRAPH_SKIP_TYPECHECK: the dev-loop type check is the project's own toolchain, and an escape hatch from it is an invitation to ship a subgraph that doesn't compile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/envio/src/subgraph/graph-ts.ts`:
- Around line 1804-1806: Replace the split-based implementation in parseCSV with
a state-based CSV parser that preserves whitespace in unquoted fields, handles
commas inside quoted fields, and omits the trailing empty field (for example,
a,"b,c", yields ["a", "b,c"]). Add conformance tests covering quoted commas,
preserved unquoted whitespace, and trailing delimiters.
- Around line 1808-1811: Update addQm to always produce a fixed 34-byte result
by allocating 34 bytes, writing the 0x1220 prefix, copying no more than 32 bytes
from a, and zero-filling any remaining bytes; alternatively, validate and reject
inputs whose length is not exactly 32 bytes.
- Around line 1030-1065: Update the EthereumValue factory methods fromTuple,
fromTupleArray, fromFixedBytes, fromAddressArray, and fromFixedBytesArray to
construct and return EthereumValue instances so their Ethereum-specific
converters such as toTuple work at runtime. Keep the existing conversion
behavior and ensure runtime conformance coverage verifies these factories return
usable EthereumValue objects.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1e0be1df-34c2-4675-97e0-f290739249db
📒 Files selected for processing (2)
packages/envio-tests/test/SubgraphContractCall_test.respackages/envio/src/subgraph/graph-ts.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/envio-tests/test/SubgraphContractCall_test.res
| export class EthereumValue extends Value { | ||
| static fromSignedBigInt(value: BigInt_): Value { | ||
| return Value.fromBigInt(value); | ||
| } | ||
| static fromUnsignedBigInt(value: BigInt_): Value { | ||
| return Value.fromBigInt(value); | ||
| } | ||
| static fromFixedBytes(value: Bytes): Value { | ||
| return Value.fromBytes(value); | ||
| } | ||
| static fromTuple(values: Value[]): Value { | ||
| return Value.fromArray(values); | ||
| } | ||
| static fromFixedSizedArray(values: Value[]): Value { | ||
| return Value.fromArray(values); | ||
| } | ||
| static fromTupleArray(values: Value[][]): Value { | ||
| return Value.fromArray(values.map(EthereumValue.fromTuple)); | ||
| } | ||
| static fromAddressArray(values: Address[]): Value { | ||
| return Value.fromArray(values.map(Value.fromAddress)); | ||
| } | ||
| static fromFixedBytesArray(values: Bytes[]): Value { | ||
| return Value.fromArray(values.map(Value.fromBytes)); | ||
| } | ||
| static fromSignedBigIntArray(values: BigInt_[]): Value { | ||
| return Value.fromArray(values.map(Value.fromBigInt)); | ||
| } | ||
| static fromUnsignedBigIntArray(values: BigInt_[]): Value { | ||
| return Value.fromArray(values.map(Value.fromBigInt)); | ||
| } | ||
| toTuple(): Value[] { | ||
| return this.toArray(); | ||
| } | ||
| toTupleArray(): Value[][] { | ||
| return this.toArray().map(value => value.toArray()); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '449,692p' packages/envio/src/subgraph/graph-ts.ts
sed -n '1024,1076p' packages/envio/src/subgraph/graph-ts.ts
curl -fsSL 'https://raw.githubusercontent.com/graphprotocol/graph-tooling/%40graphprotocol/graph-ts%400.38.1/packages/ts/chain/ethereum.ts' |
sed -n '53,120p;271,325p'Repository: enviodev/hyperindex
Length of output: 13389
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- relevant definitions and usages ---'
rg -n -C 3 'EthereumValue\.(fromTuple|fromFixedSizedArray|fromTupleArray|fromAddressArray|fromFixedBytesArray|fromSignedBigIntArray|fromUnsignedBigIntArray)|toTuple(Array)?\(' packages tests 2>/dev/null || true
printf '%s\n' '--- package and test files ---'
git ls-files | rg '(^|/)(package\.json|tsconfig[^/]*\.json|.*(test|spec).*\.(ts|tsx|js))$' | head -200
printf '%s\n' '--- exact source range ---'
cat -n packages/envio/src/subgraph/graph-ts.ts | sed -n '1018,1075p'
printf '%s\n' '--- inheritance/type declarations ---'
rg -n -C 2 'class (Value|EthereumValue)|fromTuple|toTuple' packages/envio/src packages/envio/test packages 2>/dev/null | head -300Repository: enviodev/hyperindex
Length of output: 19309
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node - <<'JS'
const fs = require('fs');
const source = fs.readFileSync('packages/envio/src/subgraph/graph-ts.ts', 'utf8');
const start = source.indexOf('export class Value {');
const end = source.indexOf('\n}', source.indexOf('export class EthereumValue extends Value {'));
console.log(source.slice(start, end + 2));
JSRepository: enviodev/hyperindex
Length of output: 20857
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- ethereum.Value imports and factory/converter call sites ---'
rg -n -C 3 'ethereum\.Value|Value\.fromTuple|fromTupleArray|fromFixedSizedArray|toTupleArray|toFixedBytes' \
packages scenarios --glob '*.{ts,tsx,js,jsx,d.ts}' 2>/dev/null | head -500
printf '%s\n' '--- graph-ts exports and module mappings ---'
rg -n -C 3 'graph-ts|ethereum|EthereumValue|export .*Value|paths|alias' \
packages/envio/package.json packages/envio/tsconfig*.json package.json packages scenarios \
--glob '*.{json,ts,tsx,js,d.ts}' 2>/dev/null | head -500Repository: enviodev/hyperindex
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
source = Path("packages/envio/src/subgraph/graph-ts.ts").read_text()
start = source.index("export class Value {")
end = source.index("\n}", source.index("export class EthereumValue extends Value {")) + 2
block = source[start:end]
checks = {
"EthereumValue extends Value": "export class EthereumValue extends Value" in block,
"fromTuple returns Value": "static fromTuple(values: Value[]): Value" in block,
"fromTuple delegates to Value.fromArray": "return Value.fromArray(values);" in block,
"toTuple exists only in EthereumValue block": "toTuple(): Value[]" in block,
}
for name, result in checks.items():
print(f"{name}: {result}")
# Model only the relevant JavaScript class semantics from the source.
class Value:
def __init__(self, data):
self.data = data
`@staticmethod`
def fromArray(values):
return Value(values)
def toArray(self):
return self.data
class EthereumValue(Value):
`@staticmethod`
def fromTuple(values):
return Value.fromArray(values)
def toTuple(self):
return self.toArray()
result = EthereumValue.fromTuple([])
print(f"factory result constructor: {type(result).__name__}")
print(f"factory result has toTuple: {callable(getattr(result, 'toTuple', None))}")
PYRepository: enviodev/hyperindex
Length of output: 383
Return EthereumValue instances from Ethereum-only factories.
fromTuple, fromTupleArray, fromFixedBytes, fromAddressArray, and fromFixedBytesArray construct base Value instances. Calls such as ethereum.Value.fromTuple(values).toTuple() therefore fail at runtime. Return EthereumValue instances or move the matching converters to Value. Add runtime conformance cases.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/envio/src/subgraph/graph-ts.ts` around lines 1030 - 1065, Update the
EthereumValue factory methods fromTuple, fromTupleArray, fromFixedBytes,
fromAddressArray, and fromFixedBytesArray to construct and return EthereumValue
instances so their Ethereum-specific converters such as toTuple work at runtime.
Keep the existing conversion behavior and ensure runtime conformance coverage
verifies these factories return usable EthereumValue objects.
| export function parseCSV(csv: string): string[] { | ||
| return csv.split(",").map(field => field.trim()); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1800,1812p' packages/envio/src/subgraph/graph-ts.ts
curl -fsSL 'https://raw.githubusercontent.com/graphprotocol/graph-tooling/%40graphprotocol/graph-ts%400.38.1/packages/ts/helper-functions.ts' |
sed -n '18,74p'Repository: enviodev/hyperindex
Length of output: 2178
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- parseCSV definitions and usages ---'
rg -n --glob '!node_modules' 'parseCSV|CSVState' packages test tests .github 2>/dev/null | head -200 || true
printf '%s\n' '--- nearby source ---'
sed -n '1798,1812p' packages/envio/src/subgraph/graph-ts.ts
printf '%s\n' '--- behavioral comparison ---'
python3 - <<'PY'
def current(csv):
return [field.strip() for field in csv.split(",")]
def target(csv):
values = []
value_start = 0
state = "BETWEEN"
for i, char in enumerate(csv):
if state == "BETWEEN":
if char != ",":
if char == '"':
state = "QUOTED_VALUE"
value_start = i + 1
else:
state = "UNQUOTED_VALUE"
value_start = i
elif state == "UNQUOTED_VALUE":
if char == ",":
values.append(csv[value_start:i])
state = "BETWEEN"
elif state == "QUOTED_VALUE" and char == '"':
values.append(csv[value_start:i])
state = "BETWEEN"
return values
for sample in ['a,"b,c",', '"a,b"', ' a , b ', ',,', 'a,b', 'a,']:
print(repr(sample), 'current=', current(sample), 'target=', target(sample))
PYRepository: enviodev/hyperindex
Length of output: 1201
Implement the graph-ts CSV parser behavior.
Replace the split(",").map(...) implementation with the state-based parser. It must preserve unquoted whitespace and parse quoted commas, so a,"b,c", returns ["a", "b,c"]. Add quoted-field conformance tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/envio/src/subgraph/graph-ts.ts` around lines 1804 - 1806, Replace
the split-based implementation in parseCSV with a state-based CSV parser that
preserves whitespace in unquoted fields, handles commas inside quoted fields,
and omits the trailing empty field (for example, a,"b,c", yields ["a", "b,c"]).
Add conformance tests covering quoted commas, preserved unquoted whitespace, and
trailing delimiters.
| /** Restores the multihash prefix an IPFS hash loses when stored in a bytes32. */ | ||
| export function addQm(a: ByteArray): ByteArray { | ||
| return ByteArray.fromHexString("0x1220").concat(a); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target implementation ---'
rg -n -C 12 'function addQm|addQm' packages/envio/src/subgraph/graph-ts.ts
printf '%s\n' '--- related ByteArray definitions and helpers ---'
rg -n -C 8 'class ByteArray|namespace ByteArray|fromHexString|concat' packages/envio/src/subgraph/graph-ts.ts packages/envio/src/subgraph
printf '%s\n' '--- repository references ---'
rg -n -C 5 'addQm|multihash|0x1220' .
printf '%s\n' '--- declared source metadata ---'
git status --short
git diff --statRepository: enviodev/hyperindex
Length of output: 38702
🏁 Script executed:
#!/bin/bash
set -eu
url='https://raw.githubusercontent.com/graphprotocol/graph-tooling/%40graphprotocol/graph-ts%400.38.1/packages/ts/helper-functions.ts'
curl --fail --silent --show-error "$url" | sed -n '/function addQm/,/^[[:space:]]*}/p'
printf '%s\n' '--- all addQm references in the fetched helper ---'
curl --fail --silent --show-error "$url" | rg -n -C 10 'addQm|1220|bytes32|ByteArray'
printf '%s\n' '--- local type declarations ---'
sed -n '14,24p' packages/envio/src/subgraph/graph-ts-types/helper-functions.d.tsRepository: enviodev/hyperindex
Length of output: 2549
🏁 Script executed:
node - <<'JS'
function referenceAddQm(input) {
const out = new Uint8Array(34);
out[0] = 0x12;
out[1] = 0x20;
for (let i = 0; i < 32; i++) out[i + 2] = input[i];
return out;
}
function currentAddQm(input) {
const out = new Uint8Array(2 + input.length);
out.set([0x12, 0x20], 0);
out.set(input, 2);
return out;
}
for (const input of [
new Uint8Array(0),
new Uint8Array([1, 2]),
new Uint8Array(32).fill(3),
new Uint8Array(33).fill(4),
]) {
const expected = referenceAddQm(input);
const actual = currentAddQm(input);
console.log(JSON.stringify({
inputLength: input.length,
expectedLength: expected.length,
actualLength: actual.length,
expectedTail: [...expected.slice(2, 6)],
actualTail: [...actual.slice(2, 6)],
}));
}
JSRepository: enviodev/hyperindex
Length of output: 562
Preserve the fixed 34-byte output in addQm. Allocate 34 bytes, write 0x1220, copy at most 32 input bytes, and zero-fill missing bytes. Alternatively, reject inputs that are not 32 bytes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/envio/src/subgraph/graph-ts.ts` around lines 1808 - 1811, Update
addQm to always produce a fixed 34-byte result by allocating 34 bytes, writing
the 0x1220 prefix, copying no more than 32 bytes from a, and zero-filling any
remaining bytes; alternatively, validate and reject inputs whose length is not
exactly 32 bytes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f2b3ec4839
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .filter(|name| !name.is_empty()) | ||
| .map(|name| name.to_string()) | ||
| .unwrap_or_else(|| format!("arg{index}")); | ||
| (name, solidity_type(input)) |
There was a problem hiding this comment.
Preserve indexed metadata for dynamically typed parameters
When an event has an indexed dynamic parameter such as indexed string, the decoder supplies its topic hash and graph-ts codegen exposes that parameter as Bytes, but this metadata records only the Solidity type and discards input.indexed. The runtime consequently takes the ordinary string conversion path and returns a JavaScript string, so mappings calling normal Bytes methods such as toHexString() fail; indexed arrays and tuples likewise cannot be distinguished from decoded values. Carry the indexed flag into the runtime parameter metadata and convert indexed dynamic values to Bytes.
Useful? React with 👍 / 👎.
| return (v) => (typeof v === "bigint" ? Number(v) : v); | ||
| } | ||
| if (/^u?int/.test(type)) return (v) => new (GraphBigInt as any)(v as bigint); | ||
| return (v) => v; |
There was a problem hiding this comment.
Recursively convert tuple event parameters
For an event parameter such as (address,uint256) order, Envio decodes the tuple as an object, but this fallback leaves every component in its raw JavaScript form. A mapping expecting graph-ts values therefore receives a string for order.maker and a native bigint for order.amount, so calls such as order.maker.toHexString() or order.amount.plus(...) fail; positional access also stringifies the tuple as [object Object]. Parse tuple component types and recursively wrap their values instead of returning the object unchanged.
Useful? React with 👍 / 👎.
| client ??= createPublicClient({ | ||
| transport: fallback(rpcUrls.map((url) => http(url, { retryCount: 0 }))), | ||
| }); |
There was a problem hiding this comment.
Route host RPC reads to the active chain
In a multichain subgraph whose mappings call ethereum.getBalance or ethereum.hasCode, these effects are partitioned by chain but their handlers always reuse this single client over the global URL list. An invocation on a second chain can therefore query the first chain's endpoint at the same block number and silently return unrelated balance or code data. Select or construct the client from the effect callback's chain context, as the handler already receives that context.
Useful? React with 👍 / 👎.
| for (const page of body.data ?? []) { | ||
| for (const block of page.blocks ?? []) { | ||
| timestamps.set(block.number, BigInt(block.timestamp)); |
There was a problem hiding this comment.
Read blocks from HyperSync's columnar response object
Whenever a block handler reads block.timestamp, the raw HyperSync endpoint returns data as one object containing a blocks array, matching the repository's canonical QueryResponseData shape in packages/cli/src/evm_hypersync_source/mod.rs. Iterating body.data here therefore throws because that object is not iterable; without an RPC fallback every such handler fails, and with RPC configured every timestamp unnecessarily falls back to an individual RPC request. Iterate body.data.blocks directly.
Useful? React with 👍 / 👎.
| div(other: BigDecimal): BigDecimal { | ||
| return new BigDecimal(this.value.div(other.value)); | ||
| } |
There was a problem hiding this comment.
Match graph-node's BigDecimal precision
When a mapping divides BigDecimal values, this uses bignumber.js without any configuration, whose default division precision is only 20 decimal places. Graph-node's BigDecimal uses decimal128-style 34-digit precision, so common calculations such as BigDecimal.fromString("1").div(BigDecimal.fromString("3")) produce a different stored value and can propagate divergent token prices or aggregates. Configure and round arithmetic to graph-node's precision and exponent rules rather than relying on the library defaults.
Useful? React with 👍 / 👎.
| export declare namespace EthereumUtils { | ||
| /** | ||
| * Returns the contract address that would result from the given CREATE2 call. | ||
| * @param from The Ethereum address of the account that is initiating the contract creation. | ||
| * @param salt A 32-byte value that is used to create a deterministic address for the contract. This can be any arbitrary value, but it should be unique to the contract being created. | ||
| * @param initCodeHash he compiled code that will be executed when the contract is created. This should be a hex-encoded string that represents the compiled bytecode. | ||
| * @returns Address of the contract that would be created. | ||
| */ | ||
| function getCreate2Address(from: Bytes, salt: Bytes, initCodeHash: Bytes): Bytes; |
There was a problem hiding this comment.
Export the declared EthereumUtils namespace
A valid mapping importing EthereumUtils and calling getCreate2Address type-checks against this accepted graph-ts declaration, but graph-ts.ts contains no EthereumUtils export anywhere. Because all graph-ts imports are redirected to that shim, Node fails during ESM linking with a missing-export error before handlers are registered. Implement the helper or export a named runtime refusal so supported projects do not fail at module linkage.
Useful? React with 👍 / 👎.
| const DEFAULT_BLOCK_FIELDS: &[BlockField] = &[ | ||
| BlockField::ParentHash, | ||
| BlockField::Miner, |
There was a problem hiding this comment.
Include unclesHash in the fallback block selection
When the usage scan gives up and requests the full block shape—for example because a mapping aliases const block = event.block before reading block.unclesHash—this default omits BlockField::Sha3Uncles even though the runtime exposes unclesHash and the explicit-property scan selects that field. The raw block consequently lacks sha3Uncles, so the mapping receives null instead of the block's hash. Add Sha3Uncles to the default selection used for ambiguous block access.
Useful? React with 👍 / 👎.
Matching a manifest signature against the ABI dropped the indexed markers first, so a name declared both ways resolved to the same ABI entry twice. The generated config then held two events with one signature, and envio refused the project outright: it cannot route a log to one of two events that match the same on-chain data. Safe declares exactly this — ExecutionSuccess(bytes32,uint256) alongside ExecutionSuccess(indexed bytes32,uint256), one per contract version — so the whole subgraph failed to start rather than any one handler misbehaving. Indexing is part of what identifies an overload, so keep it and compare it. The two then differ by topic count, which is what lets envio tell them apart.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3f5732eb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if !field | ||
| .directives | ||
| .iter() | ||
| .any(|directive| directive.name == "derivedFrom") |
There was a problem hiding this comment.
Preserve interface-derived fields in the public schema
When a valid subgraph declares a polymorphic relation such as events: [DomainEvent!]! @derivedFrom(field: "domain"), this branch neither keeps the field nor expands it across the interface's implementors, so it disappears from the parsed entity configuration. Downstream Hasura relationship generation only sees parsed DerivedFromField entries, meaning existing queries for events fail because the field is absent; implement the polymorphic relationship or reject the schema instead of silently changing its query contract.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| function toEthereumValue(value: unknown): EthereumValue { | ||
| return toValue(value) as EthereumValue; |
There was a problem hiding this comment.
Return actual EthereumValue instances from ABI decoding
When a contract call or ethereum.decode returns an aggregate ABI value, this cast leaves the object as the base store Value, whose prototype lacks Ethereum-specific accessors such as toTuple(), toAddressArray(), and toFixedBytesArray(). Generated bindings for tuple or array outputs therefore fail at runtime even though scalar outputs work; recursively construct EthereumValue instances with the appropriate Ethereum value kind instead of casting base values.
Useful? React with 👍 / 👎.
| toI64(value: JSONValue): bigint { | ||
| return value.toI64(); |
There was a problem hiding this comment.
Parse strings in the json numeric conversion helpers
The graph-ts signatures for json.toI64, toU64, toF64, and toBigInt accept decimal strings, but these implementations treat the argument as a JSONValue and invoke instance methods on it. A valid call such as json.toI64("123") consequently throws because a JavaScript string has no toI64 method; parse the supplied decimal string directly, including unsigned validation for toU64.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| export function parseCSV(csv: string): string[] { | ||
| return csv.split(",").map(field => field.trim()); |
There was a problem hiding this comment.
Preserve quoted fields when parsing CSV
When a mapping parses CSV containing quoted commas or escaped quotes, such as a,"b,c", this plain split produces three fields and retains quote characters instead of returning the two CSV fields graph-ts provides. Subgraphs ingesting CSV metadata from IPFS can therefore persist shifted or corrupted values; implement graph-ts-compatible quote-aware CSV parsing rather than splitting on every comma.
Useful? React with 👍 / 👎.
| // Nothing has been written at fetch time, so the register pass reads null | ||
| // rather than a value that would differ between the two passes. | ||
| if (mode === "register") return null; |
There was a problem hiding this comment.
Preserve entity reads during template registration
When a handler makes dataSource.create() conditional on an entity loaded from an earlier block—or creates a template and then relies on an existing entity—this register pass always reports the entity as absent. It can therefore register addresses the real graph-node handler would skip, omit addresses created only when an entity exists, or throw after an earlier registration; provide the register pass with the handler's canonical store view or reject entity-dependent template creation rather than substituting null.
Useful? React with 👍 / 👎.
| encode(_value: EthereumValue): Bytes | null { | ||
| throw unsupported("ethereum.encode", "a mapping handler"); |
There was a problem hiding this comment.
Implement the accepted ethereum.encode API
Any mapping that validly calls ethereum.encode, for example to ABI-encode a tuple before hashing or persisting it, reaches this unconditional refusal and aborts its handler. The compatibility surface exposes this graph-ts API and the design documentation classifies encode/decode as synchronous supported operations, so either encode the value according to its Ethereum value kind or reject such projects during translation rather than failing only when the event executes.
Useful? React with 👍 / 👎.
envio capitalizes a contract name for the config it stores, and the subgraph runtime registered under the manifest's name — so a data source whose name starts lowercase registered a handler against a contract that isn't there, and the whole project refused to start. Every source in the benchmark's contract call scenario shares one mapping, and one of the eight is named crvUSD, so all eight failed on the one that didn't start with a capital. A manifest that happens to capitalize every data source hides this entirely, which is why nothing caught it until a scenario named one after its token.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…t-docs-xffd8x # Conflicts: # .github/workflows/build_and_verify.yml # packages/envio-tests/test/helpers/InternalTestIndexer.res # scenarios/test_codegen/test/EventOrigin_test.res
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
RpcBlockField was removed when every selectable block field became available from eth_getBlockByNumber. The translator still imported it, so CI could not compile.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…sting Treat Graph network names (including the unofficial ethereum alias) as chain ids. Keep a single tuple/struct eth_call as one ethereum.Value so generated bindings can call toTuple(). Drop the subgraph blob from envio_info so a specVersion bump is not a reset. Point a subgraph.template.yaml-only project at generating subgraph.yaml first.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Commit graph-cli 0.97.1 output for a struct-returning eth_call and execute it through the shim, including changetype onto the generated Tuple subclass. Add fromSubgraph cases for network: ethereum and specVersion 0.0.2.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
subgraph.yaml uses The Graph's identifiers. Falling back to Envio's kebab-case hid hyperevm (Graph: hyper-evm) behind hyperliquid. Resolve only GraphNetwork names plus a short alias list, and say so when the user wrote Envio's name.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Graph identifiers still win; Envio kebab-case is a fallback so hyperliquid works as well as hyperevm. CI regenerates the graph-cli goldens and fails if generated/ drifted. A data source with call handlers is a hard translation error — events on the same source do not get a free pass.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Adds support for running unmodified subgraph projects on HyperIndex by translating
subgraph.yamlmanifests and GraphQL schemas into envio configs, and providing a graph-ts compatibility shim that runs the AssemblyScript mappings unchanged.Key Changes
Manifest and Schema Translation
packages/cli/src/subgraph/manifest.rs: Parses and validatessubgraph.yaml, enforcing strict conformance to known features and versionspackages/cli/src/subgraph/schema.rs: Translates GraphQL schemas from subgraph format to envio format, mapping scalar types and handling directivespackages/cli/src/subgraph/abi.rs: Resolves event signatures against contract ABIspackages/cli/src/subgraph/usage.rs: Analyzes mapping source to determine whichevent.transactionandevent.blockfields are actually read, avoiding unnecessary RPC callspackages/cli/src/subgraph/errors.rs: Collects and reports all validation errors in a single passgraph-ts Compatibility Shim
packages/envio/src/subgraph/graph-ts.ts: Complete implementation of@graphprotocol/graph-tsAPI surface, includingByteArray,Address,BigInt,BigDecimal, entity classes, and store operationspackages/envio/src/subgraph/calls.ts: Contract call execution as envio effects, evaluated at the event's blockpackages/envio/src/subgraph/blocks.ts: Block timestamp fetching via HyperSync or RPC fallbackpackages/envio/src/subgraph/hosts.ts: Host operations (IPFS, Arweave, ENS, balance/code checks)packages/envio/src/subgraph/division.ts: AssemblyScript-compatible integer divisionpackages/envio/src/subgraph/conformance.ts: Type conformance checking against real graph-ts packagepackages/envio/src/subgraph/runtime.ts: Handler registration and execution wrapperpackages/envio/src/subgraph/scope.ts: Async context tracking for handler mode and schema informationpackages/envio/src/subgraph/errors.ts: Error messages for unsupported features and refused operationsType Definitions
packages/envio/src/subgraph/graph-ts-types/: Complete TypeScript declarations for graph-ts covering ethereum, cosmos, near, arweave, and starknet chainspackages/envio/scripts/generate-graph-ts-types.mjs: Script to generate type declarations from the real graph-ts packageIntegration
packages/cli/src/subgraph/mod.rs: Main subgraph mode orchestration, translating manifest/schema and building envio configpackages/cli/src/napi.rs: NAPI binding for subgraph translationpackages/envio/src/Core.res: AddedfromSubgraphAPI for loading subgraph projectspackages/envio/src/UserContext.res: Added sync context state machine for handling suspended operations during mapping executionpackages/envio/src/HandlerLoader.res: Support for loading subgraph runtime handlersTesting
packages/envio-tests/test/SubgraphValidation_test.res: Manifest and schema validationpackages/envio-tests/test/SubgraphRuntimeRefusal_test.res: Runtime refusal of unsupported operationspackages/envio-tests/test/SubgraphIndexer_test.res: End-to-end subgraph executionpackages/envio-tests/test/SubgraphContractCall_test.res: Contract call executionpackages/envio-tests/test/SubgraphBlockHandler_test.res: Block handler timestamp fetchingpackages/envio-tests/test/SubgraphContext_test.res: Data source contexthttps://claude.ai/code/session_016tA2D3BrcAJS4nME7noXh4
Summary by CodeRabbit
New Features
Documentation
Tests