Fix: Attempt at improving worker actions query by mrkaye97 · Pull Request #3774 · hatchet-dev/hatchet

mrkaye97 · 2026-04-28T20:15:58Z

Description

Lots of slow traces, which I think are happening because we're pulling a ton of rows down out of the db. I figured we could hash the action names and store them with the worker, and then try to only look up actions for each hash once, and then map them back to the relevant worker that way, which should cut down the number of rows we need to pull out of the db by a factor of the number of replicas of a worker any given tenant has

Unfortunately we can't make the action hash non-null and backfill so we need to also have a lookup by worker id for backwards compat for now

Type of change

Bug fix (non-breaking change which fixes an issue)

vercel · 2026-04-28T20:16:04Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Apr 28, 2026 10:03pm

mrkaye97 · 2026-04-28T21:58:33Z

+;
+
+-- name: GetWorkerActionsByWorkerActionHash :many
+SELECT DISTINCT ON (w."actionHash")


goal here is we'll just move to this completely in a little bit, once all the workers have hashes

Copilot

Pull request overview

This PR aims to reduce slow traces in the worker actions lookup path by adding an actionHash to workers and using it to de-duplicate action lookups across identical worker replicas, falling back to worker-id lookups for backward compatibility.

Changes:

Add Worker.actionHash (BYTEA) plus a (tenantId, actionHash) index (schema + migrations).
Introduce a new repository method to fetch worker actions using action-hash de-duplication with worker-id fallback.
Update worker list/get handlers to call the new repository API and pass full worker rows.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
sql/schema/v0.sql	Adds `actionHash` column and a `(tenantId, actionHash)` index to the baseline schema.
cmd/hatchet-migrate/migrate/migrations/20260428204451_v1_0_103.sql	Adds the `actionHash` column for existing installations.
cmd/hatchet-migrate/migrate/migrations/20260428213451_v1_0_104.sql	Adds the `(tenantId, actionHash)` index concurrently for existing installations.
pkg/repository/worker.go	Implements action-hash-based action lookup and hashes actions on worker creation.
pkg/repository/sqlcv1/workers.sql	Updates worker-actions queries and adds a new query by actionHash.
pkg/repository/sqlcv1/workers.sql.go	Regenerated sqlc output for updated queries/columns.
pkg/repository/sqlcv1/models.go	Adds `ActionHash` field to the `Worker` model.
api/v1/server/handlers/workers/list.go	Uses the new worker-actions repository method and passes worker rows.
api/v1/server/handlers/workers/get.go	Uses the new worker-actions repository method for single-worker fetch.

Copilot · 2026-04-28T22:10:07Z

+JOIN "_ActionToWorker" aw ON w.id = aw."B"
+JOIN "Action" a ON aw."A" = a.id
 WHERE
    a."tenantId" = @tenantId::UUID


GetWorkerActionsByWorkerActionHash doesn’t constrain w."tenantId", so the new (tenantId, actionHash) index won’t be used and (if any cross-tenant links ever exist) results could bleed across tenants. Add AND w."tenantId" = @tenantId::UUID to the WHERE clause to match the index and keep the join tenant-scoped.

Suggested change

a."tenantId" = @tenantId::UUID

a."tenantId" = @tenantId::UUID

AND w."tenantId" = @tenantId::UUID

Copilot · 2026-04-28T22:10:07Z

+func hashActions(actions []string) []byte {
+	h := sha256.New()
+
+	for _, action := range actions {
+		h.Write([]byte(action))


hashActions is not collision-safe as written: concatenating action strings without a delimiter/length prefix means different action lists can produce identical byte streams (e.g., ["ab","c"] vs ["a","bc"]). Also, the hash is order-dependent, which reduces de-duplication if action order varies. Consider hashing a canonical encoding (sort + delimiter or length-prefix each action) to make the hash represent the set/list unambiguously.

Copilot · 2026-04-28T22:10:07Z

+		if len(worker.ActionHash) == 0 {
+			// if the worker has no action hash, we have no choice but to look
+			// it up by its id
+			workerIds = append(workerIds, worker.ID)
+			continue


GetWorkerActionsForWorkers assumes Worker.ActionHash accurately reflects the worker’s current action set. However, actionHash is only set on worker creation right now, so if actions can be updated later (e.g., via UpdateWorker), the stored hash can become stale and cause incorrect action-sharing between workers. Recommend recomputing/persisting actionHash whenever a worker’s actions are modified, or avoid using the hash unless it’s guaranteed up-to-date.

Copilot · 2026-04-28T22:10:08Z

@@ -88,7 +89,7 @@ type WorkerRepository interface {
 	ListActiveSDKsPerTenant(ctx context.Context) (map[TenantIdSDKTuple]int64, error)

 	// GetWorkerActionsByWorkerId returns a list of actions for a worker


The doc comment still refers to GetWorkerActionsByWorkerId, but the method is now GetWorkerActionsForWorkers. Update the comment to match the new API name/behavior (especially since it now takes full worker rows to use actionHash).

Suggested change

// GetWorkerActionsByWorkerId returns a list of actions for a worker

// GetWorkerActionsForWorkers returns actions keyed by worker action hash.

// It takes full worker rows so callers can reuse each worker's persisted actionHash

// instead of recomputing it from the worker ID or action list.

Copilot · 2026-04-28T22:10:08Z

-	records, err := w.queries.GetWorkerActionsByWorkerId(ctx, w.pool, sqlcv1.GetWorkerActionsByWorkerIdParams{
+func (w *workerRepository) GetWorkerActionsForWorkers(ctx context.Context, tenantId uuid.UUID, workers []sqlcv1.Worker) (map[string][]string, error) {
+	actionHashSet := make(map[string]struct{})
+	workerIds := make([]uuid.UUID, len(workers))


workerIds is initialized with len(workers) but later populated via append, which leaves len(workers) leading uuid.Nil entries in the slice. This will send a bunch of zero UUIDs to GetWorkerActionsByWorkerId and can significantly bloat/slow the query; initialize with length 0 (and capacity len(workers)) instead, or fill by index without appending.

Suggested change

workerIds := make([]uuid.UUID, len(workers))

workerIds := make([]uuid.UUID, 0, len(workers))

Copilot · 2026-04-28T22:10:08Z

+;
+
+-- name: GetWorkerActionsByWorkerActionHash :many
+SELECT DISTINCT ON (w."actionHash")


This SELECT DISTINCT ON (w."actionHash") query only returns one actionId per actionHash (the most recent by createdAt), but GetWorkerActionsForWorkers expects to build the full list of actions for each hash. Remove the DISTINCT ON or change it to de-duplicate at the (actionHash, actionId) level so all actions are returned.

Suggested change

SELECT DISTINCT ON (w."actionHash")

SELECT

mrkaye97 added 5 commits April 28, 2026 17:48

fix: attempt at improving worker actions query

616945b

feat: write hash of actions to the worker

86f2f27

fix: index, multiple migrations

8a5e712

feat: wiring, dual read using the hash sometimes

484b9b8

fix: wiring

0692172

mrkaye97 force-pushed the mk/worker-page branch from 04abf76 to 0692172 Compare April 28, 2026 21:48

vercel Bot deployed to Preview April 28, 2026 21:52 View deployment

mrkaye97 added 2 commits April 28, 2026 17:54

fix: ugh, prisma

83f9563

fix: distinct on()

b1b63fa

mrkaye97 commented Apr 28, 2026

View reviewed changes

fix: typo

efed1f3

vercel Bot deployed to Preview April 28, 2026 22:03 View deployment

mrkaye97 marked this pull request as ready for review April 28, 2026 22:05

Copilot AI review requested due to automatic review settings April 28, 2026 22:05

Copilot started reviewing on behalf of mrkaye97 April 28, 2026 22:06 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Attempt at improving worker actions query#3774

Fix: Attempt at improving worker actions query#3774
mrkaye97 wants to merge 8 commits intomainfrom
mk/worker-page

mrkaye97 commented Apr 28, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

mrkaye97 Apr 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Copilot AI Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	a."tenantId" = @tenantId::UUID
	a."tenantId" = @tenantId::UUID
	AND w."tenantId" = @tenantId::UUID

		@@ -88,7 +89,7 @@ type WorkerRepository interface {
		ListActiveSDKsPerTenant(ctx context.Context) (map[TenantIdSDKTuple]int64, error)

		// GetWorkerActionsByWorkerId returns a list of actions for a worker

-	// GetWorkerActionsByWorkerId returns a list of actions for a worker
+	// GetWorkerActionsForWorkers returns actions keyed by worker action hash.
+	// It takes full worker rows so callers can reuse each worker's persisted actionHash
+	// instead of recomputing it from the worker ID or action list.

	workerIds := make([]uuid.UUID, len(workers))
	workerIds := make([]uuid.UUID, 0, len(workers))

Conversation

mrkaye97 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Uh oh!

vercel Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrkaye97 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrkaye97 commented Apr 28, 2026 •

edited

Loading

vercel Bot commented Apr 28, 2026 •

edited

Loading