Skip to content

Feat--enforce-retention #3615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
grutt wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Feat--enforce-retention #3615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a494030
feat: enforce retention on lookup
grutt Apr 14, 2026
9b03999
feat: dont restrict api
grutt Apr 14, 2026
13d6314
feat: retention notice on fe
grutt Apr 14, 2026
8265df2
feat: more options
grutt Apr 14, 2026
81e8c50
feat: expose retention update
grutt Apr 14, 2026
7c50d13
fix: docs page
grutt Apr 14, 2026
7e55d06
chore: relint
grutt Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions api-contracts/openapi/components/schemas/tenant.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ Tenant:
environment:
$ref: "#/TenantEnvironment"
description: The environment type of the tenant.
dataRetentionPeriod:
type: string
description: The data retention period for the tenant, e.g. 720h.
required:
- metadata
- name
Expand Down
10 changes: 10 additions & 0 deletions api/v1/server/handlers/v1/events/get.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,24 @@ package eventsv1
import (
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
"github.com/hatchet-dev/hatchet/pkg/analytics"
v1 "github.com/hatchet-dev/hatchet/pkg/repository"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"
)

func (t *V1EventsService) V1EventGet(ctx echo.Context, request gen.V1EventGetRequestObject) (gen.V1EventGetResponseObject, error) {
tenant := ctx.Get("tenant").(*sqlcv1.Tenant)
event := ctx.Get("v1-event").(*v1.EventWithPayload)

if ts := event.EventSeenAt; ts.Valid && v1handlers.IsBeforeRetention(ts.Time, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx.Request().Context(), analytics.Event, analytics.Get, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +18 to +22
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

return gen.V1EventGet200JSONResponse(
transformers.ToV1Event(event),
), nil
Expand Down
16 changes: 12 additions & 4 deletions api/v1/server/handlers/v1/events/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ import (
"github.com/jackc/pgx/v5/pgtype"
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
"github.com/hatchet-dev/hatchet/pkg/analytics"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"
)

Expand All @@ -22,6 +24,16 @@ func (t *V1EventsService) V1EventList(ctx echo.Context, request gen.V1EventListR
offset := int64(0)
since := time.Now().Add(-time.Hour * 24)

if request.Params.Since != nil {
since = *request.Params.Since
}

if v1handlers.IsBeforeRetention(since, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx.Request().Context(), analytics.Event, analytics.List, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +31 to +35
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

if request.Params.Limit != nil {
limit = *request.Params.Limit
}
Expand All @@ -30,10 +42,6 @@ func (t *V1EventsService) V1EventList(ctx echo.Context, request gen.V1EventListR
offset = *request.Params.Offset
}

if request.Params.Since != nil {
since = *request.Params.Since
}

opts := sqlcv1.ListEventsParams{
Tenantid: tenantId,
Limit: pgtype.Int8{
Expand Down
10 changes: 8 additions & 2 deletions api/v1/server/handlers/v1/logs/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ import (
"github.com/google/uuid"
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
transformers "github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
"github.com/hatchet-dev/hatchet/pkg/analytics"
v1 "github.com/hatchet-dev/hatchet/pkg/repository"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"
"github.com/hatchet-dev/hatchet/pkg/telemetry"

transformers "github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
)

func (t *LogsService) V1TenantLogLineList(ctx echo.Context, request gen.V1TenantLogLineListRequestObject) (gen.V1TenantLogLineListResponseObject, error) {
Expand Down Expand Up @@ -47,6 +47,12 @@ func (t *LogsService) V1TenantLogLineList(ctx echo.Context, request gen.V1Tenant
since = request.Params.Since
}

if since != nil && v1handlers.IsBeforeRetention(*since, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx.Request().Context(), analytics.Log, analytics.List, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +50 to +54
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

if request.Params.Until != nil {
until = request.Params.Until
}
Expand Down
24 changes: 24 additions & 0 deletions api/v1/server/handlers/v1/retention.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package v1

import "time"

const defaultRetention = 720 * time.Hour

func retentionBoundary(retentionPeriod string) time.Time {
retention, err := time.ParseDuration(retentionPeriod)
if err != nil || retention <= 0 {
retention = defaultRetention
}

return time.Now().Add(-retention)
Comment on lines +5 to +13
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retentionBoundary falls back to a hard-coded 720h when parsing fails. The rest of the server already has a configurable default retention (Runtime.Limits.DefaultTenantRetentionPeriod) and the DB default may also evolve; hard-coding here risks inconsistent behavior if defaults change. Consider taking the default from config (or treating invalid/empty retention as “no enforcement” and returning false).

Copilot uses AI. Check for mistakes.
}

// IsBeforeRetention returns true when the given timestamp is older than the
// tenant's retention window (now - retentionPeriod).
func IsBeforeRetention(t time.Time, retentionPeriod string) bool {
if t.IsZero() {
return false
}

return t.Before(retentionBoundary(retentionPeriod))
}
10 changes: 10 additions & 0 deletions api/v1/server/handlers/v1/tasks/get.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ import (
"github.com/jackc/pgx/v5"
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
"github.com/hatchet-dev/hatchet/pkg/analytics"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"

transformers "github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
Expand All @@ -31,6 +33,14 @@ func (t *TasksService) V1TaskGet(ctx echo.Context, request gen.V1TaskGetRequestO
return nil, echo.NewHTTPError(500, "Task type assertion failed")
}

tenant := ctx.Get("tenant").(*sqlcv1.Tenant)

if ts := task.InsertedAt; ts.Valid && v1handlers.IsBeforeRetention(ts.Time, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx.Request().Context(), analytics.TaskRun, analytics.Get, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +38 to +42
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

attempt := request.Params.Attempt

var retryCount *int
Expand Down
8 changes: 8 additions & 0 deletions api/v1/server/handlers/v1/workflow-runs/get.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ import (
"github.com/jackc/pgx/v5"
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
"github.com/hatchet-dev/hatchet/pkg/analytics"
v1 "github.com/hatchet-dev/hatchet/pkg/repository"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"

Expand All @@ -20,6 +22,12 @@ func (t *V1WorkflowRunsService) V1WorkflowRunGet(ctx echo.Context, request gen.V
tenantId := tenant.ID
rawWorkflowRun := ctx.Get("v1-workflow-run").(*v1.V1WorkflowRunPopulator)

if ts := rawWorkflowRun.WorkflowRun.CreatedAt; ts.Valid && v1handlers.IsBeforeRetention(ts.Time, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx.Request().Context(), analytics.WorkflowRun, analytics.Get, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +25 to +29
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handler currently only tracks an analytics signal when the run is outside retention, but still returns the run details. If retention should be enforced at the API layer, consider returning a 404/410 (or a structured error) when the run is older than the tenant’s retention period.

Copilot uses AI. Check for mistakes.
Comment on lines +25 to +29
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

requestContext := ctx.Request().Context()

details, err := t.getWorkflowRunDetails(
Expand Down
34 changes: 25 additions & 9 deletions api/v1/server/handlers/v1/workflow-runs/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@ import (
"github.com/google/uuid"
"github.com/labstack/echo/v4"

v1handlers "github.com/hatchet-dev/hatchet/api/v1/server/handlers/v1"
"github.com/hatchet-dev/hatchet/api/v1/server/oas/gen"
transformers "github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
"github.com/hatchet-dev/hatchet/pkg/analytics"
v1 "github.com/hatchet-dev/hatchet/pkg/repository"
"github.com/hatchet-dev/hatchet/pkg/repository/sqlcv1"
"github.com/hatchet-dev/hatchet/pkg/telemetry"

transformers "github.com/hatchet-dev/hatchet/api/v1/server/oas/transformers/v1"
)

func allOlapStatuses(runningFilter *gen.V1RunningFilter) []sqlcv1.V1ReadableStatusOlap {
Expand Down Expand Up @@ -97,13 +98,21 @@ func normalizeWorkflowRunStatuses(statuses []gen.V1TaskStatus, runningFilter *ge
return normalized
}

func (t *V1WorkflowRunsService) WithDags(ctx context.Context, request gen.V1WorkflowRunListRequestObject, tenantId uuid.UUID) (gen.V1WorkflowRunListResponseObject, error) {
func (t *V1WorkflowRunsService) WithDags(ctx context.Context, request gen.V1WorkflowRunListRequestObject, tenant *sqlcv1.Tenant) (gen.V1WorkflowRunListResponseObject, error) {
ctx, span := telemetry.NewSpan(ctx, "v1-workflow-runs-list-with-dags-tasks")
defer span.End()

tenantId := tenant.ID
since := request.Params.Since

if v1handlers.IsBeforeRetention(since, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx, analytics.WorkflowRun, analytics.List, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +108 to +112
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.
Comment on lines +105 to +112
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IsBeforeRetention is only used to emit an analytics event; the request is still executed with the original since value. If this PR is meant to enforce retention, consider clamping since (and/or returning an empty result / 4xx) when since is before the tenant’s retention boundary so the API can’t return data older than the plan allows.

Copilot uses AI. Check for mistakes.

var (
statuses = allOlapStatuses(request.Params.RunningFilter)
since = request.Params.Since
limit int64 = 50
offset int64
)
Expand Down Expand Up @@ -244,13 +253,21 @@ func (t *V1WorkflowRunsService) WithDags(ctx context.Context, request gen.V1Work
), nil
}

func (t *V1WorkflowRunsService) OnlyTasks(ctx context.Context, request gen.V1WorkflowRunListRequestObject, tenantId uuid.UUID) (gen.V1WorkflowRunListResponseObject, error) {
func (t *V1WorkflowRunsService) OnlyTasks(ctx context.Context, request gen.V1WorkflowRunListRequestObject, tenant *sqlcv1.Tenant) (gen.V1WorkflowRunListResponseObject, error) {
ctx, span := telemetry.NewSpan(ctx, "v1-workflow-runs-list-only-tasks")
defer span.End()

tenantId := tenant.ID
since := request.Params.Since

if v1handlers.IsBeforeRetention(since, tenant.DataRetentionPeriod) {
t.config.Analytics.Count(ctx, analytics.WorkflowRun, analytics.List, analytics.Properties{
"outside_retention": true,
})
}
Comment on lines +263 to +267
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Count() instrumentation, boolean properties should follow the repo convention of has_* (to keep dashboards consistent). Consider renaming outside_retention to something like has_retention_violation (and apply consistently across all retention events).

Copilot generated this review using guidance from repository custom instructions.

var (
statuses = allOlapStatuses(request.Params.RunningFilter)
since = request.Params.Since
workflowIds = []uuid.UUID{}
limit int64 = 50
offset int64
Expand Down Expand Up @@ -353,15 +370,14 @@ func (t *V1WorkflowRunsService) OnlyTasks(ctx context.Context, request gen.V1Wor

func (t *V1WorkflowRunsService) V1WorkflowRunList(ctx echo.Context, request gen.V1WorkflowRunListRequestObject) (gen.V1WorkflowRunListResponseObject, error) {
tenant := ctx.Get("tenant").(*sqlcv1.Tenant)
tenantId := tenant.ID

spanContext, span := telemetry.NewSpan(ctx.Request().Context(), "v1-workflow-runs-list")
defer span.End()

if request.Params.OnlyTasks {
return t.OnlyTasks(spanContext, request, tenantId)
return t.OnlyTasks(spanContext, request, tenant)
} else {
return t.WithDags(spanContext, request, tenantId)
return t.WithDags(spanContext, request, tenant)
}
}

Expand Down
Loading
Loading