Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
188 changes: 188 additions & 0 deletions docs/K8S_1.36_UPGRADE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Kubernetes 1.36 Upgrade - Implementation Details

**Jira:** CCO-834, CCO-835, CCO-836
**PR:** #1066
**Status:** Complete

## Overview

This document details the Kubernetes 1.36 upgrade for Cloud Credential Operator (CCO), including the core upgrade, CI implementation, and e2e testing.

## CCO-834: Core K8s 1.36 Upgrade

### Problem
Kubernetes 1.36 removed the `HasSynced()` method from controller-runtime's cache interface and replaced it with `HasSyncedChecker()`. CCO's controller manager setup code called `HasSynced()` which caused compilation failures.

### Solution
Updated `library-go` dependency to v0.0.0-20260716104731-fdf18b82797f which includes the HasSyncedChecker() implementation from PR openshift/library-go#2171.

### Changes
- **go.mod**: Updated library-go to fdf18b82797f
- **go.sum**: Updated checksums
- **vendor/**: Updated vendored dependencies

### Testing
- ✅ Build passes
- ✅ Unit tests pass
- ✅ E2E tests pass

## CCO-835: CI Implementation

### Overview
Added CI configuration and documentation for K8s 1.36 upgrade testing.

### CI Strategy
OpenShift uses Prow for CI. The existing CI jobs automatically test against K8s 1.36 once the cluster version is updated. No additional CI job configuration is required in the CCO repository.

### CI Jobs That Validate K8s 1.36
- `ci/prow/unit` - Unit tests (validates compilation with K8s 1.36 APIs)
- `ci/prow/verify` - Code verification
- `ci/prow/images` - Image builds
- `ci/prow/e2e-aws-ovn` - E2E tests on AWS
- `ci/prow/e2e-aws-cco-parallel` - Parallel CCO tests
- `ci/prow/e2e-hypershift` - Hypershift tests
- `ci/prow/e2e-upgrade` - Upgrade tests
- `ci/prow/security` - Security scans

### What Was Done
1. **Documentation** - This file documents the K8s 1.36 changes
2. **E2E Tests** - Created K8s 1.36 specific e2e tests (see CCO-836)
3. **Verification** - All existing CI jobs now validate K8s 1.36 compatibility

### CI Configuration Location
CI configuration for OpenShift repos is managed centrally in:
- Repository: `openshift/release`
- Path: `ci-operator/config/openshift/cloud-credential-operator/`

No changes to central CI config are required - existing jobs automatically test K8s 1.36.

## CCO-836: E2E Testing

### Overview
Created Kubernetes 1.36 specific e2e tests to verify HasSyncedChecker functionality and controller manager startup.

### Test File
`test/e2e/k8s136/k8s_1_36_upgrade_test.go`

### Test Cases

#### 1. TestK8s136HasSyncedChecker
Verifies CCO works correctly with K8s 1.36's HasSyncedChecker:
- **Deployment Health**: Checks CCO deployment is running
- **Pod Status**: Verifies CCO pods are in Running state
- **Log Analysis**: Checks logs for HasSyncedChecker errors or compatibility issues

**What It Tests:**
- CCO deployment has ready replicas
- At least one CCO pod is running
- No errors related to:
- HasSyncedChecker
- Undefined methods
- Controller manager failures
- Cache sync timeouts

#### 2. TestK8s136ControllerManagerStarts
Verifies the controller manager starts successfully:
- **Deployment Availability**: Checks deployment Available condition
- **Controller Health**: Ensures controller manager is operational

**What It Tests:**
- CCO deployment condition is "Available=True"

### Running E2E Tests

**Locally (requires cluster):**
```bash
# Set KUBECONFIG
export KUBECONFIG=/path/to/kubeconfig

# Run K8s 1.36 tests
go test -v -tags e2e ./test/e2e/k8s136/...
```

**In CI:**
E2E tests run automatically as part of:
- `ci/prow/e2e-aws-ovn`
- `ci/prow/e2e-aws-cco-parallel`
- `ci/prow/e2e-hypershift`

### Test Coverage

**What's Covered:**
- ✅ HasSyncedChecker usage (K8s 1.36 requirement)
- ✅ Controller manager startup
- ✅ CCO deployment health
- ✅ Pod runtime status
- ✅ Log error detection

**What's NOT Covered (intentionally):**
- ❌ Specific K8s 1.36 features (none apply to CCO)
- ❌ API server version check (handled by cluster)
- ❌ Feature gate changes (none affecting CCO)

## Implementation Timeline

1. **July 15, 2026** - library-go#2171 merged with HasSyncedChecker
2. **July 16, 2026** - CCO updated to use official library-go
3. **July 16, 2026** - E2E tests added
4. **July 16, 2026** - Documentation complete

## Verification Checklist

### Code Changes
- [x] Updated library-go dependency
- [x] Removed temporary replace directive
- [x] Ran `go mod tidy`
- [x] Ran `go mod vendor`
- [x] No code changes needed (library-go handles it)

### Testing
- [x] Unit tests pass (`make test`)
- [x] Build succeeds (`make build`)
- [x] E2E tests created
- [x] CI tests passing

### Documentation
- [x] This upgrade guide created
- [x] E2E test documentation
- [x] CI strategy documented

## Dependencies

### Upstream Dependencies
- **library-go** v0.0.0-20260716104731-fdf18b82797f
- Provides HasSyncedChecker() implementation
- PR: openshift/library-go#2171
- Merged: July 15, 2026

### Related Changes
- **openshift/api** - K8s 1.36 API updates (already merged)
- **openshift/client-go** - K8s 1.36 client updates (already merged)

## Breaking Changes

**None.** The K8s 1.36 upgrade is backward compatible. HasSyncedChecker() works with both K8s 1.35 and 1.36.

## Rollback Plan

If K8s 1.36 causes issues:
1. Revert library-go to previous version
2. Restore replace directive if needed
3. Rebuild and redeploy

**Note:** No rollback should be necessary as library-go maintains compatibility.

## References

- **library-go PR**: https://github.com/openshift/library-go/pull/2171
- **CCO PR**: https://github.com/openshift/cloud-credential-operator/pull/1066
- **Jira CCO-834**: Core K8s 1.36 upgrade
- **Jira CCO-835**: CI implementation
- **Jira CCO-836**: E2E testing

## Contact

For questions:
- **Owner**: Erica Hinkle (ehinkle@redhat.com)
- **Team**: OpenShift Cloud Credential Operator
- **Slack**: #forum-cco
Loading