Workflow integration#9
Conversation
- plugin/skills/agentops-lifecycle/SKILL.md: 10-step dev→staging→prod guide with per-step agent actions, code examples, validation criteria, common issues - plugin/commands/agentops-lifecycle.md: /agentops-lifecycle command entry point - workflows/single-account-single-agent.json: machine-readable lifecycle definition - workflows/README.md: schema docs and workflow index - README.md: replaces "Coming next" stub with lifecycle documentation - plugin/README.md: skills table and layout updated - plugin/skills/install_skills.sh: installs both skills; stale comments fixed - plugin/skills/install_genie_code_skills.py: installs both skills; verify cell updated; was previously hardcoded to agentops-stacks only
sdonohoo-db
left a comment
sdonohoo-db
left a comment
There was a problem hiding this comment.
The 10-step structure, the per-step validations, and the machine-readable JSON look good. I want to flag one thing that needs sorting across a this and Veena's PRs, because it's my mistake.
The guide documents a scaffold layout for eval that doesn't exist. The SKILL and README state that evaluation/gate.py, evaluation/thresholds.yml, and
evaluation/golden_dataset.jsonl are "generated by the scaffold." They aren't. The scaffold template I built only stubs the CI/CD eval hook. The fully-populated
evaluation/ files exist only in examples/hello-agent/, which was a throwaway
deployment test I built to test the hooks but were no scaffold output. I think the assistant picked up on the example and assumed the scaffold created the eval gate files
It is also different from Veena's PR #15. That PR does add eval generation to the
scaffold but as src/agents/<agent>/eval/{evaluate_agent,create_dataset,utils}.py
eval/gates.ymlandsrc/components/eval/scorers.py. So even once #15 merges,
this guide would still point at the wrong paths and names. #13 establishes the same
src/agents/…+src/components/…convention for retrieval.
Proposed path: let's go with the src/agents/<agent>/… layout from #13/#15 as
canonical, merge those first, then update this PR's file references to match.
|
|
||
| --- | ||
|
|
||
| ## Reference files |
There was a problem hiding this comment.
Most of these don't match what the scaffold generates.
evaluation/gate.py / thresholds.yml / golden_dataset.jsonl only exist in hello-agent (a throwaway test example - my mistake).
Veena's PR #15 generates the real equivalents at src/agents//eval/evaluate_agent.py, eval/gates.yml, eval/create_dataset.py, and src/components/eval/scorers.py. I suggest we go with that layout and align this table to it.
|
|
||
| ## Step 4 — Offline Evaluation & Eval Gate Setup | ||
|
|
||
| Build the evaluation framework **before** any code leaves dev. The `evaluation/` |
There was a problem hiding this comment.
"the evaluation/ directory is generated by the scaffold" - it isn't today. After PR #15 it'll be src/agents//eval/. Reword to point there, and state explicitly whether the user runs a generated file or authors it.
|
|
||
| ### Register to Unity Catalog | ||
|
|
||
| Run `notebooks/register_agent.py` (generated by the scaffold). Set the |
There was a problem hiding this comment.
References src/my_agent.py and notebooks/register_agent.py. #13 establishes src/agents//graph.py. Let's settle on one agent-code path and use it consistently across all 10 steps.
|
|
||
| MLflow is the operational spine at every level: experiment tracking in dev, | ||
| eval gate in CI, trace logging in prod, user feedback via `mlflow.log_feedback()`. | ||
| The `evaluation/gate.py` pattern (generated by the scaffold) runs at three |
There was a problem hiding this comment.
Same path issue here. Also: main intentionally trimmed this section to a one-line "Coming next" stub three days before this branch - worth a quick confirm that re-expanding it now is the call, vs. keeping the README lean and letting the SKILL carry the detail.
| "escalation_hint": "If SME agreement < 80% after 2 rounds: escalate to product owner to clarify evaluation rubric. The rubric definition is a business decision, not an engineering one." | ||
| }, | ||
| { | ||
| "name": "ci_staging", |
There was a problem hiding this comment.
Nitpick - Name conflates CI and staging; SKILL treats them as separate steps 6 and 7. Suggest ci_gate for consistency since this is the machine-readable source of truth.
sdonohoo-db
left a comment
sdonohoo-db
left a comment
There was a problem hiding this comment.
- Stale reference to my agentops-stacks repo to scaffold a project. My repo is no longer current.
- Doc and example workflow rely on a workflow engine that I cannot find the implementation for. /innovate isn't available to customers as far as I know
| "Collect: project_name (^[a-z][a-z0-9_]{2,}$), cloud (aws|azure|gcp), cicd_platform (github_actions|azure_devops|gitlab), destination", | ||
| "Confirm inputs with user before running", | ||
| "Write config to /tmp/agentops-stacks-inputs.json", | ||
| "Run: databricks bundle init https://github.com/sdonohoo-db/agentops-stacks --config-file /tmp/agentops-stacks-inputs.json --output-dir <destination>", |
There was a problem hiding this comment.
This is pointing at my repo which is no longer current. Should be databricks-solutions
|
|
||
| Machine-readable lifecycle definitions for agentops-stacks projects. Each file | ||
| defines a complete end-to-end workflow that can be consumed programmatically by | ||
| workflow engines (e.g., `/innovate`) or used as a reference when building custom |
There was a problem hiding this comment.
Where is the workflow engine defined for agentops? We can't use /innovate as it isn't public.
2 participants
[agentops-lifecycle] Add 10-step lifecycle workflow + fix installers
with per-step agent actions, code examples, validation criteria, common issues
updated; was previously hardcoded to agentops-stacks only