feat: volatility targets, CZAR loss, and testnet topic examples - #31
Open
jefferythewind wants to merge 7 commits into
Open
feat: volatility targets, CZAR loss, and testnet topic examples#31jefferythewind wants to merge 7 commits into
jefferythewind wants to merge 7 commits into
Conversation
Core library changes: - workflow.py: Add target_type parameter (log_return | volatility) and compute_volatility_target_polars() for std of 1-min log returns - czar_loss.py (new): CZAR directional loss with gradient/hessian for custom LightGBM training — penalizes wrong-sign predictions, softens near-zero returns, normalizes by local volatility - __init__.py: Export czar_loss, czar_gradient, czar_hessian, make_czar_objective Tests: - test_volatility_target.py: 8 tests for volatility target computation Notebooks (example/illustration code): - Reorganize all topic scripts under notebooks/testnet/topic_*/ - Add example scripts for all testnet topics: 38, 41, 42 (8h price), 57, 83, 84 (8h log-return), 61, 62, 63 (24h log-return), 71 (NEAR 8h), 79-82 (15m volatility) - Add CZAR V1 model scripts for 8h price topics (38, 41, 42) - Add dashboard.sh convenience script - Remove notebooks/shared/ — keep deploy scripts at top level Docs: - README: Add topic reference tables, volatility workflow example - AGENTS.md: Fix paths for testnet/ subfolder - .gitignore: Add **/runs/ for training artifacts
The reputer's ground truth applies standardization_ratio = √(timeframe/frequency) to convert per-bar std to horizon volatility. Our compute_volatility_target_polars() was missing this scaling, causing workers to submit values ~√15 ≈ 3.87× too low on 15-minute volatility topics. Fix: multiply rolling std by √target_bars in the target computation. Tests updated to verify the scaling.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
40 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="notebooks/testnet/topic_42_btc_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/example.py:358">
P1: Checkpoint evaluation is outside the n_est loop, so only the last tree-count checkpoint is actually scored and ranked. Grid-search results and selected best config are therefore incorrect.</violation>
<violation number="2" location="notebooks/testnet/topic_42_btc_8h_price/example.py:566">
P2: The primary `predict` uses the full `feature_cols` set even though `final_model` was trained on the top‑k `best_selected` features. LightGBM will reject the mismatched feature count or produce incorrect inference. Use the same selected feature list as training.</violation>
</file>
<file name="notebooks/testnet/topic_62_sol_24h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_62_sol_24h_logreturn/example.py:163">
P2: Passing a notebook-local api_key_file narrows key lookup and skips the standard fallback paths, so valid repo-root keys are ignored. Use default `get_api_key()` resolution instead.
(Based on your team's feedback about preserving the canonical API key fallback chain.) [FEEDBACK_USED]</violation>
</file>
<file name="notebooks/testnet/topic_83_btc_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_83_btc_8h_logreturn/example.py:390">
P2: predict() unnecessarily requires a valid current_price for a log-return topic. If price lookup fails, inference aborts even though model output does not depend on current_price.</violation>
</file>
<file name="allora_forge_builder_kit/czar_loss.py">
<violation number="1" location="allora_forge_builder_kit/czar_loss.py:113">
P2: `czar_gradient()` is inconsistent with `czar_loss()` for `y_true == 0`, producing incorrect gradients on zero-return samples.</violation>
<violation number="2" location="allora_forge_builder_kit/czar_loss.py:156">
P1: `make_czar_objective()` uses the wrong argument order for LightGBM `fobj`, so the returned objective is not actually compatible with `lightgbm.train` and will fail at runtime.</violation>
<violation number="3" location="allora_forge_builder_kit/czar_loss.py:158">
P1: Custom objective argument handling is incompatible with LightGBM fobj signature. Using this function with `lgb.train(..., fobj=...)` will misread inputs and fail during gradient/hessian math.</violation>
</file>
<file name="notebooks/dashboard.sh">
<violation number="1" location="notebooks/dashboard.sh:7">
P2: Abort if venv activation fails; otherwise the wrapper can run with the wrong Python. Add an explicit failure check on the activation step.</violation>
</file>
<file name="notebooks/testnet/topic_41_eth_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_41_eth_8h_price/example.py:358">
P1: Checkpoint evaluation is outside the `n_est` loop, so only the last checkpoint is scored and stored. Grid-search ranking is therefore incorrect for `n_estimators`.</violation>
<violation number="2" location="notebooks/testnet/topic_41_eth_8h_price/example.py:566">
P0: Inference uses `feature_cols` instead of the trained subset, causing model input mismatch. Use `best_selected` for the primary model prediction.</violation>
</file>
<file name="notebooks/testnet/topic_42_btc_8h_price/model_v3_czar.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/model_v3_czar.py:52">
P2: Passing a notebook-local api_key_file bypasses the normal fallback search and can miss a valid root `.allora_api_key`.
(Based on your team's feedback about keeping canonical API-key resolution order and avoiding bespoke notebook paths.) [FEEDBACK_USED]</violation>
</file>
<file name="notebooks/testnet/topic_61_btc_24h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_61_btc_24h_logreturn/example.py:390">
P1: predict() can fail on missing current_price even though current_price is not used for log-return output. This creates avoidable inference failures and unnecessary network dependency.</violation>
</file>
<file name="notebooks/testnet/topic_42_btc_8h_price/model_v2_directional.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/model_v2_directional.py:352">
P1: predict() can return NaN when live price lookup fails, instead of failing fast. This creates pickles that emit non-finite outputs and break runtime inference validation.</violation>
</file>
<file name="notebooks/testnet/topic_41_eth_8h_price/model_czar.py">
<violation number="1" location="notebooks/testnet/topic_41_eth_8h_price/model_czar.py:309">
P1: The serialized predict function captures `workflow`, which can include API credentials, so the generated .pkl may leak the Allora API key.</violation>
</file>
<file name="notebooks/testnet/topic_71_near_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_71_near_8h_logreturn/example.py:219">
P2: Return features use bar offsets that do not match the configured 5-minute interval. The "1h/6h/12h/24h" features currently encode much shorter horizons.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/model_czar.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/model_czar.py:159">
P2: Rolling volatility is not clipped above zero before passing into CZAR objective. Flat target windows can produce `std == 0`, causing divide-by-zero in CZAR gradient/hessian.</violation>
</file>
<file name="allora_forge_builder_kit/workflow.py">
<violation number="1" location="allora_forge_builder_kit/workflow.py:356">
P2: Volatility target generation is not guarded to `interval='1m'`, so invalid intervals silently produce misdefined targets.</violation>
<violation number="2" location="allora_forge_builder_kit/workflow.py:356">
P1: Volatility target is computed without enforcing 1-minute interval, so non-1m workflows produce incorrect labels. Add a runtime check before computing volatility targets.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/example.py:163">
P2: The API-key lookup uses a notebook-local path that bypasses the canonical fallback chain and can cause unexpected interactive prompting/failure even when a valid repo key file exists.
(Based on your team's feedback about consistent API key resolution order and avoiding bespoke notebook-local key paths.) [FEEDBACK_USED].</violation>
<violation number="2" location="notebooks/testnet/topic_38_sol_8h_price/example.py:391">
P1: Final model config diverges from CV config, so selected best params do not match deployed behavior. This can invalidate reported scores and degrade live performance.</violation>
</file>
<file name="notebooks/testnet/topic_84_eth_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_84_eth_8h_logreturn/example.py:163">
P2: API key lookup path is non-canonical and can miss the repo-root key file. This breaks non-interactive runs even when .allora_api_key exists at repository root.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/model_v3_methodology.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/model_v3_methodology.py:61">
P2: Custom api_key_file path breaks the standard key-resolution chain and can miss valid root credentials in automated runs.
(Based on your team's feedback about API key fallback chain.) [FEEDBACK_USED]</violation>
</file>
<file name="skills/allora-model-builder/SKILL.md">
<violation number="1" location="skills/allora-model-builder/SKILL.md:180">
P2: Stale path reference: missing `testnet/` prefix in the path to the volatility example script</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| raise ValueError(f"Invalid current price for inference: {current_price}") | ||
|
|
||
| # Predict log return | ||
| predicted_log_return = final_model.predict(live_features[feature_cols].values.reshape(1, -1))[0] |
There was a problem hiding this comment.
P0: Inference uses feature_cols instead of the trained subset, causing model input mismatch. Use best_selected for the primary model prediction.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_41_eth_8h_price/example.py, line 566:
<comment>Inference uses `feature_cols` instead of the trained subset, causing model input mismatch. Use `best_selected` for the primary model prediction.</comment>
<file context>
@@ -0,0 +1,625 @@
+ raise ValueError(f"Invalid current price for inference: {current_price}")
+
+ # Predict log return
+ predicted_log_return = final_model.predict(live_features[feature_cols].values.reshape(1, -1))[0]
+
+ # Convert log return to price
</file context>
| fold_models.append((lgb, test_idx, selected)) | ||
|
|
||
| # Evaluate at tree count checkpoints | ||
| for n_est in N_ESTIMATORS_CHECKPOINTS: |
There was a problem hiding this comment.
P1: Checkpoint evaluation is outside the n_est loop, so only the last checkpoint is scored and stored. Grid-search ranking is therefore incorrect for n_estimators.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_41_eth_8h_price/example.py, line 358:
<comment>Checkpoint evaluation is outside the `n_est` loop, so only the last checkpoint is scored and stored. Grid-search ranking is therefore incorrect for `n_estimators`.</comment>
<file context>
@@ -0,0 +1,625 @@
+ fold_models.append((lgb, test_idx, selected))
+
+ # Evaluate at tree count checkpoints
+ for n_est in N_ESTIMATORS_CHECKPOINTS:
+ config_num += 1
+ df_all['pred'] = np.nan
</file context>
| fold_models.append((lgb, test_idx, selected)) | ||
|
|
||
| # Evaluate at tree count checkpoints | ||
| for n_est in N_ESTIMATORS_CHECKPOINTS: |
There was a problem hiding this comment.
P1: Checkpoint evaluation is outside the n_est loop, so only the last tree-count checkpoint is actually scored and ranked. Grid-search results and selected best config are therefore incorrect.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_42_btc_8h_price/example.py, line 358:
<comment>Checkpoint evaluation is outside the n_est loop, so only the last tree-count checkpoint is actually scored and ranked. Grid-search results and selected best config are therefore incorrect.</comment>
<file context>
@@ -0,0 +1,625 @@
+ fold_models.append((lgb, test_idx, selected))
+
+ # Evaluate at tree count checkpoints
+ for n_est in N_ESTIMATORS_CHECKPOINTS:
+ config_num += 1
+ df_all['pred'] = np.nan
</file context>
| Returns: | ||
| objective function compatible with LightGBM's fobj parameter | ||
| """ | ||
| def objective(y_true_or_dataset, y_pred): |
There was a problem hiding this comment.
P1: make_czar_objective() uses the wrong argument order for LightGBM fobj, so the returned objective is not actually compatible with lightgbm.train and will fail at runtime.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At allora_forge_builder_kit/czar_loss.py, line 156:
<comment>`make_czar_objective()` uses the wrong argument order for LightGBM `fobj`, so the returned objective is not actually compatible with `lightgbm.train` and will fail at runtime.</comment>
<file context>
@@ -0,0 +1,167 @@
+ Returns:
+ objective function compatible with LightGBM's fobj parameter
+ """
+ def objective(y_true_or_dataset, y_pred):
+ # Handle both LightGBM Dataset objects and raw arrays
+ if hasattr(y_true_or_dataset, 'get_label'):
</file context>
| h1 = d2p1 * double_derivative(delta) | ||
| h3 = d2p1 * double_derivative(d_true) | ||
|
|
||
| G1 = h1 * z_pred - np.sign(z_true) |
There was a problem hiding this comment.
P2: czar_gradient() is inconsistent with czar_loss() for y_true == 0, producing incorrect gradients on zero-return samples.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At allora_forge_builder_kit/czar_loss.py, line 113:
<comment>`czar_gradient()` is inconsistent with `czar_loss()` for `y_true == 0`, producing incorrect gradients on zero-return samples.</comment>
<file context>
@@ -0,0 +1,167 @@
+ h1 = d2p1 * double_derivative(delta)
+ h3 = d2p1 * double_derivative(d_true)
+
+ G1 = h1 * z_pred - np.sign(z_true)
+ G2 = -s * d2p1 * derivative(d_pred)
+ G3 = np.minimum(h3, h1) * (z_pred - z_true)
</file context>
Suggested change
| G1 = h1 * z_pred - np.sign(z_true) | |
| G1 = h1 * z_pred - s |
| raise ValueError(f"Invalid current price for inference: {current_price}") | ||
|
|
||
| # Predict log return | ||
| predicted_log_return = final_model.predict(live_features[feature_cols].values.reshape(1, -1))[0] |
There was a problem hiding this comment.
P2: The primary predict uses the full feature_cols set even though final_model was trained on the top‑k best_selected features. LightGBM will reject the mismatched feature count or produce incorrect inference. Use the same selected feature list as training.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_42_btc_8h_price/example.py, line 566:
<comment>The primary `predict` uses the full `feature_cols` set even though `final_model` was trained on the top‑k `best_selected` features. LightGBM will reject the mismatched feature count or produce incorrect inference. Use the same selected feature list as training.</comment>
<file context>
@@ -0,0 +1,625 @@
+ raise ValueError(f"Invalid current price for inference: {current_price}")
+
+ # Predict log return
+ predicted_log_return = final_model.predict(live_features[feature_cols].values.reshape(1, -1))[0]
+
+ # Convert log return to price
</file context>
| # Get a free key at https://developer.allora.network | ||
| # Alternatively, set data_source="binance" below to skip the API key entirely. | ||
| from allora_forge_builder_kit.utils import get_api_key | ||
| api_key = get_api_key(api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key")) |
There was a problem hiding this comment.
P2: Passing a notebook-local api_key_file narrows key lookup and skips the standard fallback paths, so valid repo-root keys are ignored. Use default get_api_key() resolution instead.
(Based on your team's feedback about preserving the canonical API key fallback chain.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_62_sol_24h_logreturn/example.py, line 163:
<comment>Passing a notebook-local api_key_file narrows key lookup and skips the standard fallback paths, so valid repo-root keys are ignored. Use default `get_api_key()` resolution instead.
(Based on your team's feedback about preserving the canonical API key fallback chain.) </comment>
<file context>
@@ -0,0 +1,427 @@
+# Get a free key at https://developer.allora.network
+# Alternatively, set data_source="binance" below to skip the API key entirely.
+from allora_forge_builder_kit.utils import get_api_key
+api_key = get_api_key(api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key"))
+
+workflow = AlloraMLWorkflow(
</file context>
Replace 14 separate model variant scripts (model_a through model_e, walkthroughs) with 1 model_grid_retrain.py per volatility topic. The grid retrain script does a systematic search over objectives (MSE, Huber) × log-space × LightGBM hyperparams and saves the top 5 models for deployment. Net reduction: -4,450 lines across vol topics (79, 80, 81, 82).
…lots Replace row-by-row df.apply with vectorized numpy operations for vol feature engineering. 1.15M rows: 40 min → 6 seconds. Also adds scatter plot of predictions vs true values at the end of each grid retrain script to visually confirm predictions are on the correct √T-scaled magnitude.
There was a problem hiding this comment.
6 issues found across 18 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="notebooks/testnet/topic_42_btc_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/example.py:358">
P1: Checkpoint evaluation is outside the n_est loop, so only the last tree-count checkpoint is actually scored and ranked. Grid-search results and selected best config are therefore incorrect.</violation>
<violation number="2" location="notebooks/testnet/topic_42_btc_8h_price/example.py:566">
P2: The primary `predict` uses the full `feature_cols` set even though `final_model` was trained on the top‑k `best_selected` features. LightGBM will reject the mismatched feature count or produce incorrect inference. Use the same selected feature list as training.</violation>
</file>
<file name="notebooks/testnet/topic_62_sol_24h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_62_sol_24h_logreturn/example.py:163">
P2: Passing a notebook-local api_key_file narrows key lookup and skips the standard fallback paths, so valid repo-root keys are ignored. Use default `get_api_key()` resolution instead.
(Based on your team's feedback about preserving the canonical API key fallback chain.) [FEEDBACK_USED]</violation>
</file>
<file name="notebooks/testnet/topic_83_btc_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_83_btc_8h_logreturn/example.py:390">
P2: predict() unnecessarily requires a valid current_price for a log-return topic. If price lookup fails, inference aborts even though model output does not depend on current_price.</violation>
</file>
<file name="allora_forge_builder_kit/czar_loss.py">
<violation number="1" location="allora_forge_builder_kit/czar_loss.py:113">
P2: `czar_gradient()` is inconsistent with `czar_loss()` for `y_true == 0`, producing incorrect gradients on zero-return samples.</violation>
<violation number="2" location="allora_forge_builder_kit/czar_loss.py:156">
P1: `make_czar_objective()` uses the wrong argument order for LightGBM `fobj`, so the returned objective is not actually compatible with `lightgbm.train` and will fail at runtime.</violation>
<violation number="3" location="allora_forge_builder_kit/czar_loss.py:158">
P1: Custom objective argument handling is incompatible with LightGBM fobj signature. Using this function with `lgb.train(..., fobj=...)` will misread inputs and fail during gradient/hessian math.</violation>
</file>
<file name="notebooks/dashboard.sh">
<violation number="1" location="notebooks/dashboard.sh:7">
P2: Abort if venv activation fails; otherwise the wrapper can run with the wrong Python. Add an explicit failure check on the activation step.</violation>
</file>
<file name="notebooks/testnet/topic_41_eth_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_41_eth_8h_price/example.py:358">
P1: Checkpoint evaluation is outside the `n_est` loop, so only the last checkpoint is scored and stored. Grid-search ranking is therefore incorrect for `n_estimators`.</violation>
<violation number="2" location="notebooks/testnet/topic_41_eth_8h_price/example.py:566">
P0: Inference uses `feature_cols` instead of the trained subset, causing model input mismatch. Use `best_selected` for the primary model prediction.</violation>
</file>
<file name="notebooks/testnet/topic_42_btc_8h_price/model_v3_czar.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/model_v3_czar.py:52">
P2: Passing a notebook-local api_key_file bypasses the normal fallback search and can miss a valid root `.allora_api_key`.
(Based on your team's feedback about keeping canonical API-key resolution order and avoiding bespoke notebook paths.) [FEEDBACK_USED]</violation>
</file>
<file name="notebooks/testnet/topic_61_btc_24h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_61_btc_24h_logreturn/example.py:390">
P1: predict() can fail on missing current_price even though current_price is not used for log-return output. This creates avoidable inference failures and unnecessary network dependency.</violation>
</file>
<file name="notebooks/testnet/topic_42_btc_8h_price/model_v2_directional.py">
<violation number="1" location="notebooks/testnet/topic_42_btc_8h_price/model_v2_directional.py:352">
P1: predict() can return NaN when live price lookup fails, instead of failing fast. This creates pickles that emit non-finite outputs and break runtime inference validation.</violation>
</file>
<file name="notebooks/testnet/topic_41_eth_8h_price/model_czar.py">
<violation number="1" location="notebooks/testnet/topic_41_eth_8h_price/model_czar.py:309">
P1: The serialized predict function captures `workflow`, which can include API credentials, so the generated .pkl may leak the Allora API key.</violation>
</file>
<file name="notebooks/testnet/topic_71_near_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_71_near_8h_logreturn/example.py:219">
P2: Return features use bar offsets that do not match the configured 5-minute interval. The "1h/6h/12h/24h" features currently encode much shorter horizons.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/model_czar.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/model_czar.py:159">
P2: Rolling volatility is not clipped above zero before passing into CZAR objective. Flat target windows can produce `std == 0`, causing divide-by-zero in CZAR gradient/hessian.</violation>
</file>
<file name="allora_forge_builder_kit/workflow.py">
<violation number="1" location="allora_forge_builder_kit/workflow.py:356">
P2: Volatility target generation is not guarded to `interval='1m'`, so invalid intervals silently produce misdefined targets.</violation>
<violation number="2" location="allora_forge_builder_kit/workflow.py:356">
P1: Volatility target is computed without enforcing 1-minute interval, so non-1m workflows produce incorrect labels. Add a runtime check before computing volatility targets.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/example.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/example.py:163">
P2: The API-key lookup uses a notebook-local path that bypasses the canonical fallback chain and can cause unexpected interactive prompting/failure even when a valid repo key file exists.
(Based on your team's feedback about consistent API key resolution order and avoiding bespoke notebook-local key paths.) [FEEDBACK_USED].</violation>
<violation number="2" location="notebooks/testnet/topic_38_sol_8h_price/example.py:391">
P1: Final model config diverges from CV config, so selected best params do not match deployed behavior. This can invalidate reported scores and degrade live performance.</violation>
</file>
<file name="notebooks/testnet/topic_84_eth_8h_logreturn/example.py">
<violation number="1" location="notebooks/testnet/topic_84_eth_8h_logreturn/example.py:163">
P2: API key lookup path is non-canonical and can miss the repo-root key file. This breaks non-interactive runs even when .allora_api_key exists at repository root.</violation>
</file>
<file name="notebooks/testnet/topic_38_sol_8h_price/model_v3_methodology.py">
<violation number="1" location="notebooks/testnet/topic_38_sol_8h_price/model_v3_methodology.py:61">
P2: Custom api_key_file path breaks the standard key-resolution chain and can miss valid root credentials in automated runs.
(Based on your team's feedback about API key fallback chain.) [FEEDBACK_USED]</violation>
</file>
<file name="skills/allora-model-builder/SKILL.md">
<violation number="1" location="skills/allora-model-builder/SKILL.md:180">
P2: Stale path reference: missing `testnet/` prefix in the path to the volatility example script</violation>
</file>
<file name="notebooks/testnet/topic_80_eth_vol/model_grid_retrain.py">
<violation number="1" location="notebooks/testnet/topic_80_eth_vol/model_grid_retrain.py:75">
P2: API key lookup is pinned to notebooks/.allora_api_key, bypassing the default fallback chain that includes repo-root .allora_api_key. This can break non-interactive runs when only the root key file is present.
(Based on your team's feedback about API key fallback chain consistency.) [FEEDBACK_USED].</violation>
</file>
<file name="notebooks/testnet/topic_82_sol_vol/model_grid_retrain.py">
<violation number="1" location="notebooks/testnet/topic_82_sol_vol/model_grid_retrain.py:75">
P2: API key path points to notebooks-local file and bypasses root-file fallback in current `get_api_key` behavior; this can break non-interactive runs when only repo-root `.allora_api_key` is present.
(Based on your team's feedback about canonical API-key fallback order and avoiding notebook-local path divergence.) [FEEDBACK_USED].</violation>
</file>
<file name="notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py">
<violation number="1" location="notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py:7">
P3: Docstring claims diversity selection, but code deploys plain top-K by score only. This is misleading for users interpreting model-selection behavior.</violation>
<violation number="2" location="notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py:74">
P2: API key loading is constrained to notebooks/.allora_api_key by passing a custom path, so root `.allora_api_key` is skipped when env var is unset. This breaks the documented key-file setup and can cause interactive prompt/failure in non-interactive runs.</violation>
<violation number="3" location="notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py:105">
P3: This file introduces a large copy-pasted pipeline duplicated across the other three new topic scripts, which increases maintenance drift risk and makes future fixes error-prone.</violation>
<violation number="4" location="notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py:189">
P2: Training and inference feature logic is duplicated in two implementations, increasing risk of train/serve skew when one side changes. Keep a single shared feature definition to prevent silent divergence.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| # ============================================================================= | ||
| print("\n[1/4] Loading data...") | ||
| api_key = get_api_key( | ||
| api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key") |
There was a problem hiding this comment.
P2: API key lookup is pinned to notebooks/.allora_api_key, bypassing the default fallback chain that includes repo-root .allora_api_key. This can break non-interactive runs when only the root key file is present.
(Based on your team's feedback about API key fallback chain consistency.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_80_eth_vol/model_grid_retrain.py, line 75:
<comment>API key lookup is pinned to notebooks/.allora_api_key, bypassing the default fallback chain that includes repo-root .allora_api_key. This can break non-interactive runs when only the root key file is present.
(Based on your team's feedback about API key fallback chain consistency.) .</comment>
<file context>
@@ -0,0 +1,427 @@
+# =============================================================================
+print("\n[1/4] Loading data...")
+api_key = get_api_key(
+ api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key")
+)
+
</file context>
| # ============================================================================= | ||
| print("\n[1/4] Loading data...") | ||
| api_key = get_api_key( | ||
| api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key") |
There was a problem hiding this comment.
P2: API key path points to notebooks-local file and bypasses root-file fallback in current get_api_key behavior; this can break non-interactive runs when only repo-root .allora_api_key is present.
(Based on your team's feedback about canonical API-key fallback order and avoiding notebook-local path divergence.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_82_sol_vol/model_grid_retrain.py, line 75:
<comment>API key path points to notebooks-local file and bypasses root-file fallback in current `get_api_key` behavior; this can break non-interactive runs when only repo-root `.allora_api_key` is present.
(Based on your team's feedback about canonical API-key fallback order and avoiding notebook-local path divergence.) .</comment>
<file context>
@@ -0,0 +1,427 @@
+# =============================================================================
+print("\n[1/4] Loading data...")
+api_key = get_api_key(
+ api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key")
+)
+
</file context>
| return pd.DataFrame(out, index=df_in.index) | ||
|
|
||
| # Row-level version for live inference (single row) | ||
| def engineer_features(row): |
There was a problem hiding this comment.
P2: Training and inference feature logic is duplicated in two implementations, increasing risk of train/serve skew when one side changes. Keep a single shared feature definition to prevent silent divergence.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py, line 189:
<comment>Training and inference feature logic is duplicated in two implementations, increasing risk of train/serve skew when one side changes. Keep a single shared feature definition to prevent silent divergence.</comment>
<file context>
@@ -0,0 +1,427 @@
+ return pd.DataFrame(out, index=df_in.index)
+
+# Row-level version for live inference (single row)
+def engineer_features(row):
+ """Single-row feature engineering for live prediction."""
+ n = NUMBER_OF_INPUT_BARS
</file context>
Comment on lines
+74
to
+76
| api_key = get_api_key( | ||
| api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key") | ||
| ) |
There was a problem hiding this comment.
P2: API key loading is constrained to notebooks/.allora_api_key by passing a custom path, so root .allora_api_key is skipped when env var is unset. This breaks the documented key-file setup and can cause interactive prompt/failure in non-interactive runs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py, line 74:
<comment>API key loading is constrained to notebooks/.allora_api_key by passing a custom path, so root `.allora_api_key` is skipped when env var is unset. This breaks the documented key-file setup and can cause interactive prompt/failure in non-interactive runs.</comment>
<file context>
@@ -0,0 +1,427 @@
+# LOAD DATA
+# =============================================================================
+print("\n[1/4] Loading data...")
+api_key = get_api_key(
+ api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key")
+)
</file context>
Suggested change
| api_key = get_api_key( | |
| api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key") | |
| ) | |
| api_key = get_api_key() |
| ======================================================== | ||
|
|
||
| Grid search over objectives × LightGBM hyperparams. | ||
| Saves top 5 diverse models for deployment. |
There was a problem hiding this comment.
P3: Docstring claims diversity selection, but code deploys plain top-K by score only. This is misleading for users interpreting model-selection behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py, line 7:
<comment>Docstring claims diversity selection, but code deploys plain top-K by score only. This is misleading for users interpreting model-selection behavior.</comment>
<file context>
@@ -0,0 +1,427 @@
+========================================================
+
+Grid search over objectives × LightGBM hyperparams.
+Saves top 5 diverse models for deployment.
+
+Now with corrected √T scaling in the target (matching the reputer).
</file context>
| # ============================================================================= | ||
| print("\n[2/4] Engineering features (vectorized)...") | ||
|
|
||
| def engineer_features_vectorized(df_in, n_input_bars): |
There was a problem hiding this comment.
P3: This file introduces a large copy-pasted pipeline duplicated across the other three new topic scripts, which increases maintenance drift risk and makes future fixes error-prone.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py, line 105:
<comment>This file introduces a large copy-pasted pipeline duplicated across the other three new topic scripts, which increases maintenance drift risk and makes future fixes error-prone.</comment>
<file context>
@@ -0,0 +1,427 @@
+# =============================================================================
+print("\n[2/4] Engineering features (vectorized)...")
+
+def engineer_features_vectorized(df_in, n_input_bars):
+ """Vectorized feature engineering for volatility topics.
+ Operates on the full DataFrame at once using 2D numpy arrays."""
</file context>
- model_grid_retrain.py: grid search over objectives × LightGBM hyperparams - model_importance_groups.py: importance-based feature group diversity search - 960 input bars (16h lookback), 240 target bars (4h horizon) - README: add topic 85 to testnet topic reference table
There was a problem hiding this comment.
4 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="notebooks/testnet/topic_85_eth_4h_vol/model_importance_groups.py">
<violation number="1" location="notebooks/testnet/topic_85_eth_4h_vol/model_importance_groups.py:45">
P3: Unused configuration constant `TOP_K_PER_GROUP` introduces dead code and misleading configurability</violation>
<violation number="2" location="notebooks/testnet/topic_85_eth_4h_vol/model_importance_groups.py:358">
P1: Smoke-test failure does not block artifact persistence. The except block catches the exception but the pickle is still written unconditionally afterward, allowing broken model artifacts to be saved.</violation>
</file>
<file name="notebooks/testnet/topic_85_eth_4h_vol/model_grid_retrain.py">
<violation number="1" location="notebooks/testnet/topic_85_eth_4h_vol/model_grid_retrain.py:75">
P2: Notebook hardcodes a non-canonical API key file path (`../../.allora_api_key`) that resolves to `notebooks/.allora_api_key` instead of the repository root. This diverges from the canonical `worker_runtime._load_api_key` fallback chain and short-circuits `get_api_key`'s well-known search paths, preventing fallback to the repo-root key.</violation>
<violation number="2" location="notebooks/testnet/topic_85_eth_4h_vol/model_grid_retrain.py:105">
P2: Duplicate feature engineering logic between companion scripts in the same directory</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
| val = fn() | ||
| print(f" {fg_label}: score={cfg['score']:+.4f} → {val:.6f} → {pkl}") | ||
| except Exception as e: | ||
| print(f" {fg_label}: FAILED ({e}) → {pkl}") |
There was a problem hiding this comment.
P1: Smoke-test failure does not block artifact persistence. The except block catches the exception but the pickle is still written unconditionally afterward, allowing broken model artifacts to be saved.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_85_eth_4h_vol/model_importance_groups.py, line 358:
<comment>Smoke-test failure does not block artifact persistence. The except block catches the exception but the pickle is still written unconditionally afterward, allowing broken model artifacts to be saved.</comment>
<file context>
@@ -0,0 +1,414 @@
+ val = fn()
+ print(f" {fg_label}: score={cfg['score']:+.4f} → {val:.6f} → {pkl}")
+ except Exception as e:
+ print(f" {fg_label}: FAILED ({e}) → {pkl}")
+ with open(pkl, "wb") as f:
+ cloudpickle.dump(fn, f)
</file context>
| # ============================================================================= | ||
| print("\n[2/4] Engineering features (vectorized)...") | ||
|
|
||
| def engineer_features_vectorized(df_in, n_input_bars): |
There was a problem hiding this comment.
P2: Duplicate feature engineering logic between companion scripts in the same directory
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_85_eth_4h_vol/model_grid_retrain.py, line 105:
<comment>Duplicate feature engineering logic between companion scripts in the same directory</comment>
<file context>
@@ -0,0 +1,427 @@
+# =============================================================================
+print("\n[2/4] Engineering features (vectorized)...")
+
+def engineer_features_vectorized(df_in, n_input_bars):
+ """Vectorized feature engineering for volatility topics.
+ Operates on the full DataFrame at once using 2D numpy arrays."""
</file context>
| # ============================================================================= | ||
| print("\n[1/4] Loading data...") | ||
| api_key = get_api_key( | ||
| api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key") |
There was a problem hiding this comment.
P2: Notebook hardcodes a non-canonical API key file path (../../.allora_api_key) that resolves to notebooks/.allora_api_key instead of the repository root. This diverges from the canonical worker_runtime._load_api_key fallback chain and short-circuits get_api_key's well-known search paths, preventing fallback to the repo-root key.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_85_eth_4h_vol/model_grid_retrain.py, line 75:
<comment>Notebook hardcodes a non-canonical API key file path (`../../.allora_api_key`) that resolves to `notebooks/.allora_api_key` instead of the repository root. This diverges from the canonical `worker_runtime._load_api_key` fallback chain and short-circuits `get_api_key`'s well-known search paths, preventing fallback to the repo-root key.</comment>
<file context>
@@ -0,0 +1,427 @@
+# =============================================================================
+print("\n[1/4] Loading data...")
+api_key = get_api_key(
+ api_key_file=os.path.join(os.path.dirname(__file__), "..", "..", ".allora_api_key")
+)
+
</file context>
| MAX_DEPTHS = [5, 7] | ||
| N_ESTIMATORS_LIST = [25, 50, 100, 150] | ||
|
|
||
| TOP_K_PER_GROUP = 1 # best model per feature group → 6 diverse models |
There was a problem hiding this comment.
P3: Unused configuration constant TOP_K_PER_GROUP introduces dead code and misleading configurability
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At notebooks/testnet/topic_85_eth_4h_vol/model_importance_groups.py, line 45:
<comment>Unused configuration constant `TOP_K_PER_GROUP` introduces dead code and misleading configurability</comment>
<file context>
@@ -0,0 +1,414 @@
+MAX_DEPTHS = [5, 7]
+N_ESTIMATORS_LIST = [25, 50, 100, 150]
+
+TOP_K_PER_GROUP = 1 # best model per feature group → 6 diverse models
+
+print("=" * 70)
</file context>
Added volatility prediction topics for various cryptocurrency pairs to the README.
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="README.md">
<violation number="1" location="README.md:193">
P3: The newly added subsection heading misspells "Volatility" as "Voltility". Since this appears in the public README topic-reference section, the typo reduces documentation polish and makes it harder for users to find volatility-topic guidance via search. Consider correcting it to "Volatility Topics (may require whitelist)".</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
| | **69** | BTC/USD - 1 Day Price Prediction | Price | Playground — example walkthroughs use this | | ||
| | **77** | BTC/USD - 5 Min Price Prediction | Price | Playground Fast | | ||
|
|
||
| Voltility Topics (may require whitelist) |
There was a problem hiding this comment.
P3: The newly added subsection heading misspells "Volatility" as "Voltility". Since this appears in the public README topic-reference section, the typo reduces documentation polish and makes it harder for users to find volatility-topic guidance via search. Consider correcting it to "Volatility Topics (may require whitelist)".
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At README.md, line 193:
<comment>The newly added subsection heading misspells "Volatility" as "Voltility". Since this appears in the public README topic-reference section, the typo reduces documentation polish and makes it harder for users to find volatility-topic guidance via search. Consider correcting it to "Volatility Topics (may require whitelist)".</comment>
<file context>
@@ -189,12 +189,15 @@ Playground topics (testnet only) are the recommended starting point — no white
| **69** | BTC/USD - 1 Day Price Prediction | Price | Playground — example walkthroughs use this |
| **77** | BTC/USD - 5 Min Price Prediction | Price | Playground Fast |
+
+Voltility Topics (may require whitelist)
+
+| Testnet ID | Name | Target type | Notes |
</file context>
Suggested change
| Voltility Topics (may require whitelist) | |
| +Volatility Topics (may require whitelist) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Core library changes:
Tests:
Notebooks (example/illustration code):
Docs:
Summary by cubic
Adds volatility targets (with √horizon scaling) and a CZAR directional loss for
LightGBM, enabling sign-aware training and correct outputs for realised volatility topics. Consolidates volatility topic scripts into a single grid-retrain per topic, vectorizes vol features (~900× faster), adds an ETH 4h vol example (Topic 85), and updates docs (README adds volatility topics and SOL log-return testnet ID).New Features
AlloraMLWorkflow:target_type("log_return" | "volatility"); volatility = std of 1‑min log returns over horizon, scaled by √H; default remains "log_return".czar_loss.py: CZAR loss with gradient/hessian andmake_czar_objective; exported viaallora_forge_builder_kit/__init__.py.model_grid_retrain.pyper topic (79–82) with vectorized features; Topic 85 (ETH 4h vol) grid retrain and importance-group scripts;notebooks/dashboard.sh;.gitignoreignores**/runs/.Bug Fixes
target_barsto match reputer ground truth; tests added.Written for commit d8513f6. Summary will update on new commits.