-
Notifications
You must be signed in to change notification settings - Fork 0
feat(recovery): five-repeat AEFA true-parameter RMSE protocol #79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seonghobae
wants to merge
26
commits into
develop
Choose a base branch
from
cursor/aefa-parameter-recovery-rmse-bab5
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
d55a2b5
ci: rebuild stringfish from source on mac/windows (RcppParallel oneTB…
seonghobae 31d8d8b
ci: rebuild stringfish on all platforms (Linux hit too)
seonghobae e7b7f3d
ci: rebuild stringfish before install in test-fast
seonghobae 21755e0
ci: rebuild stringfish before install in test-suite
seonghobae 0446de9
ci: rebuild all RcppParallel-linked packages (qs2 hit the same oneTBB…
seonghobae 648e948
ci: rebuild all RcppParallel-linked packages (qs2 hit the same oneTBB…
seonghobae 34ef00f
ci: rebuild all RcppParallel-linked packages (qs2 hit the same oneTBB…
seonghobae b944368
ci: source-build stringfish before dep setup (SimDesign source builds…
seonghobae 46012b7
ci: source-build stringfish before dep setup (SimDesign source builds…
seonghobae 4acfcf0
ci: source-build stringfish before dep setup (SimDesign source builds…
seonghobae d75e72c
ci: real source rebuilds via CRAN cloud (RSPM serves linux binaries f…
seonghobae 28e8442
ci: real source rebuilds via CRAN cloud (RSPM serves linux binaries f…
seonghobae fbba747
ci: real source rebuilds via CRAN cloud (RSPM serves linux binaries f…
seonghobae 810dfa0
Merge branch 'develop' into ci/rebuild-stringfish-abi
opencode-agent[bot] 46c76d0
chore(ci): consolidate r-lib actions v2.12.1
seonghobae bbda0c1
test(ci): reject substituted r-lib actions
seonghobae 6abb556
fix(ci): refresh macOS dependency ABI cache
seonghobae 5a2a541
test(ci): require active cache version configuration
seonghobae ce826e9
ci: rebuild macOS TBB dependents from source
seonghobae b272506
merge(ci): preserve proven oneTBB rebuild sequence
seonghobae 5092a70
test(ci): pin every oneTBB source rebuild argument
seonghobae 3792f42
feat(recovery): add five-repeat AEFA RMSE protocol
cursoragent 4cd21a8
fix(recovery): treat default rownames and NA Hessian as failures
cursoragent bd7a530
test(recovery): use FIIFM-sized N for live 2PL RMSE
cursoragent a18f1ce
test(recovery): require SE=TRUE for the live 2PL RMSE fit
cursoragent 1c5d9f0
fix(recovery): require identical item sets and complete seeds
cursoragent File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| # Internal true-parameter recovery helpers for AEFA / IRT Monte Carlo evidence. | ||
| # These functions are not exported. Buyer-facing recovery is the test protocol | ||
| # in tests/testthat/test-aefa-parameter-recovery.R and the provenance note in | ||
| # docs/traceability/aefa-parameter-recovery.md. | ||
|
|
||
| .parameterRecoveryRMSE <- function(estimated, truth) { | ||
| estimated <- as.numeric(estimated) | ||
| truth <- as.numeric(truth) | ||
| if (length(estimated) != length(truth)) { | ||
| stop("Estimated and true vectors must have the same length.", call. = FALSE) | ||
| } | ||
| if (!length(estimated)) { | ||
| stop("Cannot compute RMSE from empty parameter vectors.", call. = FALSE) | ||
| } | ||
| if (any(!is.finite(estimated)) || any(!is.finite(truth))) { | ||
| stop("RMSE requires finite estimated and true values.", call. = FALSE) | ||
| } | ||
| sqrt(mean((estimated - truth)^2)) | ||
| } | ||
|
|
||
| # R data.frames never keep rownames as NULL: `rownames(x) <- NULL` resets them | ||
| # to the sequential defaults "1", "2", .... Treat those as unnamed so callers | ||
| # cannot silently align on positional indices. | ||
| .irtItemNames <- function(x) { | ||
| rn <- rownames(x) | ||
| if (is.null(rn) || identical(rn, as.character(seq_len(nrow(x))))) { | ||
| return(NULL) | ||
| } | ||
| rn | ||
| } | ||
|
|
||
| .alignIrtItemParameters <- function(estimated, truth, columns = c("a", "b")) { | ||
| if (!is.data.frame(estimated) && !is.matrix(estimated)) { | ||
| stop("estimated must be a matrix or data.frame of item parameters.", call. = FALSE) | ||
| } | ||
| if (!is.data.frame(truth) && !is.matrix(truth)) { | ||
| stop("truth must be a matrix or data.frame of item parameters.", call. = FALSE) | ||
| } | ||
| estimated <- as.data.frame(estimated, stringsAsFactors = FALSE) | ||
| truth <- as.data.frame(truth, stringsAsFactors = FALSE) | ||
| est_names <- .irtItemNames(estimated) | ||
| true_names <- .irtItemNames(truth) | ||
| if (is.null(est_names) || is.null(true_names)) { | ||
| stop( | ||
| "Estimated and true parameter tables must have item names as row names.", | ||
| call. = FALSE | ||
| ) | ||
| } | ||
| missing_estimated <- setdiff(columns, colnames(estimated)) | ||
| missing_truth <- setdiff(columns, colnames(truth)) | ||
| if (length(missing_estimated) || length(missing_truth)) { | ||
| stop( | ||
| "Missing recovery columns: ", | ||
| paste(unique(c(missing_estimated, missing_truth)), collapse = ", "), | ||
| call. = FALSE | ||
| ) | ||
| } | ||
| missing_estimated_items <- setdiff(true_names, est_names) | ||
| missing_truth_items <- setdiff(est_names, true_names) | ||
| if (length(missing_estimated_items) || length(missing_truth_items)) { | ||
| stop( | ||
| "Estimated and true parameter tables must contain the same item names.", | ||
| call. = FALSE | ||
| ) | ||
| } | ||
| shared <- est_names | ||
| list( | ||
| estimated = estimated[shared, columns, drop = FALSE], | ||
| truth = truth[shared, columns, drop = FALSE], | ||
| items = shared | ||
| ) | ||
| } | ||
|
|
||
| .summariseRecoveryRepeats <- function(rmse_by_repeat) { | ||
| if (!is.data.frame(rmse_by_repeat)) { | ||
| stop("rmse_by_repeat must be a data.frame.", call. = FALSE) | ||
| } | ||
| required <- c("seed", "parameter", "rmse") | ||
| missing <- setdiff(required, names(rmse_by_repeat)) | ||
| if (length(missing)) { | ||
| stop( | ||
| "rmse_by_repeat must contain columns: ", | ||
| paste(required, collapse = ", "), | ||
| call. = FALSE | ||
| ) | ||
| } | ||
| seeds <- unique(rmse_by_repeat$seed) | ||
| if (length(seeds) != 5L) { | ||
| stop("Recovery protocol requires exactly 5 repeats.", call. = FALSE) | ||
| } | ||
| if (anyNA(rmse_by_repeat$seed) || | ||
| anyNA(rmse_by_repeat$parameter) || | ||
| anyNA(rmse_by_repeat$rmse)) { | ||
| stop("Recovery repeats cannot contain missing values.", call. = FALSE) | ||
| } | ||
| parameters <- unique(as.character(rmse_by_repeat$parameter)) | ||
| repeat_counts <- table( | ||
| as.character(rmse_by_repeat$parameter), | ||
| rmse_by_repeat$seed | ||
| ) | ||
| if (!length(repeat_counts) || any(repeat_counts != 1L)) { | ||
| stop( | ||
| "Each parameter must have exactly one RMSE value for each recovery seed.", | ||
| call. = FALSE | ||
| ) | ||
| } | ||
| summary_rows <- lapply(parameters, function(parameter_name) { | ||
| values <- rmse_by_repeat$rmse[as.character(rmse_by_repeat$parameter) == parameter_name] | ||
| data.frame( | ||
| parameter = parameter_name, | ||
| n_repeats = length(values), | ||
| mean_rmse = mean(values), | ||
| sd_rmse = stats::sd(values), | ||
| stringsAsFactors = FALSE | ||
| ) | ||
| }) | ||
| list( | ||
| n_repeats = 5L, | ||
| seeds = seeds, | ||
| per_run = rmse_by_repeat[order(rmse_by_repeat$seed, rmse_by_repeat$parameter), ], | ||
| summary = do.call(rbind, summary_rows) | ||
| ) | ||
| } | ||
|
|
||
| .extractAefaIrtItems <- function(fit) { | ||
| if (inherits(fit, "aefa") || (is.list(fit) && !is.null(fit$estModelTrials))) { | ||
| trials <- fit$estModelTrials | ||
| if (!length(trials)) { | ||
| stop("aefa history has no estimated model trials.", call. = FALSE) | ||
| } | ||
| fit <- trials[[length(trials)]] | ||
| } | ||
| if (!methods::is(fit, "SingleGroupClass") && !methods::is(fit, "MixedClass")) { | ||
| stop("Recovery extraction requires an aefa history or a mirt model.", call. = FALSE) | ||
| } | ||
| items <- mirt::coef(fit, IRTpars = TRUE, simplify = TRUE)$items | ||
| if (is.null(items)) { | ||
| stop("Could not extract IRT item parameters.", call. = FALSE) | ||
| } | ||
| as.data.frame(items, stringsAsFactors = FALSE) | ||
| } | ||
|
|
||
| .recoveryCoverageExclusions <- function() { | ||
| data.frame( | ||
| surface = c( | ||
| "unidimensional 2PL via .mirt", | ||
| "AEFA greedy search on unidimensional 2PL", | ||
| "mixedmirt multilevel / random effects", | ||
| "multiple-membership crossed random effects", | ||
| "time-flow / longitudinal membership" | ||
| ), | ||
| status = c( | ||
| "covered", | ||
| "covered when RUN_FULL_AEFA_TESTS=1", | ||
| "excluded", | ||
| "excluded", | ||
| "excluded" | ||
| ), | ||
| reason = c( | ||
| "Known-true 2PL simulation with IRT a/b RMSE.", | ||
| "Known-true 2PL simulation through aefa() with five seeds.", | ||
| "Engine exposes .mixedmirt, but no true-parameter RMSE protocol yet.", | ||
| "random = ~1|G formulas exist, but no recovery design is registered.", | ||
| "No time-indexed membership design is implemented in kaefa-core." | ||
| ), | ||
| stringsAsFactors = FALSE | ||
| ) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
각
parameter-seed조합을 정확히 한 번씩 검증하십시오.Line 83은 전체 고유
seed수만 검사합니다. 예를 들어a에 한 seed가 중복되고b에서 같은 seed가 누락되면 검사를 통과합니다. 이 경우mean_rmse는 5회 반복 평균이 아닙니다.각 parameter에 모든 5개 seed가 정확히 한 번 있어야 합니다. 중복, 누락,
NARMSE를 오류로 처리하십시오.수정 예시
seeds <- unique(rmse_by_repeat$seed) if (length(seeds) != 5L) { stop("Recovery protocol requires exactly 5 repeats.", call. = FALSE) } + if (anyNA(rmse_by_repeat$seed) || + anyNA(rmse_by_repeat$parameter) || + anyNA(rmse_by_repeat$rmse)) { + stop("Recovery repeats cannot contain missing values.", call. = FALSE) + } parameters <- unique(as.character(rmse_by_repeat$parameter)) + repeat_counts <- table(as.character(rmse_by_repeat$parameter), rmse_by_repeat$seed) + if (any(repeat_counts != 1L)) { + stop( + "Each parameter must have exactly one RMSE value for each recovery seed.", + call. = FALSE + ) + }📝 Committable suggestion
🤖 Prompt for AI Agents