[Discussion]: Handling operations that use/modify RNG

[averissimo]

What is your question?

Per discussion with @llrs-roche, here are some topics for the contributors to discuss and research.

Problem

Random/stoichastic operations are not being well handled in within/teal.code::eval_code, in particular when code parse is being used.

In the section below you can observe that q |> get_code(names = "IRIS") is not fully reproducible as the random operations before were not included, nor was the RNG state updated before code execution

Reproducible code

This code highlights the 2 main problems:

1. set.seed() and other random operations are not kept in partial code (q |> get_code(names = "XXX"))
2. MTCARS/IRIS random operations are influenced by preceding code

library(teal.code)
q <- qenv() |> 
  within({
    IRIS <- datasets::iris
    MTCARS <- datasets::mtcars
    set.seed(1)
    .random <- runif(1)
    IRIS$new <- runif(nrow(IRIS))
    MTCARS$new <- runif(nrow(MTCARS))
  })

q |> get_code() |> cat()
#> iris <- datasets::iris
#> mtcars <- datasets::mtcars
#> set.seed(1)
#> .random <- runif(1)
#> iris$new <- runif(nrow(iris))
#> mtcars$new <- runif(nrow(mtcars))

q |> get_code(names = "IRIS") |> cat()
#> IRIS <- datasets::iris
#> IRIS$new <- runif(nrow(IRIS))

local({
  eval(str2expression(q |> get_code(names = "IRIS")))
  q$IRIS |> rlang::hash() |> cat("# hash qenv IRIS\n")
  IRIS |> rlang::hash() |> cat("# hash reproduced IRIS\n")
})
#> 9bba33c8ed5cd90cd13316f35f812f55 # hash qenv IRIS
#> c90a89c1814206bc19fdc32d04d1eefa # hash reproduced IRIS

q |> get_code(names = "MTCARS") |> cat()
#> MTCARS <- datasets::mtcars
#> MTCARS$new <- runif(nrow(MTCARS))

local({
  eval(str2expression(q |> get_code(names = "MTCARS")))
  q$mtcars |> rlang::hash() |> cat("# hash qenv MTCARS\n")
  mtcars |> rlang::hash() |> cat("# hash reproduced MTCARS\n")
})
#> 2c0a8a99dc147d5445c3b49d035665b2 # hash qenv MTCARS
#> 6755d143ff87b73a1196c186cec7e86a # hash reproduced MTCARS

Reference material

• ref1: [Bug]: Functions with side effects are not detected as dependencies #278
• ref2: https://www.jottr.org/2020/09/21/detect-when-the-random-number-generator-was-used/
  • Detect when random seed is modified
• ref3: https://blog.djnavarro.net/posts/2023-12-27_seedcatcher/

Possible set of solutions

• Use teal.code::eval_code() and # @linksto XXXXX to include random operations
  • Status quo
  • Cons: Requires character base code execution and requires manual maintenance
• Detect random seed at start and inform user
• Track random seed and incorporate it in code execution
• When using code parser, re-run partial code to sync modules with reproducible code (while keeping set.seed() expressions)
  • cons: slow down teal app even more and data is not consistent with initial data argument in teal.
• Always include set.seed() expression and hope it doesn't have a big impact

Code of Conduct

• I agree to follow this project's Code of Conduct.

Contribution Guidelines

• I agree to follow this project's Contribution Guidelines.

Security Policy

• I agree to follow this project's Security Policy.

[averissimo]

My latest idea to tackle this problem is to inject a random seed state before each code LINE execution, and remove it if random seed does not change (see ref2)

Key ideas

• Keep indirect state of random seed (replicating exact state would be too verbose)
• Track random seed and only keep injected code if operation has a random component

Workflow

Let's say if we would execute code:

q <- teal_data() |> within({
  ADSL <- teal.data::rADSL
  ADSL <- dplyr::select(ADSL, 1:10)
  ADSL$yada <- runif(nrow(ADSL))
})

Then teal.code would inject an indirect state of the current seed and execute in the background:

set.seed(876990) # 876990 is generated by teal.code
ADSL <- teal.data::rADSL
set.seed(873884) # 873884 is generated by teal.code
ADSL <- dplyr::select(ADSL, 1:10)
set.seed(513737) # 513737 is generated by teal.code
ADSL$yada <- runif(nrow(ADSL))

Discard from code set.seed() that are not needed, as internal random seed did not change

q |> get_code() |> cat()
#> ADSL <- teal.data::rADSL
#> ADSL <- dplyr::select(ADSL, 1:10)
#> set.seed(513737)
#> ADSL$yada <- runif(nrow(ADSL))

We would need to link every injected set.seed() with the following expression so that the injected line would be included.

Limitations

• Changes original code as injection of set.seed(<generated number by teal.code>) guarantees a different state
• Adds overhead to code execution

[llrs-roche]

Then teal.code would inject an indirect state of the current seed and execute in the background:

set.seed(876990) # 876990 is generated by teal.code
ADSL <- teal.data::rADSL
set.seed(873884) # 873884 is generated by teal.code
ADSL <- dplyr::select(ADSL, 1:10)
set.seed(513737) # 513737 is generated by teal.code
ADSL$yada <- runif(nrow(ADSL))

I don't don't think we need to change the random seed number generation each time. The seed is fine for parallel process and doesn't change the actual numbers only the index:

set.seed(41)
head(.Random.seed)
#> [1]      10403        624 1957627861 1533286034 1524464875 2141189936
runif(1)
#> [1] 0.2134905
head(.Random.seed)
#> [1]       10403           1   491320415  1131310133 -1634207796    93692706
runif(1)
#> [1] 0.9712283
head(.Random.seed)
#> [1]       10403           2   491320415  1131310133 -1634207796    93692706
runif(1)
#> [1] 0.578187
head(.Random.seed)
#> [1]       10403           3   491320415  1131310133 -1634207796    93692706
runif(1)
#> [1] 0.14208
head(.Random.seed)
#> [1]       10403           4   491320415  1131310133 -1634207796    93692706

^{Created on 2026-08-27 with reprex v2.1.1}

The .Random.seed is filled from the main R process and it is not "available" on the qenv, based on these tests:

library(teal.code)

set.seed(42)
qa <- within(qenv(), {
  a <- .Random.seed
})
qa |> get_code() |> cat()
#> Error:
#> ! object '.Random.seed' not found 
#>  when evaluating qenv code:
#> a <- .Random.seed
#> 
#> trace: 
#>  a <- .Random.seed
qa |> get_outputs()
#> Error:
#> ! unable to find an inherited method for function 'get_outputs' for signature 'object = "qenv.error"'

q <- within(qenv(), {
  for (i in seq_len(10)) {
    cat(runif(1), sep = "\n")
  }
})

q |> get_code() |> cat()
#> for (i in seq_len(10)) {
#>     cat(runif(1), sep = "\n")
#> }
q |> get_outputs()
#> [[1]]
#> [1] "0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n0.4577418\n0.7191123\n"

q <- within(qenv(), {
  for (i in seq_len(10)) {
    set.seed(42)
    cat(runif(1), sep = "\n")
  }
})

q |> get_code() |> cat()
#> for (i in seq_len(10)) {
#>     set.seed(42)
#>     cat(runif(1), sep = "\n")
#> }
q |> get_outputs()
#> [[1]]
#> [1] "0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n0.914806\n"

But uses the external one:

q |> get_code() |> cat()
#> for (i in seq_len(10)) {
#>     cat(runif(1), sep = "\n")
#> }
q |> get_outputs()
#> [[1]]
#> [1] "0.9346722\n0.2554288\n0.4622928\n0.9400145\n0.9782264\n0.1174874\n0.4749971\n0.5603327\n0.9040314\n0.1387102\n"

set.seed(42)
q <- within(qenv(), {
  for (i in seq_len(10)) {
    cat(runif(1), sep = "\n")
  }
})

q |> get_code() |> cat()
#> for (i in seq_len(10)) {
#>     cat(runif(1), sep = "\n")
#> }
q |> get_outputs()
#> [[1]]
#> [1] "0.914806\n0.9370754\n0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n"

But they can use the internal one:

set.seed(42)
q4 <- within(q, {
  for (i in seq_len(10)) {
    cat(runif(1), sep = "\n")
  }
})

q5a <- within(q4, {
  for (i in seq_len(10)) {
    cat(runif(1), sep = "\n")
  }
})
q5a |> get_outputs()
#> [[1]]
#> [1] "0.914806\n0.9370754\n0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n"
#> 
#> [[2]]
#> [1] "0.914806\n0.9370754\n0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n"
#> 
#> [[3]]
#> [1] "0.9346722\n0.2554288\n0.4622928\n0.9400145\n0.9782264\n0.1174874\n0.4749971\n0.5603327\n0.9040314\n0.1387102\n"

q5b <- within(q4, {
  set.seed(48)
  for (i in seq_len(10)) {
    cat(runif(1), sep = "\n")
  }
})
q5b |> get_outputs()
#> [[1]]
#> [1] "0.914806\n0.9370754\n0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n"
#> 
#> [[2]]
#> [1] "0.914806\n0.9370754\n0.2861395\n0.8304476\n0.6417455\n0.5190959\n0.7365883\n0.1346666\n0.6569923\n0.7050648\n"
#> 
#> [[3]]
#> [1] "0.5791662\n0.03382941\n0.002716767\n0.3773311\n0.243292\n0.11888\n0.9810253\n0.03831031\n0.7853011\n0.4635564\n"

^{Created on 2026-08-27 with reprex v2.1.1}

The good news is that adding seeds won't slow down the report as when we save it, it runs all the code chunks again IIRC.