Add core symbolic gradient support to Pyomo.DoE by snarasi2 · Pull Request #3928 · Pyomo/pyomo

snarasi2 · 2026-05-04T19:14:44Z

Fixes # .

This PR ports the core symbolic-gradient functionality from the historical pyomo-doe-symbolic work into the current pyomo.contrib.doe implementation.

Summary/Motivation:

Rather than merging the old branch directly, this change transplants the symbolic DoE pieces onto current main so that symbolic sensitivities work with the newer DoE implementation already present in Pyomo, including the current objective and GreyBox-oriented code paths.

This PR ports the core symbolic-gradient functionality from adowling2#7 onto current Pyomo main and adapts it to the current pyomo.contrib.doe implementation.

This contribution was prepared with coding assistance from OpenAI Codex. All design decisions, validation, testing, and quality-assurance responsibility remain with Shilpa Narasimhan.

Changes proposed in this PR:

Add GradientMethod support to DesignOfExperiments
Preserve the existing finite-difference workflow through the new gradient-method interface
Add a symbolic / PyNumero gradient path for DoE
Add an analytic FIM computation path for non-optimization FIM evaluation
Export ExperimentGradients from pyomo.contrib.doe
Refactor ExperimentGradients so symbolic and automatic differentiation are set up together
Add supporting symbolic-gradient utilities for labeled experiment models
Update DoE model construction so symbolic sensitivities can be used in the Jacobian-based DoE machinery
Add a guard preventing run_doe() from being called with GradientMethod.kaug
Add the polynomial example and polynomial-focused regression coverage
Add broader symbolic-versus-automatic gradient consistency tests
Add factorial-result dataframe tests
Add reactor regression tests against expected solutions
Clarify GreyBox cyipopt / MA57-HSL test behavior where relevant

Validation performed locally:

Black: python -m black -S -C --check --diff pyomo/contrib/doe → passed
Typos: typos --config .github/workflows/typos.toml pyomo/contrib/doe DOE_SYMBOLIC_PR_NOTES.md → passed
Focused DoE/GreyBox validation: 134 passed, 0 failed, 0 skipped
GreyBox cyipopt tests were run with MA57/HSL available in the local environment
The existing K_AUG-dependent DoE test path was validated locally after configuring the required runtime library path

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Add schema, filtering, and error-path tests for factorial-result tables and DataFrame input handling. Validation: - 101 passed, 5 warnings, 10 subtests passed in 30.00s Notes: - plotting tests emit expected matplotlib warnings under the non-interactive Agg backend because draw_factorial_figure() calls plt.show().

Add a determinant-based reactor regression test, guard Codecov uploads when coverage.xml is absent, and align local formatting with Black expectations. Validation: - focused reactor regression subset: 2 passed, 28 deselected in 10.08s - full DoE suite: 102 passed, 5 warnings, 10 subtests passed in 30.29s Notes: - plotting-related tests continue to emit expected matplotlib warnings under the non-interactive Agg backend because draw_factorial_figure() calls plt.show().

Add coverage for draw_factorial_figure guard branches covering more-than-two sensitivity variables and missing fixed design variables. Validation: - focused plotting-guard subset: 2 passed, 37 deselected in 1.92s - full DoE suite: 104 passed, 5 warnings, 10 subtests passed in 31.47s Notes: - plotting-related tests continue to emit expected matplotlib warnings under the non-interactive Agg backend because draw_factorial_figure() calls plt.show().

snarasi2 · 2026-05-04T19:15:30Z

@adowling2 @slilonfe5 @smondal13 @sscini This is the new symbolic PR (created from the unpolluted branch)

snarasi2 · 2026-05-04T19:19:08Z

@blnicho — this PR (#3928) is the clean replacement for #3898. The original PR included unintended extra changes; this one contains only the intended symbolic DoE work and has been fully validated.

Could you please switch review/merge to this PR and we can close #3898?

mrmundt · 2026-05-04T19:22:01Z

@blnicho — this PR (#3928) is the clean replacement for #3898. The original PR included unintended extra changes; this one contains only the intended symbolic DoE work and has been fully validated.

Could you please switch review/merge to this PR and we can close #3898?

@snarasi2 - Done. #3898 is closed in favor of this one.

snarasi2 · 2026-05-11T17:24:37Z

+        experiment_grad = None
+        if self._gradient_method == GradientMethod.pynumero:
+            experiment_grad = ExperimentGradients(
+                model.scenario_blocks[0], automatic=True, symbolic=True


@adowling2 @smondal13 , per discussion with Shuvo on 05/11: Shuvo and I decided that the cleanest path forward to address potential conflicts with Shuvo's multi-experiment PR #3866 is to leave symbolic PR as is for now. When PR3866 is merged, we will address the conflict with fd_scenario_blocks and fd_scenarios within multi-experiment vs scenatio_blocks and scenarios within the current Pyomo main and the symbolic PR.

Alternative would be to merge this PR with multi-experiment as of now, but it will create a dependency and we are not sure we should proceed this way.

Shilpa to double-check and address: If main does not use model.scenarios currently, @snarasi2 will modify all occurrences of model.scenarios to to model.fd_scenarios to facilitate multi-experiment merge. Additionally, this will help distinguish scenario building for finite difference methods for parameter perturbations vs parameter scenarios for uncertainty-aware design.

@smondal13 @adowling2 I checked upstream/main and confirmed that literal model.scenarios already exists in pyomo/contrib/doe/doe.py and pyomo/contrib/doe/tests/test_doe_build.py. Since the proposed rename was conditional on main not currently using model.scenarios, I did not rename model.scenarios to model.fd_scenarios in this PR. I’m leaving this deferred to the multi-experiment merge as discussed on 05/11.

mrmundt · 2026-05-20T15:05:50Z

+    # focused on replacing active general-purpose coverage with
+    # Rooney-Biegler or polynomial examples rather than rewriting inactive,
+    # branch-specific tests.
+    def DISABLE_test_rooney_biegler_obj_cholesky_solve_bad_prior(self):


Maybe this should be deleted? It's been disabled for a while, and it'll still live in the commit history.

Removed the disabled Rooney-Biegler Cholesky/bad-prior test based on Miranda’s review comment and alignment from @slilonfe5 and @smondal13 on Slack. Commit: 5ad99f0

mrmundt · 2026-05-20T15:52:34Z

+            # The GreyBox path here uses cyipopt rather than the standalone
+            # IPOPT executable, and the local failure mode we diagnosed was a
+            # missing MA57/HSL runtime on that cyipopt solve path.
+            cyipopt_skip_reason = (


This is a nitpick. All of the skip reasons are variations of the same thing (MA57/HSL issues). Do you really need all three different messages? Or can you just standardize to the one that's on line 315?

Thanks — standardized the MA57/HSL-related skip reasons to use the same message as line 315 in ab2f729.

mrmundt · 2026-05-20T15:53:37Z



+@unittest.skipIf(not ipopt_available, "The 'ipopt' command is not available")
+@unittest.skipIf(not numpy_available, "Numpy is not available")


Aren't these numpy_available checks redundant (because of lines 29-30)?

Thanks — removed the redundant NumPy skip decorators in test_utils.py because the file-level SkipTest already handles missing NumPy in f1c9a8f.

mrmundt · 2026-05-20T15:54:16Z

+        self.assertEqual(jacobian.shape, expected.shape)
+        self.assertTrue(np.allclose(jacobian, expected, atol=1e-7, rtol=1e-7))
+
+    @unittest.skipIf(not scipy_available, "scipy is not available")


Similar comment - isn't this scipy_available check redundant?

Thanks — removed the redundant SciPy skip decorator in test_utils.py because the file-level SkipTest already handles missing SciPy in dcbfee6.

@mrmundt, @adowling2 I also noticed the same pattern in test_greybox.py: the file has a file-level SkipTest for NumPy/SciPy, but the TestFIMExternalGreyBox class still has NumPy/SciPy skipIf decorators. Would you prefer that I keep this PR scoped to the test_utils.py comments, or should I also remove the redundant decorators within test_greybox.py and similar files for consistency?

mrmundt · 2026-05-20T15:56:45Z

                      default: None --> don't save

        """
+        if self._gradient_method == GradientMethod.kaug:


Maybe I am misunderstanding - kaug is listed as an acceptable option in the docstring, but it's not actually?

Thanks — I checked the code path more closely. kaug is supported for FIM computation through compute_FIM(), but run_doe() explicitly rejects GradientMethod.kaug because it is not supported for DoE optimization. I clarified that scope in the gradient_method docstring in 1096e52.

Shilpa Narasimhan added 30 commits April 1, 2026 14:20

Add symbolic gradient support to Pyomo.DoE

50cbc4a

Add symbolic DoE regression tests

85116d5

Allow writeable in typos config

12e5e95

Port polynomial and reactor symbolic tests

b35253a

Document symbolic gradient methods in DoE

431b28e

Refactor ExperimentGradients setup

f55a294

Add polynomial DoE regression coverage

ca66ac6

Add broader gradient consistency tests

8d8b8bd

Clarify cyipopt HSL skip reason

f28d86d

Reuse polynomial example in DoE tests

80552f2

Drop unused data arg from polynomial example

d708d55

Clarify symbolic sensitivity math comments

f53fe17

Remove no-op polynomial finalization

11d475e

Simplify polynomial test imports

e704f68

Remove redundant polynomial FIM smoke test

0d0a14f

Strengthen polynomial DoE regression test

401c03a

Use symbolic path for dataframe tests

0dd08eb

Align symbolic DoE test coverage

65571da

Guard DoE utils tests on ipopt

9eb4959

Strengthen symbolic Jacobian tests

702dc89

Remove out-of-scope FIM metric test

d652cea

Tighten Jacobian regression assertions

b5ea7f1

Drop workflow changes from symbolic PR

9a78180

Checkpoint reactor test conversions to RB/polynomial

c593f0b

Checkpoint remaining reactor test cleanup

36d5694

Replace remaining reactor DoE tests

dae72f4

Clarify GreyBox MA57 skip reason

0545dcf

Shilpa Narasimhan and others added 7 commits April 10, 2026 22:13

Add symbolic DoE PR notes

270bf8c

Refactor Jacobian rule setup

33d0b98

Tighten FD scenario validation

4209df5

Format DoE module with black

653e3be

Merge branch 'Pyomo:main' into clean_symbolic

2b8ba66

Cleanup: remove trailing whitespace in DOE tests and utils

734fd58

Format DOE files with Black

06dda52

mrmundt mentioned this pull request May 4, 2026

Add core symbolic gradient support to Pyomo.DoE #3898

Closed

blnicho added this to ParmEst & Pyomo.DoE Development May 4, 2026

github-project-automation Bot moved this to Todo in ParmEst & Pyomo.DoE Development May 4, 2026

blnicho added the pyomo.DoE label May 4, 2026

Guard DOE test requiring pandas (optional dependency)

67f23c8

sscini moved this from Todo to Ready for design review in ParmEst & Pyomo.DoE Development May 6, 2026

snarasi2 commented May 11, 2026

View reviewed changes

mrmundt reviewed May 20, 2026

View reviewed changes

snarasi2 added 8 commits May 21, 2026 16:41

Remove disabled Rooney-Biegler Cholesky test

5ad99f0

Standardize GreyBox MA57 skip reason

ab2f729

Remove redundant NumPy skips in DoE utils tests

f1c9a8f

Remove redundant SciPy skip in DoE utils tests

dcbfee6

Clarify k_aug gradient method scope

1096e52

Merge remote-tracking branch 'upstream/main' into clean_symbolic

1e46960

Defer DoE optional dependency imports

cbb1582

Apply black formatting to DoE imports

53c80be



		@unittest.skipIf(not ipopt_available, "The 'ipopt' command is not available")
		@unittest.skipIf(not numpy_available, "Numpy is not available")

Conversation

snarasi2 commented May 4, 2026

Fixes # .

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

snarasi2 commented May 4, 2026

Uh oh!

snarasi2 commented May 4, 2026

Uh oh!

mrmundt commented May 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snarasi2 May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

snarasi2 May 21, 2026 •

edited

Loading