Circular reference in OpenAPI spec causes a stack overflow

[umeshkumhar]

tfplugingen-openapi CLI version

tfplugingen-openapi module: v0.3.0

OpenAPI Spec File

https://gist.github.com/umeshkumhar/97c7888fbcf6f294bb54210e0faec918

Generator Config

provider:
  name: backupdr

resources:
  slp:
    create:
      path:  /slp
      method: POST
    read:
      path:  /slp
      method: GET

Expected Behavior

It should have generated provider spec json.

Actual Behavior

This just executed and later on get killed due to OOM (i guess). in some iteration it shows stack overflow error. The activity monitor states that it consumes 70+GB memory, please find the attached screenshot below. Seems like its failing during Marshalling the openapi file.

Additional Information

Please suggest some workaround this.. if we can still generate some provider spec json for at least /slt, /sla, /slp. to kickstart.

Code of Conduct

• I agree to follow this project's Code of Conduct

[austinvalle]

Hey there @umeshkumhar 👋🏻, thanks for the detailed bug report and sorry you're running into trouble here.

I took a quick look at your OpenAPI spec and I believe the root issue is coming from some circular references in the spec. Currently our tooling doesn't throw an error if a circular reference exists, however, the current logic for building a schema assumes there are no circular references (which should be adjusted).

There are a couple things we'll need to consider for the tfplugingen-openapi tool here:

• Currently the only circular references detected are "infinite loops", where the circular reference property is marked as required. (which there are not any in your OpenAPI spec, otherwise you would have gotten a warning message)
• If we detect any circular reference, we'll need to determine if tfplugingen-openapi can actually handle them. Perhaps by attempting to skip said circular reference properties.

Locally I adjusted the logic to print out all the circular references found in the OpenAPI spec you provided:

time=2024-01-29T15:21:14.346-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SourceRest -> SlpRest -> DiskPoolRest -> DiskPoolCopiesRest -> CloudCredentialRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SourceRest -> SlpRest -> DiskPoolRest -> DiskPoolCopiesRest -> CloudCredentialRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SourceRest -> SlpRest -> DiskPoolRest -> DiskPoolCopiesRest -> CloudCredentialRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SourceRest -> HostRest -> SourceRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SourceRest -> HostRest -> HostRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest -> SltRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest -> SlaRest -> AdvancedOptionRest -> LogicalGroupRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest -> SlaRest -> AdvancedOptionRest -> LogicalGroupRest -> ExportVolumeInfoRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest -> SlaRest -> AdvancedOptionRest -> LogicalGroupRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="BackupNowRest -> PolicyRest -> SltRest -> AdvancedOptionRest -> SlaRest -> AdvancedOptionRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="MountRest -> SystemStateOptionRest -> SystemStateOptionRest"
time=2024-01-29T15:21:14.347-05:00 level=WARN msg="circular reference found in OpenAPI spec" circular_ref="SessionRest -> UserRest -> RightRest -> RightRest"

I will spend some time investigating what we can do here 👍🏻

[austinvalle]

Also worth noting here that Terraform resource/data source schemas don't support a notion of a recursive schema, so the final solution may end up behind some form of skip/controlled depth configuration.

At the very least, an explicit error should be returned in a case where we can't create a schema due to a circular reference.

[hegerdes]

Was also facing this issue when i tested out this tool using the keycloak openapi. Most likely also caused by a recursive schema.

It chews up memory binging the whole system down if not killed. A informative error message would be very helpful as long as recursive schemas are not supported. It might also be sensible to add some safety nets for unsupported/unexpected configs. Like setting a max execution time or detect if things go wrong and just crash.

[mrngm]

Encountered this as well (on v0.3.0 as "latest release" as well as bc39a95). One thing that stood out was the enormous amount of goroutines spawned when triggering SIGQUIT with Ctrl-. Each stack trace pointed to this method in orderedmap spawning a goroutine for iterating that map. After a few seconds running and triggering SIGQUIT some 20k+ goroutines were already part of the dump. That might clarify the rapid increase in memory.

As a workaround, use schema:ignores as documented here to omit the properties that would otherwise form the circular reference. Applying that to the generator configuration almost instantly returned with a generated provider spec.