Skip to content

preserve_heredocs=False replaces real newlines with literal backslash-n instead of stripping only the markers #303

Description

@pazderak

Describe the bug

On python-hcl2 8.x, hcl2.loads()/hcl2.load() no longer returns heredoc content as a plain string with real newlines by default — heredocs are preserved verbatim including the <<-EOT/EOT markers (related to the general quoting regression tracked in #289). The documented migration workaround for heredocs, setting SerializationOptions(preserve_heredocs=False), does strip the markers, but it also converts every real newline inside the heredoc body into the two-character literal sequence \n (backslash + n) instead of leaving an actual newline character. This makes it impossible to recover v7's heredoc semantics (a plain string with real line breaks) via any combination of the documented SerializationOptions.

We hit this in production: a heredoc-defined multi-line PGP private key block, parsed with the documented v8 compat options, came out as a single line with literal \n text instead of real line breaks — silently producing an unusable key.

Software:

  • OS: Linux
  • Python version: 3.13
  • python-hcl2 version: 8.1.2 (also reproduced on 8.1.0 and 8.1.1)

Snippet of HCL2 code causing the unexpected behaviour:

audience_export_pgp_keys = {
  private = <<-EOT
  -----BEGIN PGP PRIVATE KEY BLOCK-----
  line1
  line2
  -----END PGP PRIVATE KEY BLOCK-----
  EOT
}
import hcl2
from hcl2 import SerializationOptions

hcl_text = '''audience_export_pgp_keys = {
  private = <<-EOT
  -----BEGIN PGP PRIVATE KEY BLOCK-----
  line1
  line2
  -----END PGP PRIVATE KEY BLOCK-----
  EOT
}
'''

opts = SerializationOptions(strip_string_quotes=True, preserve_heredocs=False)
result = hcl2.loads(hcl_text, serialization_options=opts)
value = result["audience_export_pgp_keys"]["private"]
print(repr(value))
print("real newlines:", value.count("\n"), "| literal backslash-n:", value.count("\\n"))

Expected behavior (matches python-hcl2==7.3.1's plain hcl2.loads(hcl_text), no options needed):

'-----BEGIN PGP PRIVATE KEY BLOCK-----\nline1\nline2\n-----END PGP PRIVATE KEY BLOCK-----'
real newlines: 3 | literal backslash-n: 0

Actual behavior (python-hcl2==8.1.2 with preserve_heredocs=False):

'-----BEGIN PGP PRIVATE KEY BLOCK-----\\nline1\\nline2\\n-----END PGP PRIVATE KEY BLOCK-----'
real newlines: 0 | literal backslash-n: 3

Impact

There is currently no combination of SerializationOptions on 8.x that reproduces v7's heredoc parsing behavior for multi-line content: preserve_heredocs=True (the default) keeps the <<-EOT/EOT markers embedded in the string, and preserve_heredocs=False strips them but corrupts every embedded newline instead. For any consumer extracting multi-line secrets/config out of a heredoc, both options currently produce unusable output, and the "V7_COMPAT" recipe in the v8 migration guide doesn't mention this gap since it only covers strip_string_quotes.

Related: #289 (same underlying default-change concern, for quoted strings rather than heredocs).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions