Describe the bug
On python-hcl2 8.x, hcl2.loads()/hcl2.load() no longer returns heredoc content as a plain string with real newlines by default — heredocs are preserved verbatim including the <<-EOT/EOT markers (related to the general quoting regression tracked in #289). The documented migration workaround for heredocs, setting SerializationOptions(preserve_heredocs=False), does strip the markers, but it also converts every real newline inside the heredoc body into the two-character literal sequence \n (backslash + n) instead of leaving an actual newline character. This makes it impossible to recover v7's heredoc semantics (a plain string with real line breaks) via any combination of the documented SerializationOptions.
We hit this in production: a heredoc-defined multi-line PGP private key block, parsed with the documented v8 compat options, came out as a single line with literal \n text instead of real line breaks — silently producing an unusable key.
Software:
- OS: Linux
- Python version: 3.13
- python-hcl2 version: 8.1.2 (also reproduced on 8.1.0 and 8.1.1)
Snippet of HCL2 code causing the unexpected behaviour:
audience_export_pgp_keys = {
private = <<-EOT
-----BEGIN PGP PRIVATE KEY BLOCK-----
line1
line2
-----END PGP PRIVATE KEY BLOCK-----
EOT
}
import hcl2
from hcl2 import SerializationOptions
hcl_text = '''audience_export_pgp_keys = {
private = <<-EOT
-----BEGIN PGP PRIVATE KEY BLOCK-----
line1
line2
-----END PGP PRIVATE KEY BLOCK-----
EOT
}
'''
opts = SerializationOptions(strip_string_quotes=True, preserve_heredocs=False)
result = hcl2.loads(hcl_text, serialization_options=opts)
value = result["audience_export_pgp_keys"]["private"]
print(repr(value))
print("real newlines:", value.count("\n"), "| literal backslash-n:", value.count("\\n"))
Expected behavior (matches python-hcl2==7.3.1's plain hcl2.loads(hcl_text), no options needed):
'-----BEGIN PGP PRIVATE KEY BLOCK-----\nline1\nline2\n-----END PGP PRIVATE KEY BLOCK-----'
real newlines: 3 | literal backslash-n: 0
Actual behavior (python-hcl2==8.1.2 with preserve_heredocs=False):
'-----BEGIN PGP PRIVATE KEY BLOCK-----\\nline1\\nline2\\n-----END PGP PRIVATE KEY BLOCK-----'
real newlines: 0 | literal backslash-n: 3
Impact
There is currently no combination of SerializationOptions on 8.x that reproduces v7's heredoc parsing behavior for multi-line content: preserve_heredocs=True (the default) keeps the <<-EOT/EOT markers embedded in the string, and preserve_heredocs=False strips them but corrupts every embedded newline instead. For any consumer extracting multi-line secrets/config out of a heredoc, both options currently produce unusable output, and the "V7_COMPAT" recipe in the v8 migration guide doesn't mention this gap since it only covers strip_string_quotes.
Related: #289 (same underlying default-change concern, for quoted strings rather than heredocs).
Describe the bug
On python-hcl2 8.x,
hcl2.loads()/hcl2.load()no longer returns heredoc content as a plain string with real newlines by default — heredocs are preserved verbatim including the<<-EOT/EOTmarkers (related to the general quoting regression tracked in #289). The documented migration workaround for heredocs, settingSerializationOptions(preserve_heredocs=False), does strip the markers, but it also converts every real newline inside the heredoc body into the two-character literal sequence\n(backslash +n) instead of leaving an actual newline character. This makes it impossible to recover v7's heredoc semantics (a plain string with real line breaks) via any combination of the documentedSerializationOptions.We hit this in production: a heredoc-defined multi-line PGP private key block, parsed with the documented v8 compat options, came out as a single line with literal
\ntext instead of real line breaks — silently producing an unusable key.Software:
Snippet of HCL2 code causing the unexpected behaviour:
Expected behavior (matches
python-hcl2==7.3.1's plainhcl2.loads(hcl_text), no options needed):Actual behavior (
python-hcl2==8.1.2withpreserve_heredocs=False):Impact
There is currently no combination of
SerializationOptionson 8.x that reproduces v7's heredoc parsing behavior for multi-line content:preserve_heredocs=True(the default) keeps the<<-EOT/EOTmarkers embedded in the string, andpreserve_heredocs=Falsestrips them but corrupts every embedded newline instead. For any consumer extracting multi-line secrets/config out of a heredoc, both options currently produce unusable output, and the "V7_COMPAT" recipe in the v8 migration guide doesn't mention this gap since it only coversstrip_string_quotes.Related: #289 (same underlying default-change concern, for quoted strings rather than heredocs).