Skip to content

fix(config): read YAML config files as UTF-8#124

Open
nankingjing wants to merge 1 commit into
microsoft:mainfrom
nankingjing:fix-config-utf8-encoding
Open

fix(config): read YAML config files as UTF-8#124
nankingjing wants to merge 1 commit into
microsoft:mainfrom
nankingjing:fix-config-utf8-encoding

Conversation

@nankingjing

Copy link
Copy Markdown

Problem

skillopt/config.py::_load_yaml opens YAML config files without specifying an
encoding:

with open(abs_path) as f:
    cfg = yaml.safe_load(f) or {}

open() therefore decodes using the platform's locale-dependent default
encoding (locale.getpreferredencoding()), not UTF-8. The shipped configs
contain UTF-8 non-ASCII characters — for example the very first line of
configs/_base_/default.yaml:

# SkillOpt default configuration — base for all environments.

The (U+2014 em-dash), along with and box-drawing used elsewhere in
the configs, are multi-byte in UTF-8. On any host whose default encoding is not
UTF-8 (e.g. Windows with a cp936/cp1252 locale), load_config() raises
UnicodeDecodeError before training can start, breaking both entry points
(scripts/train.py and scripts/eval_only.py).

Reproduction (locale where preferred encoding is not UTF-8):

UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 35: illegal multibyte sequence

Fix

Open the YAML file as UTF-8. YAML is UTF-8 by spec, and the rest of the
codebase already opens text files with encoding="utf-8" (e.g.
skillopt/prompts/__init__.py, skillopt/datasets/base.py), so this just
restores parity.

-    with open(abs_path) as f:
+    with open(abs_path, encoding="utf-8") as f:

Verification

  • python -m py_compile skillopt/config.py passes.
  • Before the change, load_config("configs/_base_/default.yaml") raised
    UnicodeDecodeError under a cp936 locale; after the change,
    load_config(...) + flatten_config(...) succeed for the base config and
    every env config that inherits it (searchqa, spreadsheetbench, alfworld, and
    features/soft_gate.yaml).
  • On a UTF-8 locale behavior is unchanged.

_load_yaml opened config files with the platform default (locale)
encoding instead of UTF-8. The shipped configs (e.g.
configs/_base_/default.yaml) contain UTF-8 non-ASCII characters
(em-dash, arrows, box-drawing), so load_config() raises
UnicodeDecodeError on any non-UTF-8 locale (e.g. Windows cp936/cp1252),
breaking scripts/train.py and scripts/eval_only.py before startup.

YAML is UTF-8 by spec, and the rest of the codebase already opens text
files with encoding="utf-8". Pass encoding="utf-8" here for parity.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant