Pr/multilingual by raghavm243512 · Pull Request #121 · ServiceNow/eva

raghavm243512 · 2026-05-18T21:35:49Z

initial multilingual version

Easily extendable to many language using the add_culture_data script. This will do translation, gender consistent naming, suggest names, extend data, etc. So if anyone wants to run a language not committed in EVA data, it is trivially easy to do so.
Readme section showing basic of adding a language.

This adds:
Multilingual data schema and content (initial utterances, system prompt, name aliases)
multilingual support in code
Prompt updates to support multi languages
Script to "add a language" with high degree of automation
WER metric normalization rules, dynamically set per language and creatable via LLM through adding script
Automatic .env.example adjustments (maintains config app accuracy)

Still TODO:

Currencies
Phone numbers (airline only problem)
Actually committing the translations (didn't want to burn credits until finalized)
Analysis
Testing a large variety of models to ensure they actually get the language code they expected (es-MX vs es, for example)

katstankiewicz

can you also add ensure_ascii=False to AuditLog save()

katstankiewicz · 2026-05-21T16:31:09Z

                ),
                audit_log=audit_log,
                api_key=params["api_key"],
+                base_url=params.get("url", ""),


Suggested change

base_url=params.get("url", ""),

we don't need a url for openai services

I added this because I was testing a side project (self hosted S2S agent). With any luck we will need this so we can leave or remove up to you

…behavioral_fidelity judge prompt

fanny-riols · 2026-05-29T14:27:36Z

+de: Hallo! Wie kann ich dir heute helfen?
+en: Hello! How can I help you today?
+es: ¡Hola! ¿En qué puedo ayudarte hoy?
+fr: Bonjour ! Comment puis-je vous aider aujourd'hui ?


We don't have fq?

easy to add, will do. The existing languages are just things I saved from testing because why waste them

fanny-riols · 2026-05-29T14:35:38Z

      - Using parameter values returned by a prior tool response (e.g., reusing an ID or record returned by an earlier lookup in a subsequent call)
      - Using reasonable defaults that are standard for the tool (e.g., a date format conversion)
-      - Standard domain mappings from user-stated information (e.g., "Chicago O'Hare" → "ORD", "Miami" → "MIA"; or other unambiguous geographic, enterprise, or industry-standard mappings present in the agent's domain) — unambiguous mappings are considered grounded
+      - Standard domain mappings from user-stated information (e.g., "Chicago O'Hare" → "ORD"; "sore throat and fever" → ICD-10 code "J06.9"; or other unambiguous geographic, enterprise, or industry-standard mappings present in the agent's domain) — unambiguous mappings are considered grounded


I'm not sure I understand that part, where is it coming from?

just an example of medical stuff. I know we are doing HR and not actual patient care but I figure general examples are always good to include even if not about each domain directly

fanny-riols · 2026-05-29T14:37:57Z

      - **0 (Corrupted)**: One or more corruption types occurred — the user's behavior caused the agent to be evaluated against an incorrect database state.
+
+      **Language:**
+      - Always provide analysis in English


Suggested change

- Always provide analysis in English

- Always provide analysis in English.

fanny-riols · 2026-05-29T14:44:45Z

              "turn_id": <int: the turn number from the Intended Turns>,
-              "transcript": <string: your transcription of the audio for this turn, use only the audio for this not the intended text>
-              "explanation": "<string: 1-3 sentence analysis of fidelity for this turn, citing specific intended vs actual mismatches, noting any regions skipped due to interruption flags>",
+              "transcript": <string: your transcription of the audio for this turn using the appropriate script for the language spoken, use only the audio for this not the intended text>


I'm confused on this addition, what do we mean by "appropriate script"?

should write 你好 not ni hao

fanny-riols · 2026-05-29T14:50:57Z

-      * Quantities: "twenty dollars" not "$20"
-      * Years: "twenty twenty-four" not "2024"
+    - Express numbers in linguistically/culturally appropriate spoken form for the conversation language:
+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"


Suggested change

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre"; not "1/15/2024")

fanny-riols · 2026-05-29T14:51:27Z

-      * Years: "twenty twenty-four" not "2024"
+    - Express numbers in linguistically/culturally appropriate spoken form for the conversation language:
+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
+      * Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"


Suggested change

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig"; not "3:30 PM")

fanny-riols · 2026-05-29T14:51:44Z

+    - Express numbers in linguistically/culturally appropriate spoken form for the conversation language:
+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
+      * Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"
+      * Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"


Suggested change

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러"; not "$20")

fanny-riols · 2026-05-29T14:52:43Z

+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
+      * Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"
+      * Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"
+      * Years: spoken form appropriate to the language, not "2024"


Suggested change

* Years: spoken form appropriate to the language, not "2024"

* Years: spoken form appropriate to the language (e.g. English: "twenty twenty-four"; not "2024")

fanny-riols · 2026-05-29T14:53:37Z

+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
+      * Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"
+      * Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"
+      * Years: spoken form appropriate to the language, not "2024"


Suggested change

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"

* Years: spoken form appropriate to the language, not "2024"

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre"; not "1/15/2024")

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig"; not "3:30 PM")

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러"; not "$20")

* Years: spoken form appropriate to the language (e.g. English: "twenty twenty-four"; not "2024")

fanny-riols · 2026-05-29T14:54:28Z

+      * Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
+      * Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"
+      * Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"
+      * Years: spoken form appropriate to the language, not "2024"


Suggested change

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"

* Years: spoken form appropriate to the language, not "2024"

* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre"; not "1/15/2024")

* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig"; not "3:30 PM")

* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러"; not "$20")

* Years: spoken form appropriate to the language ("twenty twenty-four" not "2024")

fanny-riols · 2026-05-29T14:59:00Z

+          "centre d’ingénierie du centre-ville",
+          "ingénierie centre-ville",
+          "bâtiment ingénierie centre-ville",
+          "le centre d’ingénierie en ville",


I would drop this one as it wouldn't be accurate (translating downtown to town)

Suggested change

"le centre d’ingénierie en ville",

fanny-riols · 2026-05-29T15:02:21Z

+          "downtown",
+          "engineering center",


Since we have this alias in English, I think we should also add the translations for these two in other languages.

Also, can you sort translation by language please?

fanny-riols · 2026-05-29T15:02:56Z

-          "Main Garage"
+          "a garage",
+          "main garage",
+          "garage a",


Wouldn't that match with the name already?

fanny-riols · 2026-05-29T15:05:07Z

        "name_aliases": [
-          "Downtown",
-          "Engineering Center"
+          "downtown",


Also, since we have all these alias in each file, shouldn't we put all the aliases in a separate common file?

raghavm243512 force-pushed the pr/multilingual branch from bd0e0d9 to 81923ab Compare May 19, 2026 18:39

raghavm243512 added 3 commits May 19, 2026 14:21

initial multilang impl

a4bcb4d

test fix

8ac2aaa

date formats

f5a5b52

raghavm243512 force-pushed the pr/multilingual branch from 68fb05a to f5a5b52 Compare May 19, 2026 21:28

translations and supporting stuff

ada9699

raghavm243512 force-pushed the pr/multilingual branch from 606dc7d to ada9699 Compare May 20, 2026 23:38

katstankiewicz reviewed May 21, 2026

View reviewed changes

raghavm243512 added 2 commits May 21, 2026 10:08

use display name for client

abc80e0

many finer points

b3ad5dc

raghavm243512 force-pushed the pr/multilingual branch from aa152bd to b3ad5dc Compare May 22, 2026 19:09

alias itsm

ff7bd59

raghavm243512 force-pushed the pr/multilingual branch from 26fa5ec to ff7bd59 Compare May 22, 2026 21:19

raghavm243512 and others added 10 commits May 22, 2026 21:20

Apply pre-commit

9b368f0

updated expected_db in dataset when adding culture data. update user_…

e3d90a4

…behavioral_fidelity judge prompt

update test

990eb76

add french number normalizer

b0d3a13

simplify adding languages

27be206

add language for elevenlabs

81c20a9

update stt_wer to handle french numbers

5a2377a

initial WER schema generation

bdbfe0b

docs and cleanup

132d19d

result improvements

30c6d43

raghavm243512 marked this pull request as ready for review May 28, 2026 21:14

update services to use 'settings' from pipecat update

dc910d9

fanny-riols reviewed May 29, 2026

View reviewed changes

	- Always provide analysis in English
	- Always provide analysis in English.

	* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre") not "1/15/2024"
	* Dates: spoken form matching local convention (e.g. English: "January 15th, 2024"; French: "le quinze janvier deux mille vingt-quatre"; not "1/15/2024")

	* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig") not "3:30 PM"
	* Times: spoken form (e.g. English: "three thirty PM"; German: "fünfzehn Uhr dreißig"; not "3:30 PM")

	* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러") not "$20"
	* Quantities: spoken form (e.g. English: "twenty dollars"; Korean: "이십 달러"; not "$20")

	* Years: spoken form appropriate to the language, not "2024"
	* Years: spoken form appropriate to the language (e.g. English: "twenty twenty-four"; not "2024")

Conversation

raghavm243512 commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katstankiewicz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fanny-riols May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raghavm243512 commented May 18, 2026 •

edited

Loading

fanny-riols May 29, 2026 •

edited

Loading