Skip to content

test Llm/deepeval api tool calling #232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nuwangeek wants to merge 5 commits into
base: deepeval-temp
Choose a base branch
from
Open

test Llm/deepeval api tool calling #232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cf3d82f
Fixed deepeval tests for api tool calling
nuwangeek Jun 25, 2026
a08ab1d
updated deepeval branch
nuwangeek Jun 25, 2026
d74b1d2
reduced dataset
nuwangeek Jun 26, 2026
43e4565
fixed issue
nuwangeek Jun 26, 2026
8edf65d
fixed issue
nuwangeek Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/deepeval-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: DeepEval RAG System Tests
on:
pull_request:
types: [opened, synchronize, reopened]
branches: ["wip-eval"]
branches: ["deepeval-temp"]
paths:
- 'src/**'
- 'tests/**'
Expand Down
24 changes: 0 additions & 24 deletions tests/data/test_dataset.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,29 +34,5 @@
"expected_output": "Eesti sotsiaaluuring aitab hinnata leibkondade ja isikute sissetulekute jaotust, elamistingimusi ning sotsiaalset tõrjutust. Samasugust uuringut korraldatakse kõikides Euroopa Liidu riikides. Teie osalemine aitab kujundada Eesti sotsiaalpoliitikat ja arendada teenuseid. Miks see on oluline: Need andmed aitavad riigijuhtidel, teadlastel ja arvamusliidridel mõista inimeste toimetulekut, vaesust, ilmajäetust, sissetulekut, elamistingimusi ja arstiabi kättesaadavust. Iga uuringusse sattunud inimene esindab umbes sadat Eesti elanikku, seega on teie vastused olulised. Kuidas osaleda: Kui teid valitakse, saate teavituskirja e-postile (rahvastikuregistris märgitud aadressile) või postiga. Saate küsimustikule vastata veebis aadressil https://estat.stat.ee, logides sisse ID-kaardi, mobiil-ID või Smart-ID-ga, või osaleda telefoniintervjuus. Uuring koosneb kahest osast: leibkonnaküsimustik (vastab põhiisik) ja isikuküsimustik (võib vastata iga vähemalt 15-aastane leibkonnaliige). Osalemine on vabatahtlik, kuid Statistikaamet kaitseb kõigi vastajate andmeid riikliku statistika seaduse ja isikuandmete kaitse seaduse alusel. Iga leibkonda küsitletakse neli aastat järjest 12 kuu tagant.",
"category": "statistics",
"language": "et"
},
{
"input": "Kas ma saan kodus elektritöid ise teha või vajan spetsialisti?",
"expected_output": "Enamik elektritöid nõuab ohutuse huvides kvalifitseeritud spetsialisti. Valesti tehtud elektritöö võib põhjustada elektrilöögi ning tulekahjuohtu. Siiski saate lihtsamaid töid ise teha, kui teil on vastavad teadmised. Mida VÕITE ise teha: Vahetada lüliteid, pistikupesi, lambipesi ja kaitsmeid (kuid MITTE paigaldada uusi) Parandada ja asendada juhtmelüliteid, lambipesi, pikendusjuhtmeid ja juhtmepistikuid Milleks PEATE palkama spetsialisti: Uute elektripaigaldiste ehitamine Uute pistikupesade ja lülitite paigaldamine Kohtkindlate kodumasinate ühendamine ja lahti ühendamine Kaitsekontaktita (maandamata) pistikupesade vahetamine kaitsekontaktiga (maandatud) pistikupesade vastu Elektritöö ettevõtjad peavad olema esitanud majandustegevuse registrisse majandustegevuseteatise ning neil peab olema tööde eest vastutav kompetentne elektritöö juht.",
"category": "ttja",
"language": "et"
},
{
"input": "What is an electrical installation audit and when do I need one?",
"expected_output": "An electrical installation audit checks whether your electrical system meets safety requirements and is safe to use. During the audit, the auditor visually assesses the installation's condition, reviews documentation and test/measurement results, and performs additional control measurements if necessary. When you need an audit: Before commissioning: Required before putting a new or renovated building's electrical installation into use Periodic audits: Regular checks at intervals depending on the installation type and age. While not mandatory for residential spaces (private houses, apartments, summer cottages), you should still periodically check them to ensure safety and functionality How to get an audit: Only contractors with appropriate accreditation can perform audits Results and documents are digitally formatted in TTJA's information system at https://jvis.ttja.ee, where they're always accessible to the electrical installation owner For residential electrical system checks, contact a competent electrical professional or auditor who will perform necessary operations and provide feedback on the system's condition and safety.",
"category": "ttja",
"language": "en"
},
{
"input": "How long is the e-residency digi-ID valid for?",
"expected_output": "The e-residency digi-ID is valid for 5 years",
"category": "digital_identity",
"language": "en"
},
{
"input": "Предоставляет ли электронное резидентство эстонское гражданство или налоговое резидентство?",
"expected_output": "Нет, электронное резидентство не предоставляет эстонское гражданство или налоговое резидентство.",
"category": "digital_identity",
"language": "ru"
}
]
10 changes: 5 additions & 5 deletions tests/deepeval_tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ def _write_test_secrets(self, client: hvac.Client) -> None:
"endpoint": azure_endpoint,
"api_key": azure_api_key,
"deployment_name": azure_deployment or "gpt-4o-mini",
"environment": "testing",
"environment": "development",
"model": "gpt-4o-mini",
"model_type": "chat",
"api_version": "2024-02-15-preview",
Expand Down Expand Up @@ -401,7 +401,7 @@ def _write_test_secrets(self, client: hvac.Client) -> None:
"endpoint": azure_endpoint,
"api_key": azure_api_key,
"deployment_name": azure_embedding_deployment, # This is the embedding deployment
"environment": "testing",
"environment": "development",
"model": "text-embedding-3-large",
"model_type": "embedding",
"api_version": "2024-02-15-preview",
Expand Down Expand Up @@ -434,7 +434,7 @@ def _write_test_secrets(self, client: hvac.Client) -> None:
try:
# Verify LLM path
verify_llm = client.secrets.kv.v2.read_secret_version(
path="llm/connections/azure_openai/development/evalconnection-1",
path="llm/connections/azure_openai/evalconnection-1",
mount_point="secret",
)
llm_data = verify_llm["data"]["data"]
Expand All @@ -443,7 +443,7 @@ def _write_test_secrets(self, client: hvac.Client) -> None:

# Verify embeddings path
verify_embedding = client.secrets.kv.v2.read_secret_version(
path="embeddings/connections/azure_openai/development/evalconnection-1",
path="embeddings/connections/azure_openai/evalconnection-1",
mount_point="secret",
)
embedding_data = verify_embedding["data"]["data"]
Expand Down Expand Up @@ -617,7 +617,7 @@ def _verify_token_permissions(self, client: hvac.Client) -> None:
"""Verify the token has correct permissions to read secrets"""
try:
client.secrets.kv.v2.read_secret_version(
path="llm/connections/azure_openai/development/evalconnection-1",
path="llm/connections/azure_openai/evalconnection-1",
mount_point="secret",
)
logger.info("Token has correct permissions to read secrets")
Expand Down
Loading