-
Notifications
You must be signed in to change notification settings - Fork 6
Improvement/234 create main evaluation script #243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dkkdark
wants to merge
46
commits into
main
Choose a base branch
from
improvement/234-create-main-evaluation-script
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
ceee927
create base metric class
46b7456
return pre commit config versions
8c96457
#233 add new classes
950964c
#232 fix class
33624b2
#232 fix class
1053b57
Merge branch 'improvement/232-create-abstract-class-for-metric' of ht…
27f7888
#232 add aggregation
c38331a
#232 fix methods
901dd5a
#232 fix nits
c4efd79
improvement/233-create-classes-for-metrics add classes
800f8c0
improvement/232-create-abstract-class-for-metric fixes
2014e22
improvement/232-create-abstract-class-for-metric fixes
363551c
improvement/232-create-abstract-class-for-metric remove agent evaluation
9ab36e2
Merge branch 'improvement/232-create-abstract-class-for-metric' of ht…
a253f74
improvement/233-create-classes-for-metrics edit classes
777e00e
improvement/233-create-classes-for-metrics remove old files
5a55b43
improvement/233-create-classes-for-metrics remove extra files
c5bb100
improvement/233-create-classes-for-metrics remove changes
a1fac9f
improvement/233-create-classes-for-metrics remove changes
4ed5727
move file
e6f0ef3
move file
431ae40
refactoring
6f74153
add script
773822d
fixes
5e1b3a6
fixes
558b588
made fixes
d78bda3
new structure
9146421
remove class from other pr
fe8ea9c
changes after new structure
d749156
remove main
8f333e7
changes
6e6e8b2
changes
1c802ff
changes
8335005
simplyfying
f006930
remove utility class
e0f3c56
fix get_recommendation_rounds
4157fdc
fixes
3b068b4
resolve issues
d7efc12
Merge branch 'main' of https://github.com/iai-group/UserSimCRS into i…
8944621
fixes
335d3bd
Merge branch 'improvement/234-create-main-evaluation-script' of https…
98564af
Merge branch 'improvement/233-create-classes-for-metrics' of https://…
8be81e4
234-create-main-evaluation-script add eval script
a637a40
Merge branch 'main' into improvement/234-create-main-evaluation-script
NoB0 b3cd18f
fixes
938ccac
fix evaluation
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| dialogues: data/datasets/moviebot/annotated_dialogues.json | ||
| debug: False | ||
| metrics: | ||
| - satisfaction | ||
| - success_rate | ||
| - successful_recommendation_round_ratio | ||
| - reward_per_dialogue_length | ||
| output_dir: data/evaluation/moviebot_non_quality_results | ||
|
|
||
| quality_llm_interface: | ||
| llm_interface_class_path: "usersimcrs.llm_interfaces.ollama_interface.OllamaLLMInterface" | ||
| llm_interface_args: | ||
| configuration_path: config/llm_interface/config_ollama_default.yaml | ||
| default_response: "" | ||
| quality_aspects: | ||
| - REC_RELEVANCE | ||
| - COM_STYLE | ||
| - FLUENCY | ||
| - CONV_FLOW | ||
| - OVERALL_SAT | ||
|
|
||
| annotate_dialogues: False | ||
| user_nlu: | ||
| type: "cosine" | ||
| agent_nlu: | ||
| type: "cosine" | ||
|
|
||
| recommendation_intent_labels: | ||
| - REVEAL | ||
| - REVEAL.SIMILAR | ||
| - REVEAL.NONE | ||
| - REVEAL.REVISE | ||
| accept_intent_labels: | ||
| - NOTE.ACCEPT | ||
| reject_intent_labels: | ||
| - NOTE.DISLIKE | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.