Assignment 1

[moecui22]

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

• Implemented structured summarization output using a Pydantic model (author/title/relevance/summary/tone + token counts).
• Added DeepEval-based evaluation (summarization + coherence + tonality + safety) with custom assessment questions.
• Added an “enhancement” step that rewrites the summary based on evaluation feedback and re-evaluates it.

What did you learn from the changes you have made?

How to evaluate LLM summaries using DeepEval with custom questions and multiple metrics.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

Yes: using chunked summarization (map-reduce) for long documents.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

Avoiding secret leakage (API key) in the notebook.

How were these changes tested?

Confirmed DeepEval returns scores + explanations for each metric and that the enhanced summary is re-evaluated successfully.

A reference to a related issue in your repository (if applicable)

NA

Checklist

• I can confirm that my changes are working as intended

[moecui22]

Hi,

Thanks for the help! I would like to bring up two minor issues I encountered while completing this assignment:

1. Course API key issue: The API key provided for the module was not working on my end (authentication failures). To complete the assignment and ensure the notebook runs successfully, I temporarily used my own personal OpenAI API key during development.

2. GitHub push protection (secret detection)
   When creating the pull request, GitHub blocked the push due to a secret detection violation. This appears to be related to an API key was accidentally included in the notebook output/history.
   I have since removed the key.

If you would prefer a different approach for handling the API configuration, or if there is a course-specific setup I should follow, please let me know. I will make necessary updates accordingly.

Thank you for your understanding. :)
Mo

[khsergvl]

Great work! No need to worry about usage of your own api / self-hosted model.