Task 1. LongEval-Retrieval:
(2) When do we need to update an IR system as the collection of documents to be searched in changes? If we are able to assess the decrease in performance (if any) of a system on an evolving collection, we may then decide if the system needs to be updated. -To assess an information retrieval system, we provide several datasets which are 3 snapshots of a changing Web documents and users’ queries:
+To assess an information retrieval system, we provide several datasets which are snapshots of changing Web documents and users’ queries over a period of 15 months:
- - - Training set, acquired at January 2023 composed of documents, queries, and qrels. + - Training set, acquired from June 2022 to February 2023, composed of documents, queries, and qrels. -
- - Lag 5 test set, acquired at June 2023, composed of documents and queries. -
- - Lag 7 test set, acquired at August 2023, composed of documents and queries. +
- - Test set 1, acquired in March 2023, composed of documents and queries. +
- - Test set 2, acquired in April 2023, composed of documents and queries. +
- - Test set 3, acquired in May 2023, composed of documents and queries. +
- - Test set 4, acquired in June 2023, composed of documents and queries. +
- - Test set 5, acquired in July 2023, composed of documents and queries. +
- - Test set 6, acquired in August 2023, composed of documents and queries.
Task 2. LongEval-Classification:
-The goal of LongEval-Classification Subtask B of CLEF 2024 Task 2 is to propose a temporal persistence classifier which can mitigate performance drop over short and long periods of time compared to a test set from the same time frame as training.
+The goal of LongEval-Classification Subtask B of CLEF 2025 Task 2 is to propose a temporal persistence classifier which can mitigate performance drop over short and long periods of time compared to a test set from the same time frame as training.
The organizers will provide a training set collected over a time frame up to a time t and two test sets: test set from time t and test set from time t+i where i=1 for sub-task A and i>1 for subtask B.
Sub-task short-term persistence. Short-term persistence. In this sub-task participants will develop models which demonstrate performance persistence over short periods of time, i.e. using test set within 2-3 years apart from the training data.
Sub-task long-term persistence. Long-term persistence. In this sub-task participants will develop models which demonstrate performance persistence over longer period of time, i.e. test set within 4-5 years apart from the training data and also distant from the short-term test set.