Dear all,

The upcoming seminar with Jeremy Barnes is cancelled. Unfortunately, our invited guest has fallen ill and is unable to travel to Bergen to join us. We were very much looking forward to the discussions and insights he was planning to share, and we hope to reschedule his visit at a later date. Thank you all for your interest in the seminars, and I hope to see you at other future events. Have a great week!

 

WP5 invites to a MediaFutures lunch seminar with Assistant Professor Jeremy Barnes, IXA Taldea / HiTZ Zentroa, University of the Basque Country (UPV/EHU).

Tittel: Challenges for automatic summarization evaluation in low-resource languages

Abstract:
In this talk, I will describe a transferable methodology for determining how effective available metrics are at identifying quality differences in automatic news summary quality in Basque and Spanish. After collecting a metevaluation dataset (BASSE) to evaluate five criteria (coherence, consistency, fluency, relevance, and 5W1H), we benchmark how well automatic evaluation metrics and LLM judges are able to capture differences in these criteria. Our results suggest that proprietary LLM judge models currently give the most consistent results across criteria, while a careful combination of automatic metrics can give similar results.