BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//MediaFutures - ECPv6.15.13.1//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:MediaFutures
X-ORIGINAL-URL:https://mediafutures.no
X-WR-CALDESC:Events for MediaFutures
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:Europe/Oslo
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20250330T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20251026T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20260329T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20261025T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20270328T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20271031T010000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Oslo:20260912T111000
DTEND;TZID=Europe/Oslo:20260912T121000
DTSTAMP:20260904T230549
CREATED:20260811T093805Z
LAST-MODIFIED:20260813T080846Z
UID:23212-1789211400-1789215000@mediafutures.no
SUMMARY:PYData Seminar: Stop Early\, Decide Smarter: Bayesian Sequential Testing for LLM Benchmarking
DESCRIPTION:Ryan Marinelli will be holding a tutorial at PyData\, Amsterdam. \nEvaluating large language models is expensive. Standard benchmarking practices run fixed numbers of trials regardless of how quickly signal emerges\, wasting compute when differences are obvious and under-sampling when they are subtle. \nThis talk introduces Bayesian sequential stopping rules as a principled alternative. Rather than committing to a sample size upfront\, sequential methods continuously update a posterior over model performance and halt evaluation once sufficient evidence has accumulated\, or once further sampling is unlikely to change the conclusion. \nWe walk through the statistical foundations\, show how this framework applies to common benchmarking scenarios (accuracy comparisons\, pass@k\, agent task completion)\, and demonstrate a practical open-source Python implementation attendees can use immediately. Real benchmark datasets are used throughout to ground the approach. \nAudience: ML engineers\, researchers\, and data scientists who run model evaluations and care about making reliable comparisons without burning unnecessary compute. \nTakeaways: A conceptual understanding of sequential testing\, practical guidance on when and how to apply it\, and a working tool to take home. \nFixed-sample benchmarking is the default in NLP and LLM evaluation\, but it has two failure modes: it over-runs when models are clearly different early on\, and under-runs when tight comparisons require more evidence than the budget allows. Sequential hypothesis testing\, long standard in clinical trials and A/B testing\, offers a cleaner framework: keep sampling until a decision threshold is crossed\, or until a maximum budget is exhausted. \nThis talk applies that framework specifically to LLM and agent benchmarking and covers: \n\nThe statistical case: Why pass@k evaluation is often wasteful\, and what sequential Bayes factors and posterior stopping criteria offer instead\nDesign choices: Prior elicitation\, stopping thresholds\, and how to handle multi-model comparisons without inflating error rates\nPractical implementation: A walkthrough of a pip-installable Python package built on this framework\nLimitations and scope: When sequential testing helps\, when it doesn’t\, and how to sanity-check results\n\nThe talk is self-contained; attendees need only basic familiarity with probability and Python. \nOutline (30 min):\nMotivation: the cost of fixed-N benchmarking — 5 min\nSequential testing foundations (Bayes factors\, stopping rules) — 8 min\nDemo: the Python package on real benchmark data — 10 min\nDesign guidance\, edge cases\, and Q&A — 7 min
URL:https://mediafutures.no/event/pydata-seminar-stop-early-decide-smarter-bayesian-sequential-testing-for-llm-benchmarking/
LOCATION:NDSM-Plein 85 1033 WC Amsterdam\, NDSM-Plein 85 1033 WC Amsterdam\, Netherlands
CATEGORIES:Events
ATTACH;FMTTYPE=image/png:https://mediafutures.no/wp-content/uploads/Screenshot-2026-08-07-at-15.54.34.png
END:VEVENT
END:VCALENDAR