Measuring the performance of LLM products
Title Speaker Date Description
A Deep Dive on LLM Evaluation Hailey Schoelkopf Jul 8, 2024 Doing LLM evaluation right is crucial, but very challenging! We’ll cover the basics of how LLM evaluation can be performed, many (but not all) of the ways it can go wrong. We’ll also discuss tools available to make life easier, including the LM Evaluation Harness, along with domain-specific use cases.
Inspect, An OSS framework for LLM evals JJ Allaire Jul 6, 2024 This talk will cover using and extending Inspect, a new OSS Python framework for LLM evals. We’ll walk through the core concepts and design of Inspect and demonstrate its use for a variety of evaluation tasks. Inspect makes it very straightforward to implement simple “benchmark” style evaluations, but also has the depth and flexibility to support highly complex evals. Inspect supports both…
No matching items