congrats!

Work by Nawaf Alampara and Mara Schilling-Wilhelmi.

Support by @carlzeissstiftung.bsky.social, Intel & Merck, OpenPhilanthropy, @fairmat.bsky.social

Work done at the Friedrich-Schiller University Jena and HIPOLE Jena.

As a first step, we propose "eval cards" github.com/lamalab-org/... which hopefully encourage developers of eval methods to slow down (another thing I hope we do as a field: kjablonka.com/blog/posts/t...) and craft evals that really help move the field forward.

We hope to find ways to make it more transparent what assumptions go into building evaluations and to be more thoughtful when designing measurement instruments.

This article ended up including inspiration from many different fields and is again written in a (probably) unconventional form: But I really hope that some of the ideas in there spark a discussion as the field really needs those discussions.

And add this on top of the fact that most of our measurements in machine learning are actually defined by the act of the measurement itself - which is very unintuitive for a physical scientist.

To me, there is really a need for a "science of evals".

When we build benchmarks or other methods for testing machine learning systems, we make many assumptions - too often hidden - that can drastically change what we measure (see the parallel coordinates plots).

what is wrong with the link?

More details: www.dropbox.com/scl/fi/mlew5...

It would help us a lot if you could support us in spreading the word about the openings in our team! 🙏🏼

We strongly encourage candidates of all different backgrounds and identities to apply. Each new hire is an opportunity for us to bring in a different perspective, and we are always eager to diversify our team further.

In particular, we are looking for PostDocs to support our efforts in building and testing frontier models.

😂