Day 1 – The Origin Story – Training the Eval Mindset

This all started with a word salad on LinkedIn the week before…

EVALS.

You’re seeing them more and more as LinkedIn’s algorithm feeds your interest in AI.

You probably know “Eval” comes from “Evaluate”. You know evals are important, but… are they important for you? where do you start?

If your job is tied in any way to improving the answer or task completion quality of users interacting with your models, The art of writing good Evals starts with knowing what they are. Like all good guidelines, these flex to the situation, but Evals are generally :

  • Yes/No Questions
  • Concise and Precise (and generally pronoun-free) with no supersets.
    • You wouldn’t need both “Is an animal” and “is a mammal” to describe a cat. All mammals are animals, so “is an animal” is unnecessary.
  • Atomic (“One Thought per Line”)
  • When combined, they uniquely describe your object
    • (in the real world, they’d be uniquely combining your project success criteria, but for the purposes of this game, we’re doing evals on objects).

But I didn’t need words to inspire me. I needed to git gud. And I wanted my teammates to git gud. So I fired up lovable and started with this:

Project Goals:

The overarching goal of this site is, through habitual daily use, to train product managers and AI engineers to “adopt the eval mindset”, which might also be expressed as “thinking in terms of evals” or could also be expressed as “training yourself to think eval-natively”.

But to be habitual, you need to be fun!

So to make learning this mindset fun, we’re going to use gamification techniques… nay, an actual game! To help get people hooked.

I’ve invented such a game, called Eval Genius, and you’re going to help me build it.

Overview

This is a daily puzzle app.

Step 1: Login/Create Account

Standard stuff here, nothing exciting, do whatever’s easiest including requiring login w/ google.

A user must be able to login, change their password, delete their account, and track their progress. More details below.

Step 2: See Today’s Object & Enter Evals

After login, the user is directed to that day’s “Game”

The game screen consists of a large single word/small phrase (3 words max) at the top, and then a series of text boxes, aligned on both edges in a vertical row, with a button to add more rows, and a button to submit, both at the bottom.  The user adds as many evals to the list as they think they need (remember the goal of evals is as few as possible to do the job), and then clicks submit and moves to step 3.  There are five lines visible by default.  The user must be instructed to phrase their question so that the object in question is matched to a “YES” answer for all questions.

Step 3: Submit & Score

In step 3, we score their work. we gather each of the entered evals and build this prompt:

[here’s where I added my scoring algorithm. Happy to share it if you message me, but not publicly posting this part]

This all started with a word salad on LinkedIn the week before…

EVALS.

You’re seeing them more and more as LinkedIn’s algorithm feeds your interest in AI.

You probably know “Eval” comes from “Evaluate”. You know evals are important, but… are they important for you? where do you start?

If your job is tied in any way to improving the answer or task completion quality of users interacting with your models, The art of writing good Evals starts with knowing what they are. Like all good guidelines, these flex to the situation, but Evals are generally :

  • Yes/No Questions
  • Concise and Precise (and generally pronoun-free) with no supersets.
    • You wouldn’t need both “Is an animal” and “is a mammal” to describe a cat. All mammals are animals, so “is an animal” is unnecessary.
  • Atomic (“One Thought per Line”)
  • When combined, they uniquely describe your object
    • (in the real world, they’d be uniquely combining your project success criteria, but for the purposes of this game, we’re doing evals on objects).

But I didn’t need words to inspire me. I needed to git gud. And I wanted my teammates to git gud. So I fired up lovable and started with this:

Project Goals:

The overarching goal of this site is, through habitual daily use, to train product managers and AI engineers to “adopt the eval mindset”, which might also be expressed as “thinking in terms of evals” or could also be expressed as “training yourself to think eval-natively”.

But to be habitual, you need to be fun!

So to make learning this mindset fun, we’re going to use gamification techniques… nay, an actual game! To help get people hooked.

I’ve invented such a game, called Eval Genius, and you’re going to help me build it.

Overview

This is a daily puzzle app.

Step 1: Login/Create Account

Standard stuff here, nothing exciting, do whatever’s easiest including requiring login w/ google.

A user must be able to login, change their password, delete their account, and track their progress. More details below.

Step 2: See Today’s Object & Enter Evals

After login, the user is directed to that day’s “Game”

The game screen consists of a large single word/small phrase (3 words max) at the top, and then a series of text boxes, aligned on both edges in a vertical row, with a button to add more rows, and a button to submit, both at the bottom.  The user adds as many evals to the list as they think they need (remember the goal of evals is as few as possible to do the job), and then clicks submit and moves to step 3.  There are five lines visible by default.  The user must be instructed to phrase their question so that the object in question is matched to a “YES” answer for all questions.

Step 3: Submit & Score

In step 3, we score their work. we gather each of the entered evals and build this prompt:

[here’s where I added my scoring algorithm. Happy to share it if you message me, but not publicly posting this part]

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts