Enroll for free to learn more!
www.deeplearning.ai/short-course...

Why this matters for on-prem:
🔒 Private CA support—no public cert required
🏠 Data never leaves your network
👥 Leverage existing AD groups for access control
Requires Phoenix 12.20.0+

arize.com/docs/phoeni...

Watch the webinar here! www.youtube.com/watch?v=n-O...

Demo Code is here: github.com/Arize-ai/tu...

Phoenix Evals now handles:

• string OR message templates
• {var} and {{var}} syntax
• automatic provider-specific transformations
• consistent, reproducible scoring

More prompt-engineering context here: arize.com/docs/phoeni...

Message-separated prompts fix this by isolating:

🔹 Evaluator instructions (system/developer)
from
🔹 Content being judged (user)

This reduces false positives from content filters and stabilizes judge behavior.

This is critical because providers interpret roles differently:

• OpenAI: system becomes developer for reasoning models
• Anthropic and Gemini: system extracted into a top-level param

If everything is shoved into a message, you get safety flags, refusals, and polluted evals.

With Phoenix Evals you can now define evaluators using OpenAI-style message lists:

[
{"role": "system", "content": "You evaluate helpfulness."},
{"role": "user", "content": "Question: {question}\nAnswer: {answer}"}
]

Variables get slotted into the correct role, preserving evaluator intent.

Historically, many eval pipelines used single-string prompts. But today’s LLMs are role-aware, and that old pattern can lead to:

• task vs. judge confusion
• content-filter refusals
• inconsistent or drifted scoring
• evaluator instructions being ignored

Message formatting is now a critical piece

This matters even more in light of OpenAI’s latest model spec update, which emphasizes stricter role semantics across system / developer / user messages:
model-spec.openai.com/2025-10-27....

If you have any feedback, please let @arizeai @mikeldking and the team know!

API docs: arize.com/docs/phoeni...

This process has been core to ML error analysis for decades. @GergelyOrosz and @HamelHusain wrote an excellent deep-dive on how teams are applying it to LLM development: newsletter.pragmaticengineer.com/p/evals

What you get: a data-driven priority list of your system's real failure modes, not what a generic benchmark thinks might be wrong.

The workflow:

Review 100+ diverse traces
Write descriptive notes on anything that feels wrong
Group similar notes into themes (axial coding)
Count the frequency of each theme

"Asked for confirmation twice"
"Kept trying to solve instead of handing off"
These descriptive observations are more valuable than generic scores—they tell you what's actually breaking.

The idea: instead of starting with predefined categories like "hallucination" or "helpfulness," you review your traces and write open-ended notes about what you observe. Let the failure modes emerge from your data.
"Missed opportunity to re-engage a price-sensitive user"

Open coding is a technique from qualitative research that's becoming essential for LLM evaluation.