Graphs come into play here, since context from knowledge graphs can help guide agents toward better outcomes, and also the “competency questions” used for knowledge graph construction feed nicely into generating evaluations to use for LLM observability. Let’s talk about optimization with Claire!

Senzing has been using Opik in our AI tutorials, and Claire’s talks about leveraging math-driven thinking to build better AI caught our attention, especially her analogies which help explain relatively complex mathematical approaches in terms of real world examples.

This is a real world example of a math-guided design. This feedback mirrors the dopamine signaling in animal brains which regulates emotional and motivational behavior through “rewards” given by neurotransmitters.

If you step back and look in big picture terms, this feedback loop is essentially the same used in reinforcement learning, where agents learn and improve through feedback based on how they interact with their environment.

Agent optimizers include GEPA, HRPO, and more -- and can be extended.

Open a browser-based dashboard for your application and see how the evaluations of LLM integrations have run on different sessions. At last count, Opik includes 52 integrations with popular Python tooling for AI (we counted) plus more for Ruby, TypeScript, .NET, and Java.

Opik is a popular open source platform which gives you all the tools you need for LLM observability (logging, debugging, evaluations, experiments), enabling end-to-end AI Engineering through development, testing, and production. Plus there’s an open-source SDK for agent and prompt optimization.

What’s been emerging is a feedback loop involving observation and optimization. If you've worked with AutoML approaches or reinforcement learning, you know this already.

As work on AI applications – agents, tool uses, external memory modules, chain-of-thought and related methods – becomes more sophisticated, the need for evaluating results and putting the results of evaluations to good use becomes more acute.

Beyond her technical work, Claire is a Speaker, Advisor, YouTuber, and Poker Player, dedicated to mentoring Engineers and Data Scientists while championing diversity and inclusion in AI. Her mission is to empower the next generation of AI practitioners. statisticianinstilettos.com

Claire is an AI leader and Mathematician with a decade+ experience in Data Science and AI. She's led cross-functional AI teams at Twilio, Opendoor, and Arize AI -- currently Lead AI Researcher at Comet. Claire holds an Applied Mathematics BS and Statistics MS from U New Mexico.
www.comet.com