4 posts tagged with "Langfuse"

Two agents answer the same user query. Both return the identical string—correct, well-formatted, cited. An answer-level eval gives them both a perfect score, identical down to the decimal.

One agent made three redundant retrieval calls (same tool, same query, same corpus) before stumbling on the right source. The other called exactly the right tool once and answered. The answer-level eval cannot tell the difference. It never could.

The keys are in the trajectory.

I built a trajectory observability lane for my agents in three small pieces: the JSONL traces every workflow already emits but nobody reads, a judge that scores the tool-call sequence instead of the answer, and a Langfuse uploader written against the raw REST ingestion API—no SDK. Publication volumes indicate this is the moment: agent-observability research jumped sharply into 2026 (the phrase barely existed before), and the first dedicated fault-detection benchmark for agent observability was published this week.

This post is the full walkthrough: what trajectory observability is, why answer-level evals miss half the story, the three-module build, and how the research on partial observability validates the approach.

Running AI agents in a real codebase means solving three intertwined problems at once: planning and quality gates (so agents don't drift), observability (so you know what's working), and orchestration (so multiple agents divide work without clobbering each other). In nomadically.work — a remote EU job board with an AI classification and skill-extraction pipeline — these problems are solved by three complementary systems: BMAD v6, Langfuse, and Claude Code Agent Teams. This article explains how each works and how they compose.

A comprehensive guide to implementing Langfuse features for production-ready AI applications, covering prompt management, tracing, evaluation, and observability.

Overview​

This guide covers:

• Prompt management with caching and versioning
• Distributed tracing with OpenTelemetry
• User feedback and scoring
• Usage tracking and analytics
• A/B testing and experimentation

This article documents an evaluation harness for a Remote EU job classifier—but the real focus is AI observability: how to design traces, spans, metadata, scoring, and run-level grouping so you can debug, compare, and govern LLM behavior over time.