Framework-agnostic agent evaluation still has an instrumentation contract

AWS describes Amazon Bedrock AgentCore Evaluations as a framework-agnostic path built on OpenTelemetry or OpenInference telemetry. That description does not make compatibility automatic. The integration still depends on recognized instrumentation scope names, session grouping, required span roles, accessible message content, and telemetry reaching the backend before runtime suspension. Read the AWS description.

For a platform team, the framework label is therefore less useful than the records produced by one deployed configuration. Library versions, middleware, exporter settings, runtime lifecycle, and application code can all change those records while the agent itself continues to run.

Where framework agnosticism ends

The documented integration is agnostic about the agent framework only to the extent that the resulting telemetry satisfies its input conditions. A deployment can fail that condition in several distinct ways: it may emit nothing, export records that never arrive, use an unrecognized scope or trace shape, combine unrelated sessions, or omit content needed for the intended evaluation.

OpenTelemetry illustrates the expected kind of structure with a top-level invoke-agent span and child chat and execute-tool spans identified through GenAI semantic-convention attributes. The project also says those conventions remain under active development. The example documents a trace shape and its versioning risk; it does not establish what an arbitrary deployment emits. See the OpenTelemetry example and status note.

That distinction suggests a compatibility record tied to the deployed configuration rather than a general claim about its framework. Our proposed SGL record has six parts. It is an engineering prescription derived from the documented dependencies, not an AWS certification.

The six parts of a compatibility record

One small scenario can produce the necessary evidence: a session with two messages, one tool invocation, one inference response, and a normal runtime suspension or shutdown. The relevant output is not only the trace visible at the application boundary. It includes both the raw exported telemetry and the backend’s ingested representation.

The record should answer six questions:

1. Scope recognition: Which instrumentation scope names appear on evaluation-relevant records, and does the backend classify them as intended?
2. Span roles: Are agent, inference, and tool activity distinguishable, with the expected parent-child relationships?
3. Session grouping: Do multiple turns from one session share the intended grouping key while a second session remains separate?
4. Message availability: Can the evaluator access the input and output fields required for the intended evaluation under the team’s collection policy?
5. Delivery before suspension: Do buffered spans and event records reach the backend through the runtime’s normal lifecycle, with expected and received counts reconciled?
6. Version identity: Which framework, instrumentation, exporter, and semantic-convention versions produced the accepted representation?

The first five questions correspond to compatibility dependencies AWS identifies. Version identity accounts for OpenTelemetry’s warning that the GenAI conventions remain under active development. Review the AWS compatibility conditions and the OpenTelemetry convention status.

Message availability needs a policy decision alongside the technical result. The AWS description identifies accessible content as a dependency, but the supplied evidence does not prescribe what an organization should collect. The record can instead name the fields required by a specific evaluation and show whether that configuration made them available. It should not treat maximal collection as the default.

A result belongs to a configuration

The six answers are most useful when kept with enough context to locate a failure. That context can include framework and adapter versions, scope names, semantic-convention version, trace and session identifiers, expected and received span counts, span-role distribution, exporter flush results, backend ingestion timestamps, and redacted samples of required message fields.

This supporting material is our recommended artifact, not a requirement stated by either source. Its purpose is diagnostic. It separates absent telemetry from failed delivery, an unrecognized representation, incorrect session grouping, and missing evaluation inputs.

Compatibility can also expire. A framework upgrade may alter instrumentation; an exporter change may affect delivery; a runtime change may alter shutdown behavior; and a semantic-convention revision may change the accepted trace shape. The stored scenario should therefore travel with the configuration and be revisited when any of those components or the evaluation backend changes.

The operational claim remains specific: the named configuration produced interpretable records and delivered them to the intended backend under the recorded conditions. “Framework-agnostic” describes the integration approach, not every deployment built with a supported framework.

Teams can keep this compatibility record beside the configuration in their broader agent-system build process. That makes an evaluation dependency visible at the point where versions, lifecycle behavior, collection policy, and backend interpretation meet.