Agentic AI is having a moment, and for good reason — AI agents that can perceive their environment, reason autonomously, use tools, and collaborate with each other represent a giant leap beyond the chatbots and automation scripts that came before.

Unfortunately, most organizations don’t yet know how to effectively deploy those agents, manage their performance, and get them working together. They’re spinning up agents, giving them access to a bunch of systems, and hoping for the best. Some companies manage their people like that, and if you’ve had the misfortune of working for one, you know how frustrating and ineffective it can be. Which leads me to the premise of learning lessons from HR to improve the effectiveness of agentic AI initiatives.

The Key to Making Your Agents Effective

Think about what makes an AI agent tick, and what they need to do their job. They have:

• A brain (a large language model, or LLM) capable of reasoning and planning
• Specialized skills tailored to a domain
• Tools that let them take action — searching, writing, calling APIs or MCP servers, etc.
• Access to the information they need to do their job.

First, note that people you hire have the same things. Then consider the fact that even the best and brightest people that you hire don’t show up on day one ready to operate autonomously across every corner of your business, and you don’t just let them do their thing forever without checking in on them. They go through a very deliberate process, i.e. they are:

• Onboarded into systems and workflows
• Trained on company-specific knowledge
• Supervised as they build trust; then evaluated and developed over time
• Integrated into high-performing teams

AI agents need those same things!

What Happens When You Skip the Process

Picture an organization where every employee is a generalist, there’s no hierarchy or structure, and all communication happens via synchronous one-to-one phone calls. No email. No Slack. No shared systems. That organization can’t scale. Information doesn’t flow. People drown in data from every direction, with no real understanding of what’s going on or what to do. There are no accountability structures, no access controls, no coordination mechanisms beyond individual conversations.

This is how most early agentic AI deployments work: complex monolithic agents tasked with doing everything, accessing all data, communicating synchronously, and trying to maintain very large context or working synchronously with sub-agents. The result? A system that’s brittle, expensive, inconsistent, and completely unable to handle real enterprise complexity.

The Better Model: Specialists, Hierarchy, and Asynchronous Communication

Effective organizations look different – people are specialized, and organized hierarchically, so coordination happens in parallel, and they communicate primarily through asynchronous channels that let information flow without requiring everyone to be online at the same time. They implement focused context and role-based access to systems, so people see only what they need.

Solace Agent Mesh, Solace’s agent development and runtime platform, applies these same concepts to agentic AI. Agent Mesh helps you build agentic applications that actually work for your enterprise: business users build agents with no code while developers code sophisticated ones. They’re all deployed and managed via one interface, fed real-time data through pre-built connectors, and governed by the role-based access controls we’re known for, with a visual interface for monitoring agent activity in real time.

With Agent Mesh, agents are specialized for particular functions, organized hierarchically, and their actions are orchestrated by agents that specialize in delegating tasks to appropriate agents. Communication between agents is asynchronous and event-driven — built on Solace’s proven real-time data infrastructure – and access to data and systems is governed by role-based access controls, giving agents exactly the permissions they need and nothing more.

 

How Solace Agent Mesh Lets You Treat Your Agents Like Employees

This kind of structured, lifecycle-based thinking has a name in the agentic AI world: the agent development lifecycle, or ADLC. Just as the software development lifecycle (SDLC) brought repeatable discipline to how applications get built, shipped, and maintained, the ADLC brings that same discipline to how AI agents are hired, onboarded, trained, supervised, organized into teams, and continuously improved. It’s how the industry is starting to formalize what good organizations have always done with their people — applied to the new workforce of software-based agents. We need a new kind of development lifecycle rather than the traditional SDLC because agents are different than traditional software – so they require specialized consideration. (For a deeper look at the framework itself, see our paper Managing AI Agents at Scale: The Agent Development Lifecycle.)

Solace Agent Mesh organizes the full journey of an AI agent into six lifecycle stages and supports those who develop agents with capabilities at each stage of the lifecycle — each reflective of a phase of the human employee experience, and mapped to the ADLC, seen here:

1. Hiring: Define Role, Responsibilities, Expectations

Before a human employee starts, you write the job description. You define what role they’ll fill, what they’re responsible for, what behaviors you expect, and what boundaries they operate within.

Agent Mesh’s agent builder uses an internal AI agent to provide a guided interface for defining an agent’s purpose, scope, and configuration. You set up the instructions and system prompts that shape the agent’s persona and behavioral parameters. And you configure guardrails — hard constraints on what the agent can and cannot do — that prevent it from going off-script or taking unsafe actions.  Alternatively, you can take a pro-code/pro-config approach to building agents in your favorite AI coding assistant aided by an Agent Mesh skill that make your coding assistant an expert at creating Agent Mesh agents.

2. Onboarding: Give Access to Systems

A new employee’s first weeks are spent getting access to tools, systems, and data. Agent onboarding is the same.

Agent Mesh includes a number of data connectors – pre-built integrations to enterprise databases, data warehouses, data lakes, enterprise application APIs, and MCP servers. They enable agents to access real-time data from day one. Access is provisioned through role-based access controls (RBAC), enforcing least-privilege principles so agents only see what they need to see.  You can create your own agent tools to access your systems or create a semantic data layer or to write code to do whatever you need an agent to do using our SDK, then load it into Agent Mesh for use by the agents. You can also define Skills that go with your tools so agents know best how to use them.

3. Coaching: Internal Training to Ensure Competence

After getting access to systems, human employees undergo training. This is where general ability becomes company and job-specific skills.

Agent Mesh provides an Eval function that uses AI to suggest tests for your agents, allows you to add more and then run them against your agents. This allows you to validate agent competence against defined success criteria during development.  This supports both initial testing and regression tests as you make changes. You wouldn’t put a new hire in front of customers without verifying they know what they are doing. The same logic applies here.

4. Supervision: Trust, But Verify

Even the most capable employee gets close oversight when they’re new to a role. Supervision isn’t micromanagement — it’s the safety net that ensures quality, builds trust, and catches errors before they compound.

Agent Mesh’s human-in-the-loop architecture lets you route specific agent actions or decisions to human reviewers for approval before execution. This is important in agentic systems because LLMs are non-deterministic, they make mistakes so when the impact of agents being wrong is too high humans should validate their actions, responses or conclusions.    Integrated human-in-the-loop functionality is what makes it possible to deploy agents in high-stakes enterprise contexts with confidence.

5. Teamwork: Where the Real Value Emerges

The most transformative phase of the employee lifecycle is when individuals become part of high-performing teams — where collective capability exceeds the sum of its parts. For AI agents, this is where things get really interesting.

Agent Mesh supports two types of agent teamwork:

• Workflows let you define repeatable processes such as the steps involved in approving a loan or providing an insurance quote or opening an account.
• Dynamic agent orchestration lets you coordinate multiple agents in sequence or parallel, with orchestrator agents intelligently routing tasks to specialists based on context for situations where the input is more dynamic and reasoning is required to determine and execute a plan to respond to an input.

The result is a multi-agent mesh topology — hierarchical agent organizations coordinating specialist agents across various functions— that mirrors the structure of an effective human organization. And like an organization, the communication between Agent Mesh agents is asynchronous via events for better scalability, robustness and loose coupling.

6. Improvement: Deployment Is Not the Finish Line – it’s the Starting Line

Effective organizations don’t deploy employees and forget about them. They monitor performance, provide feedback, and create mechanisms for continuous improvement.

Agent Mesh’s visualizer gives you a real-time graphical interface for tracing agent interactions, tool and LLM calls, and decision pathways — so you can see exactly what your agents are doing and why. Ongoing evaluations detect performance drift or emerging failure modes via online evals that run in the background monitoring production execution. And OpenTelemetry instrumentation surfaces performance trends over time, giving you the data to make informed decisions about when and how to tune your agents.

Beyond ADLC: Other Advantages of Solace Agent Mesh

Beyond the lifecycle framework, there are a few things worth calling out about how Agent Mesh is built.

• It scales without chaos. Individual agents are straightforward. Dozens or hundreds of coordinated agents across business functions are a different beast entirely. The combination of hierarchy, specialization, dynamic agent discovery, async communication, and role-based access provides structural foundations for AI deployments that grow without falling apart.
• It builds trust through governance. Enterprise agentic AI production deployments stall not because of technical limitations but because leaders can’t be confident agents will behave predictably. Guardrails at hiring, RBAC at onboarding, human-in-the-loop supervision, and continuous monitoring and improvement are designed to build that trust incrementally.
• It avoids lock-in. Agent Mesh is LLM-agnostic, cloud-agnostic, and can work with 3^rd party agents via the Agent-to-Agent protocol. You can swap models, swap hyperscalers, hot-swap agents, and integrate new tools without rearchitecting your agent If you’ve already invested in LangGraph, AWS Bedrock AgentCore, or Azure AI, Agent Mesh complements those investments rather than replacing them — providing the event backbone, orchestration layer, and enterprise scalability and reliability they lack on their own. As the capabilities and price of models changes dramatically, where you want to deploy agents (typically near your data) can vary and where you want no lockin for the future –Agent Mesh gives you this flexibility.
• Event-driven is the native architecture. Solace’s core expertise is real-time data and microservices, steeped in two decades of experience as the leading enabler of event-driven architecture — the same pattern that powers many of the world’s leading enterprises. Applying EDA to multi-agent AI isn’t a retrofit; it’s a natural extension. Agents communicating via asynchronous, decoupled events are inherently more resilient, more scalable, and better suited to the messy reality of enterprise workflows than anything built on synchronous API calls. These are the same lessons leared with traditional microservices that became event driven for all the same reasons. Agent Mesh is the only agentic AI platform that’s “event-driven” from the ground up.

The Bottom Line: Agent Mesh Makes it Easy to Implement ADLC

The parallel between human workers and AI agents isn’t just a convenient metaphor. It’s a design principle. Organizations have spent decades — centuries, really — figuring out how to hire, train, supervise, and organize human talent into high-performing teams. That institutional wisdom doesn’t become irrelevant just because the workers are made of software instead of carbon.

Solace Agent Mesh is how that wisdom gets applied to AI agents. It’s an agent development and runtime platform that lets you implement the ADLC end-to-end, at enterprise scale. The result is a platform that doesn’t just deploy agents — it manages them through their entire lifecycle, from creation to continuous improvement to full multi-agent orchestration.

If AI agents are truly your new teammates, they deserve infrastructure that makes them great ones.

To see a demo of Solace Agent Mesh, visit https://solace.com/agent-mesh-demo/

You’ve built an agent mesh…tasks flow in, agents collaborate, results come back. It works! But when a response takes 12 seconds instead of 2, where did the time go? When an agent produces an unexpected answer, what prompt did it receive? When token costs spike on a Tuesday afternoon, which agent is responsible? These are observability questions. And they are harder to answer in an agent mesh than in a traditional microservices architecture.

Observability is Tricky in Agentic Systems

In a microservices system, you know the call graph in advance: Service A calls Service B and C, every time, in a predictable pattern. You instrument the paths, set up dashboards, and monitor. In an agent mesh, the execution graph is not known until it happens. An orchestrator delegates to specialist agents it discovers at runtime. LLMs are called with unpredictable latency. Tools are invoked based on reasoning, not routing rules. The number of steps, which agents participate, and how long each step takes can vary dramatically from one request to the next.

How Agent Mesh Offers Rich Observability

Solace Agent Mesh addresses this with observability that works at two levels: standard infrastructure-grade tools for production monitoring, and an agent-aware visual system purpose-built for debugging multi-agent workflows.

This post covers what Solace Agent Mesh provides across the three pillars of observability: logs, metrics, and tracing, and introduces the “dig as deep as you want” philosophy that ties them together.

Logs: The Familiar Pillar

Logging in Agent Mesh works exactly the way you would expect. It uses Python’s built-in module to provide flexible and powerful logging capabilities. You can configure it with the same YAML or JSON files you use for any Python application.

Three formatters are available out of the box:

• Simple: human-readable output with timestamp, level, thread, logger name, and message. Good for development.
• Colored: color-coded terminal output for quick visual scanning during local development.
• JSON: structured output via python Json logger for production environments. Feeds directly into Datadog, Splunk, Elasticsearch, or any log aggregation system that consumes structured JSON.

The JSON formatter supports static fields for adding context like service name and environment to every log line:

formatters:
  jsonFormatter:
    class: pythonjsonlogger.json.JsonFormatter
    format: "%(asctime)s %(name)s %(levelname)s %(message)s"
    static_fields:
      service: ${SERVICE_NAME, payment-service}
      environment: ${ENV, "production"}

Agent Mesh uses three logger namespaces: solace_ai_connector for the runtime layer, solace_agent_mesh for the framework, and sam_trace for detailed troubleshooting. Each can be configured independently, so you can run the framework at INFO level while setting a specific agent’s logger to DEBUG.

For file output, a rotating file handler ships by default with 50 MB rotation and 10 backup files. Environment variable substitution works throughout the config, so you can change log levels per deployment without modifying files.

The takeaway: logging is the one pillar that requires no learning curve. Plug it into your existing log aggregation stack and it works.

Metrics: LLM-Aware, Built into the WebUI

This is where Agent Mesh goes beyond traditional observability. Standard APM metrics track request latency, error rates, and throughput. Those matter in an agent mesh too, but they are not enough. You also need to know how much time the LLM spent thinking, how many tokens each request consumed, and what it costs.

Solace Agent Mesh tracks these metrics per task, automatically, with no external infrastructure required.

Stimulus Files for Analysis and Debugging

Every stimulus file (.stim) captures timing data for the full task lifecycle. Because stimulus files are structured YAML, you can parse them programmatically to extract latency distributions, token counts, and cost data across tasks. Teams that need to build dashboards or track trends over time can download the file (via REST API) or CLI and write custom scripts to extract timing metrics and feed them to existing analytics or visualization tools.

Token Usage Tracking

Every task records detailed token consumption: total input tokens, output tokens, cached input tokens, and a full breakdown by model and by source. This data is stored in the task database and visible directly in the WebUI. This data is stored in the task database and visible directly in the WebUI. For offline analysis, token usage details may also appear within the event payloads of downloaded .stim files when agents include this information in their A2A messages

This means you can answer questions like: how many tokens did this task consume? Which model was used? How much of the input was served from the cache? All from the WebUI, without setting up a separate metrics pipeline.

 

Tracing: Visual, Agent-Aware, Purpose-Built

Traditional distributed tracing follows a request through services using spans and trace IDs. That model works well for known call graphs. In an agent mesh, the call graph is dynamic. An orchestrator might delegate to two agents or five, depending on the question. Each agent might call an LLM zero, one, or multiple times. Tools are invoked based on the LLM’s reasoning, not predefined routes.

Currently, Solace Agent Mesh community edition does not use OpenTelemetry spans for tracing. Instead, it captures the full stimulus lifecycle by recording A2A messages as they flow through the Solace event broker. These events are stored as .stim files and can be visualized in the WebUI’s activity viewer, which reconstructs multi-agent interactions into flow diagrams. Because the system is event-driven and all communication passes through the broker, capturing the complete execution path requires no additional instrumentation.

For enterprise deployments Solace Agent Mesh will offer native OpenTelemetry integration with a Prometheus-compatible /metrics endpoint. This upcoming capability will enable direct ingestion of agent health, performance latencies, and LLM usage metrics into your existing observability stack-Datadog, Grafana, Dynatrace, or any OTEL-compatible platform.

The Activity Viewer

The centerpiece is the activity viewer in the WebUI. When you send a task to the agent mesh, the Activity view shows you the live execution as an interactive node graph.

Each step in the execution is a node. Common node types include:

• User nodes mark the start and end of stimulus. The top node shows the input as it enters through the entrypoint, and the bottom node shows the final response delivered back.
• Agent nodes represent each agent that participated, with nesting for peer delegations.
• LLM nodes show individual LLM calls with the model’s name.
• Tool nodes show tool invocations with artifact counts.
• Control flow nodes show conditional branches (Switch), iteration loops (Loop), and parallel fan-out (Map).
• Workflow nodes show structured multi-step workflows and parallel execution blocks.

Additional node types may appear depending on your configuration.

You can toggle between a collapsed overview and full detail mode that shows nested agent internals. Click any node to inspect the actual data: the LLM prompt, the tool arguments, the artifact content, the raw response. Processing indicators show animated halos on nodes that are still in progress.

Real-Time Activity Streaming

For live debugging, the WebUI provides a real-time SSE feed of A2A messages as they flow through your agent mesh. This is powered by the Solace event broker at the heart of Agent Mesh, which uses a hierarchical topic structure for all agent-to-agent communication. Every interaction-requests, responses, status updates-is published to structured topics following patterns like:

Purpose	Topic Pattern
Task Requests	{namespace}/a2a/v1/agent/request/{agent_name}
Status Updates	{namespace}/a2a/v1/gateway/status/{gateway_id}/{task_id}
Final Responses	{namespace}/a2a/v1/gateway/response/{gateway_id}/{task_id}

The WebUI’s activity viewer taps into this live message flow, displaying each event as it occurs. For broader monitoring, you can subscribe directly to the Solace event broker using wildcard topics like {namespace}/a2a/v1/> via tools such as the Solace Try Me VSCode Extension, Solace Try-me CLI or the Broker Manager’s Try Me! tab. This lets you scope the stream to all messages in a namespace, messages to a specific agent, or any slice you need—all through topic filtering, with no custom instrumentation required.

 

This is the power of event-driven architecture and publish-subscribe messaging: the broker already carries every interaction, and Solace’s topic-based routing gives you precisely the visibility you need.

Performance Visualization

The WebUI computes performance details from the activity data: per-agent LLM call durations, tool call durations, parallel execution grouping, and total task duration. This helps you identify which agents and which models are contributing to the most latency.

Dig As Deep as You Want

The philosophy behind Solace Agent Mesh’s observability is layered depth. You start with the simplest view and drill down only when you need to.

• Surface. You ask your agent mesh a question and get an answer in 3 seconds. Everything looks fine. Move on.
• One level down. A response takes 12 seconds. Open the Activity view. You see the Orchestrator delegated to two specialist agents. One returned in 200 milliseconds. The other took 11 seconds. Now you know where to look.
• Deeper. Click the slow agent’s LLM node. You see the prompt, the model (GPT-4), and the token count (8,200 input tokens). The prompt is bloated. Maybe the context window is carrying too much history.
• Deeper still. Check the performance report. This agent’s LLM call durations have been climbing. Compare token counts across recent tasks. The context window is growing with each session. This is not a one-off. Something changed.
• Deepest. Download the .stim file. Every A2A message in chronological order. Trace exactly where time was spent. Compare it to a stimulus file from last week when latency was normal. Share both files with your team.

You do not need to set up observability infrastructure to start debugging. The visual tools are built in from the first solace agent mesh run. Token tracking, performance reports, activity visualization, and stimulus files all work out of the box.

What This Means for Your Agent Mesh

Observability in an agent mesh is not the same problem as observability in a microservices architecture. The execution graph is dynamic. LLMs introduce a new class of latency. Token costs are a first-class concern. Traditional tools cover part of the picture, but they miss the agent-specific dimensions.

Solace Agent Mesh’s approach is practical: logs work the way you expect; metrics give you per-task token usage and LLM latency data built into the WebUI, and tracing is reimagined as a visual, interactive experience built for the way agent meshes execute.

In upcoming posts, I’ll dive deeper into each pillar: configuring structured logging for production, working with token usage data and performance reports, and using the activity viewer and stimulus files to debug complex multi-agent workflows.

Most enterprises are building AI agents.
Few are able to run them in production at scale.

The “Production Gap” in Agentic AI

The gap between building agents and running them is where most agentic AI initiatives stall out these days. Gartner forecasts that more than 40% of agentic AI projects will be canceled by end of 2027. MIT research found that 95% of generative AI pilots have produced no measurable return on an estimated $30-40 billion in spend. The diagnosis was not that the models are failing. It was that organizations lack the integration, feedback discipline, and operational structure to move agents into production.

Building an agent that works in a notebook is a solved problem. Building one that works reliably in production, connected to real enterprise systems, coordinated with other agents, governed, observable, continuously improving, and providing business value is a different discipline entirely; one that has come to be called the agent development lifecycle, or ADLC.

Managing Agent Lifecycles with Solace Agent Mesh

Solace Agent Mesh is a complete platform for building, testing, deploying, and operating AI agents in production. It covers the full ADLC across six stages: hiring an agent by defining its role, onboarding it to enterprise systems and data, coaching it through structured evaluation, supervising it with appropriate human oversight, enabling teamwork through multi-agent coordination, and continuously improving its performance over time. Agent Mesh includes the operational infrastructure that makes all six stages work in real enterprise environments.

Agent Mesh is more than an agentic AI framework. The distinction matters because agents require a platform for orchestrating them within workflows, governing how they participate, routing events between them, and managing them across the full ADLC. Those agents can be built on LangGraph, AWS Bedrock, custom code, or Agent Mesh’s own primitives.

Agent Mesh is built on event-driven architecture so it can handle both deterministic workflows and dynamic orchestration. That means it supports structured, repeatable processes as well as agents that reason and adapt at runtime. Agents communicate asynchronously through a real-time data distribution layer rather than synchronous API calls, which makes the system inherently more resilient, scalable, and capable of operating across clouds, data centers, and edge environments simultaneously.

It is also deliberately open. Agent Mesh supports the Model Context Protocol (MCP) and Agent2Agent (A2A), works with any large language model (LLM), and runs on any cloud. That openness reflects a conviction that enterprises will build agents using many different tools and frameworks, and that the platform connecting them should never become a lock-in risk. It is also an acknowledgement that enterprises will use agents from multiple vendors and built with multiple technologies and that enterprises expect them to work together. That is where LangGraph comes into the picture

Beyond LangGraph’s Workflow Frameworks

LangGraph is LangChain’s graph-based agent framework. It lets teams define agent workflows as explicit graphs, where nodes represent agents or processing steps and edges define the flow between them. It handles state management within the workflow, supports deterministic sequences, and gives developers precise control over how a reasoning process unfolds step by step.

For teams building tightly -controlled reasoning workflows within a single process, it works well. But getting these capabilities right is the starting point, not the finish line. Production enterprise systems demand a platform that also delivers:

• Discoverability in a Distributed Environment: Agent Mesh manages agents as independent participants in a distributed mesh, where they can be discovered at runtime and coordinate through a shared messaging fabric. It also offers deterministic workflows, quite like LangGraph, which can be sequenced through DAG-based graph relationships.
• Enterprise Reach: Agent Mesh agents can be triggered by a wide range of inputs generated by your enterprise: API calls, webhooks, emails, user interactions through web or Slack interfaces, messages from operational systems, and real-time events like transactions, alerts, and sensor readings. Agent Mesh provides a built-in entrypoint framework that handles all these input types, with authentication, authorization, and session management built in as a common layer across all of them. LangGraph agents can be wired to any of these inputs as well (usually via third-party plug-ins), but the security layer, and the governance must be built and maintained by the team. With Agent Mesh, that infrastructure is part of the platform and LangGraph agents can enjoy the benefits too.
• Lifecycle Management: Building an agent –with LangGraph or any other tool –is one step. Managing it from definition through production and continuous improvement is the harder problem. Agent Mesh operationalizes the full ADLC), i.e. defining an agent’s role and guardrails, onboarding it to enterprise systems, evaluating it before and after deployment, supervising it with calibrated human oversight, coordinating it with other agents, and improving its performance through observability and feedback.

The Need for Real-Time Data in Production

When microservices first emerged, teams connected them with synchronous REST calls between services. It worked in development, but in production, at scale, the tightly coupled nature of those connections became a liability. Slow services became bottlenecks. Restarts cascaded into failures across the system.

The industry learned over time that distributed systems need to be asynchronous and decoupled to be resilient and scalable, which is why event-driven architecture became the standard for production microservices.

Agentic AI is following the same path. Agents that work beautifully in test environments, chained together with synchronous calls, hit the same wall in production that microservices did. One slow LLM call blocks the next agent in the chain. One process restart loses everything in flight. One new agent requires changes to every component that needs to talk to it. These are not edge cases. They are the predictable consequences of building distributed systems without an asynchronous communication layer.

Agent Mesh is built with this reality as a first principle, not an afterthought. Agents communicate through a persistent messaging layer rather than direct function calls. Tasks survive process restarts because they are persisted in the broker, not held in memory. Agents run in parallel rather than sequentially, which means a slow or unavailable agent does not block the rest of the system. And because agents communicate through a shared event fabric rather than point-to-point connections, the system scales without rearchitecting every time a new agent is added.

This is also what makes Agent Mesh capable of connecting to the rest of the enterprise in real time. Real business systems, including transaction platforms, IoT sensors, CRM systems, and operational alerts, already communicate through events. An agentic AI framework that speaks the same language as those systems does not need a custom integration layer to connect to them. Agent Mesh participates in the event mesh natively, which is why it can respond to real business events in real time rather than waiting to be invoked by code.

How Agent Mesh and LangChain Work Together

One of Agent Mesh’s core design principles is that enterprises should never be forced to standardize on a single agent framework. The reality of enterprise AI deployment is that different teams, partners, and vendors will build agents using different tools, and the platform connecting them should never become a lock-in risk.

Since Agent Mesh supports MCP and A2A, agents built on other frameworks can participate in the agent mesh without requiring changes to how they were built.

Some enterprises will already have LangGraph agents in pockets of their environment. For those teams looking to move toward a production-grade agentic platform without throwing away that initial work, Agent Mesh provides two integration paths that enable existing LangGraph agents to participate in the mesh while the broader architecture matures.

Path 1: A2A Proxy with LangGraph Agent

LangGraph agents can be exposed as A2A-compliant endpoints in several ways: via a hosted deployment to LangSmith, a package built with LangGraph Build, or a self-hosted graph wrapped behind a FastAPI interface. Once that endpoint exists, point the Agent Mesh A2A proxy at it. The proxy registers the agent with the mesh, and Agent Mesh routes tasks to it like any other agent. The graph itself stays untouched.

The trade-off is that the agent is reachable through Agent Mesh but sits behind an HTTP bridge rather than running as a native event-driven participant. For agents you do not own or cannot redeploy, this is usually the best you can do, and it works well.

Path 2: Deploy the agent with a Solace-native A2A wrapper

The cleaner approach is to give the LangGraph agent its own A2A interface, so it joins Solace Agent Mesh the same way any native Agent Mesh agent does. Solace built and maintains the LangGraph A2A Executor for exactly this purpose. It is a lightweight runtime that wraps your existing LangGraph agent without modifying the graph itself and runs it as a native A2A participant on the mesh.

https://github.com/SolaceLabs/solace-agent-mesh-langgraph-exec?tab=readme-ov-file#readme

Once running, the LangGraph agent participates on the mesh natively over A2A, with agent card discovery and conversation continuity handled by the executor. The current release focuses on getting LangGraph agents into the mesh as first-class A2A participants. Deeper integration with SAM’s broader platform services, including durable queueing, richer authentication, and multi-modal content support, continues to evolve.

Value When Expanding Your Agent Ecosystem

As the agent economy matures, enterprises will increasingly consume agent capabilities from partners and vendors who made their own framework choices:

• Your logistics partner built their shipment tracking agent on LangGraph
• Your financial data provider’s risk assessment agent runs on AWS Bedrock
• Your ERP vendor shipped a procurement agent on their own proprietary stack

SAM connects all of them through the same event fabric, governs how they participate in your workflows, and maintains the observability and reliability your production environment requires.

Conclusion: The Framework Is the Starting Point, Not the Finish Line

Many organizations already have LangChain or LangGraph agents in their environment. Agent Mesh does not ask you to start over – it runs those agents within a production-grade platform that solves for the things the frameworks leave to you: deployment across distributed environments, runtime discoverability, enterprise governance, and full lifecycle management. Any of your LangGraph investment works harder inside Agent Mesh.

If you are starting fresh, you now understand the scope of your choices. LangGraph coordinates agents within a graph the developer defines in advance, with structural relationships set at design time. Agent Mesh is built for the realities that come after: agents connected to live enterprise systems, triggered by real business inputs, governed via enterprise security controls and session management, a distributed environment, and management through every stage of their lifecycle from definition through continuous improvement.

For organizations serious about putting AI agents into production at enterprise scale, the question is not which workflow framework to start with. It is which platform will help you get the most out of every agent you build or bring in.

I hope I’ve made the case that that’s Agent Mesh.

Financial services firms face a challenging technical puzzle: how do you reliably stream account updates, transaction notifications, and balance changes to millions of retail banking customers across web portals and mobile applications without drowning in infrastructure complexity? Major banks have been solving this problem with Solace for years. When your front-end applications need to display live account balances, payment confirmations, and fraud alerts to customers, the underlying streaming infrastructure becomes mission-critical. Understanding why requires examining the limitations of existing approaches and how purpose-built solutions address them.

The Kafka Gap in Retail Banking Architectures

Besides leveraging Solace to power their global event mesh, many retail banks have already invested heavily in Apache Kafka for backend event streaming. Kafka excels at analytics pipelines and event sourcing, but it doesn’t work for front-end streaming, because the Kafka protocol does not support WebSockets.

If you’re an architect, this leaves you with a difficult choice: build your own scalable, resilient, and secure WebSocket layer on top of Kafka, or choose a purpose-built solution. Building custom WebSocket infrastructure means taking on significant engineering effort for connection management, protocol translation, security, and horizontal scaling. You will need to solve slow consumer handling, implement proper authentication flows, and ensure the entire tier can handle millions of concurrent connections without becoming a bottleneck.

Fortunately, Solace offers an easy, proven solution. Solace does support WebSockets, and integrates with Kafka infrastructure via a built-in bridge. And many of the world’s leading financial institutions already run and trust Solace for enterprise-grade security, and proven scalability. This complementary approach preserves your Kafka investment while extending its reach to customer-facing applications.

Bridging the Protocol Divide

The protocol mismatch between front-end and back-end systems illustrates why this integration matters. Modern front-end components typically use JavaScript and WebSockets for displaying live data, while your backend systems publish account events using Java, Python, or C++ with enterprise messaging protocols. This mismatch creates complexity that compounds as you scale.

An event mesh built with Solace Platform handles protocol translation natively, without requiring additional proxies or plugins. Backend applications distribute events using Solace’s high-performance SMF protocol while HTML5 web apps and mobile applications can subscribe via MQTT over WebSockets. The broker handles translation seamlessly, keeping your architecture clean and your operational burden low. This native multi-protocol support becomes even more valuable when you consider the scale retail banks must achieve.

Hierarchical Topics and Wildcard Filtering

Solace’s hierarchical topic structure provides one of the most powerful capabilities for retail banking applications: the ability to organize events into logical, filterable namespaces that consumers can subscribe to using wildcards.

Topics in Solace follow a hierarchical format using forward slashes as delimiters, enabling you to encode rich context directly into the topic string. For a retail bank, a well-designed topic taxonomy might look like this:
retail/customer/{customerId}/account/{accountId}/transaction/completed
retail/customer/{customerId}/account/{accountId}/transaction/pending
retail/customer/{customerId}/account/{accountId}/balance/updated
retail/customer/{customerId}/account/{accountId}/alert/fraud
retail/customer/{customerId}/account/{accountId}/alert/lowbalance
retail/customer/{customerId}/notification/marketing
retail/customer/{customerId}/notification/security

With this structure, a single customer’s mobile banking app can subscribe to exactly the data it needs using wildcard expressions:

Topic Pattern	Description
retail/customer/12345/account/*/balance/updated	Stream balance updates for all accounts belonging to customer 12345
retail/customer/12345/account/67890/transaction/*	Receive all transaction events (completed and pending) for a specific account
retail/customer/12345/account/*/alert/>	Capture all alert types across all accounts using the multi-level wildcard
retail/customer/12345/>	Subscribe to everything for a customer with a single subscription

The single-level wildcard (*) matches exactly one level in the hierarchy, while the multi-level wildcard (>) matches one or more levels at the end of a topic. This flexibility means front-end applications can dynamically adjust their subscriptions based on which screen the customer is viewing. When a customer opens their checking account details, the app subscribes to that specific account’s transaction feed. When they return to the dashboard, it switches to a broader subscription covering all accounts.

This approach eliminates the need for client-side filtering logic or multiple specialized endpoints. Publishers simply emit events to fully-qualified topics, and the broker routes filtered stream(s) to the appropriate subscribers based on their wildcard patterns. The result is cleaner application code, reduced network overhead, and a more responsive customer experience.

Moreover, onboarding additional applications becomes incredibly straightforward. Users can subscribe to topics that correspond to events they are interested in, eliminating the need to wait for publishers to make changes. For instance, an analytics application responsible for detecting fraud can subscribe to all customer transaction events by subscribing to the following topic: retail/customer/*/account/*/transaction/>

Scaling to Millions of Customers

Capacity planning reveals one of Solace’s most compelling advantages: a single Solace software broker supports up to 200,000 concurrent WebSocket connections, meaning you can reach one million connections with just five broker instances. Solace brokers can easily be deployed in clusters to form an event mesh and scaled horizontally as more customers are on-boarded. This allows companies to start small and gradually expand their Solace deployments with business.

For a retail bank with millions of active digital banking users, this small footprint matters enormously. Custom-built WebSocket tiers or generic message brokers often require dozens of nodes to achieve similar scale. Fewer nodes translate directly to reduced infrastructure complexity, simpler upgrades, fewer failure domains, and lower total cost of ownership. However, raw scalability alone does not satisfy banking requirements. Security architecture must be equally robust.

Reference Architecture: Secure DMZ Deployment for External Clients

Retail banks require strict network segmentation between external customers and internal systems, and the recommended deployment topology addresses this directly. Solace brokers are placed in both the internal network and the DMZ, with external clients connecting via WebSockets to DMZ brokers that bridge to internal brokers.

This design enforces a critical security principle that the bridge connection is initiated by internal brokers, ensuring no inbound connections from external components into the internal network. External clients never directly connect to internal brokers, and all external traffic terminates at the DMZ with only trusted broker-to-broker communication crossing into internal systems.

This architecture also enables independent scaling. DMZ brokers scale horizontally based on WebSocket connections, while internal brokers scale independently based on message throughput since they handle aggregated traffic rather than individual customer sessions. With the infrastructure foundation established, attention can turn to optimizing the actual customer experience.

Optimizing the Customer Experience

Two specialized features deserve attention for fine-tuning customer experience at scale, each addressing distinct challenges that emerge when streaming data to millions of devices.

• Message Eliding addresses real-time data visualization challenges. A customer’s account balance might update frequently during active transaction periods, but the mobile app only needs the current state. Message eliding lets you define throttling at the client profile level, so a banking app configured for one update per second receives exactly that. Customer devices stay responsive, and mobile apps avoid unnecessary CPU, network, and battery usage.
• On-Behalf-Of (OBO) Subscriptions enable centralized subscription management essential at retail scale. Rather than exposing your topic taxonomy to browsers and mobile apps, a subscription manager integrates with your authorization and account management systems, then issues subscriptions on behalf of each WebSocket client session. This provides topic abstraction, dynamic subscription assignment, and a single audit point for compliance teams. These features work in concert with broader security controls to create a comprehensive authorization model.

Meeting Security and Compliance Requirements

ACL profiles complete the security picture by enabling fine-grained authorization tied directly to your topic hierarchy. A customer can be restricted to subscribing exclusively to updates for accounts they own, with no possibility of receiving another customer’s data.

Combined with OBO subscriptions and the DMZ deployment model described earlier, this approach fits cleanly with internal entitlement systems and regulatory requirements. Built-in slow consumer handling ensures problematic client connections will not destabilize healthy consumers or create back-pressure on publishers, meaning operations teams do not need to hand-tune complex buffering strategies.

See It in Action

The capabilities described here are not theoretical. Watch the demo video showing a six-broker deployment handling over one million concurrent WebSocket connections streaming live data. If you are evaluating real-time streaming options for retail banking portals or mobile banking apps, especially alongside an existing Kafka deployment, reach out to discuss how these capabilities map to your specific requirements.

You’ve built an agent. It seems to work. Someone on your team said “looks good” in a Slack thread. Are you ready to ship it? Not quite. The hard truth about AI agents is that seeming to work and actually working reliably are very different things. This post is about how you close that gap using the built-in evaluation framework in Solace Agent Mesh.

We’ll cover what evaluations are, why they matter, how the framework is structured, and walk through real examples from simple to complex. By the end you’ll be able to write your own test cases and run them against your agents locally.

This post assumes you already have a working Solace Agent Mesh environment. If you’re starting from scratch, check out the getting-started guide first.

What Are Evaluations?

An evaluation (or eval) is a structured, repeatable test of an AI system’s behavior against defined expectations. Think of it as the AI equivalent of an integration test: instead of asserting that a function returns true, you’re asserting that an agent did the right thing given a realistic input.

At its core, an eval answers: given this input, does the system behave the way I expect?

That might mean:

• Did the agent respond with a greeting when I said hello?
• Did it call the right tool to process a file?
• Did it delegate to the right peer agent to complete a task?
• Was the final response accurate and complete?

Why Evaluations Matter

AI agents fail differently from traditional software. There’s no stack trace when an agent gives a mediocre answer. No alarm fires when a model update quietly changes how your agent reasons about a task.

Here’s what you’re up against without evals:

• Silent degradation: model updates, tool changes, and config drift can all shift agent behavior without breaking anything in an obvious way
• Immeasurable variance: LLMs are probabilistic by nature the same prompt will produce different outputs across runs. Limited ability to distinguish acceptable variance from a dangerous one (different tool calls, wrong decisions, dropped steps).
• Multi-hop complexity: enterprise agents invoke tools, delegate to peer agents, and process artifacts; every hop is a potential failure point
• No shared baseline: without a repeatable test, “it works” means something different to every person on your team

Evals are how you move from vibes to evidence.

Workflow Evaluations

Solace Agent Mesh Workflows can also be evaluated in a similar manner. Keep an eye out for a deeper dive on workflows and workflow evaluations with Solace Agent Mesh

How Evaluations Work in Solace Agent Mesh

Solace Agent Mesh’s evaluation framework is built into the CLI and runs real requests through real agents on a real broker (not mocked simulations). That means the results actually reflect how your system behaves in production.

There are three building blocks:

• Test Case: A single JSON file describing one scenario. It includes the prompt, any file attachments (artifacts), which agent to target, and the criteria for a successful outcome.
• Test Suite: A JSON file that groups one or more test cases into a single run. It also defines the environment: which agents to start, which LLM models to use, broker connection details, and how many times to run each test.
• Evaluation Settings: A configuration block inside the test suite that specifies how to score results. There are three scoring methods, and you can use any combination of them:
  

  Method	How it works	Best for
  Tool Match	Checks whether the agent called the tools listed in expected_tools	Verifying correct tool usage
  Response Match	ROUGE score comparing the actual response to expected_response	Factual responses, extraction tasks
  LLM Evaluator	A separate LLM judges the full interaction against your criterion	Complex, holistic quality assessment

  

On ROUGE

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) measures word overlap between two pieces of text. It’s a common NLP metric, good for catching when agents produce the right words, but it won’t catch a correct-but-differently-worded answer. That’s where LLM Evaluator picks up the slack.

Walking Through Test Cases

Agent Mesh Vibe Coding

If you build with an AI assistant you can add the Solace Agent Mesh Context server to provide much-needed context to your process of building Solace Agent Mesh evaluations

Let’s look at three test cases of increasing complexity, all from the Solace Agent Mesh repo at tests/evaluation/test_cases/.

Example 1: Hello World

The simplest possible test. Does the agent respond to a greeting?

{
  "test_case_id": "hello_world",
  "category": "Content Generation",
  "description": "A simple test case to check the basic functionality of the system.",
  "query": "Hello, world!",
  "target_agent": "OrchestratorAgent",
  "wait_time": 30,
  "evaluation": {
    "expected_tools": [],
    "expected_response": "Hello! How can I help you today?",
    "criterion": "Evaluate if the agent provides a standard greeting."
  }
}

A few things worth noting here:

• target_agent must match the agent’s display name as configured in your Solace Agent Mesh setup
• expected_tools is empty, meaning this test doesn’t expect any tool calls, just a conversational response
• wait_time is 30 seconds, which is generous for a simple greeting but is fine as a baseline
• criterion is the plain-language instruction passed to the LLM Evaluator if you have it enabled

This kind of test is your canary. If it fails, something is fundamentally broken with your agent setup, and you want to know that before anything else.

Example 2: PDF to Markdown (Orchestration)

This test verifies that the orchestrator correctly delegates to a peer agent when asked to process a file.

{
  "test_case_id": "convert_pdf_to_md",
  "category": "Orchestration",
  "description": "A test case to convert a PDF file to markdown.",
  "target_agent": "OrchestratorAgent",
  "query": "Please convert the attached PDF file to markdown using the Markitdown Agent.",
  "artifacts": [
    {
      "type": "file",
      "path": "artifacts/sample.pdf"
    }
  ],
  "wait_time": 120,
  "evaluation": {
    "expected_tools": ["peer_MarkitdownAgent"],
    "expected_response": "I have converted the PDF file to markdown and attached it.",
    "criterion": "Evaluate if the agent successfully uses the MarkitdownAgent to convert the PDF file to a markdown file and confirms task completion."
  }
}

The jump in complexity here is meaningful:

• artifacts: a real PDF file is attached, just like a user would upload one in the UI. Paths are relative to the test suite config file location
• wait_time is now 120 seconds, reflecting that multi-agent orchestration takes longer
• expected_tools includes peer_MarkitdownAgent, which is how the orchestrator delegates to a peer agent. If it never calls that peer, the tool match score will be 0
• The criterion is more specific: it asks the LLM Evaluator to check for both the delegation and a confirmation of task completion

This test validates an entire agent-to-agent delegation path, not just a single response.

Example 3: CSV Filtering (Tool Usage)

This one tests whether the agent correctly uses a built-in tool to process structured data from an artifact.

{
  "test_case_id": "filter_csv_employees_by_age_and_country",
  "category": "Tool Usage",
  "description": "A test case to filter employees from a CSV file based on age and country.",
  "target_agent": "OrchestratorAgent",
  "query": "From the attached CSV, please list the names of all people who are older or equal to 30 and live in the USA.",
  "artifacts": [
    {
      "type": "file",
      "path": "artifacts/sample.csv"
    }
  ],
  "wait_time": 120,
  "evaluation": {
    "expected_tools": ["extract_content_from_artifact"],
    "expected_response": "The person who is 30 or older and lives in the USA is John Doe.",
    "criterion": "Evaluate if the agent correctly filters the CSV data."
  }
}

This test has a deterministic expected response. There’s one correct answer (John Doe), which makes both the Response Match and LLM Evaluator scores very meaningful here. It also checks that the agent uses extract_content_from_artifact rather than trying to invent a workaround.

Running a Test Suite

Individual test cases don’t run on their own. You group them into a Test Suite and run the whole thing with sam eval.

Here’s a local test suite that covers all three examples above (plus more), tested across multiple LLM models:

{
  "agents": [
    "examples/agents/a2a_agents_example.yaml",
    "examples/agents/a2a_multimodal_example.yaml",
    "examples/agents/orchestrator_example.yaml"
  ],
  "broker": {
    "SOLACE_BROKER_URL_VAR": "SOLACE_BROKER_URL",
    "SOLACE_BROKER_USERNAME_VAR": "SOLACE_BROKER_USERNAME",
    "SOLACE_BROKER_PASSWORD_VAR": "SOLACE_BROKER_PASSWORD",
    "SOLACE_BROKER_VPN_VAR": "SOLACE_BROKER_VPN"
  },
  "llm_models": [
    {
      "name": "azure-gpt-4o",
      "env": {
        "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/azure-gpt-4o",
        "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
        "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
      }
    },
    {
      "name": "gemini-3-pro-preview",
      "env": {
        "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-3-pro-preview",
        "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
        "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
      }
    }
  ],
  "results_dir_name": "my-eval-run",
  "runs": 3,
  "workers": 4,
  "test_cases": [
    "tests/evaluation/test_cases/hello_world.test.json",
    "tests/evaluation/test_cases/convert_pdf_to_md.test.json",
"tests/evaluation/test_cases/filter_csv_employees_by_age_and_country.test.json"
  ],
  "evaluation_settings": {
    "tool_match": { "enabled": true },
    "response_match": { "enabled": true },
    "llm_evaluator": {
      "enabled": true,
      "env": {
        "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-3-pro-preview",
        "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
        "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
      }
    }
  }
}

A few things worth calling out:

• agents lists the YAML configs for any agents the framework needs to start locally. This is what makes it a local eval run vs. a remote one
• runs: 3 runs each test case 3 times. Agent behavior has variance; a single run isn’t enough to draw conclusions
• workers: 4 runs tests in parallel to keep things fast
• The llm_evaluator uses a different model than the agents under test. This is intentional: your judge shouldn’t be the same model as the one being judged

To run it:

sam eval tests/evaluation/my_suite.json
# or with verbose output to see full message traces
sam eval tests/evaluation/my_suite.json --verbose

The framework starts your agents, submits each test case via the Solace broker, collects responses, scores everything, and writes results to the results/ directory. You get a per-test JSON summary and an HTML report with visual charts.

Local vs. Remote

The suite above runs agents locally, which is great for development and model comparisons. If you want to evaluate against an already-running Solace Agent Mesh deployment (staging, production), swap agents and llm_models for a remote block pointing at your instance’s REST gateway. The test cases themselves stay exactly the same.

Model Comparison Evals

This is where the Solace Agent Mesh eval framework earns its keep. Most eval tools test one model. Solace Agent Mesh runs your entire test suite against every model you specify in a single command, then renders a side-by-side comparison in the HTML report.

This matters because model choice is rarely a one-time decision. Models get updated, costs change, and new models ship that might outperform what you’re currently using for specific tasks.

To compare Claude Sonnet, Claude Opus, and Gemini, add them all to llm_models:

"llm_models": [
  {
    "name": "claude-sonnet-4-6",
    "env": {
      "LLM_SERVICE_PLANNING_MODEL_NAME": "anthropic/claude-sonnet-4-6",
      "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
      "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
    }
  },
  {
    "name": "claude-opus-4-6",
    "env": {
      "LLM_SERVICE_PLANNING_MODEL_NAME": "anthropic/claude-opus-4-6",
      "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
      "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
    }
  },
  {
    "name": "gemini-2.5-pro",
    "env": {
      "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-2.5-pro",
      "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
      "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
    }
  }
]

The framework runs the full suite once per model. The HTML report shows each model’s scores broken down by test category, so you can see, for example, that Opus edges out Sonnet on holistic quality scores for complex orchestration tasks, while Sonnet and Gemini are comparable on straightforward tool usage tests.

Some practical guidance for getting clean comparisons:

• Use at least 3 runs per model. One run is not a data point; it’s a coin flip
• Keep your LLM Evaluator model separate from the models under test. Pick a stable, strong model you’re not comparing (e.g. GPT-4o or Gemini 2.5 Pro) as the judge
• Organize test cases by category: the HTML report groups scores by category>/code>, so you'll get more actionable charts if your categories are meaningful
• Watch for variance, not just average scores. A model that scores 0.9 on average but has high variance between runs is less reliable than one that consistently scores 0.8

Reading Your Results

Running sam eval produces two things: a summary printed to the terminal when the run finishes, and a results/ directory with everything stored in detail.

What you get in the terminal

As soon as the run completes you’ll see a table like this, averaged across all runs per test case:

Model                     | Test Case                               | Tool Match | Response Match | LLM Eval
------------------------------------------------------------------------------------------------------------------
bedrock-claude-4-5-sonnet | filter_csv_employees_by_age_and_country | 0.00       | 0.03           | 0.00
bedrock-claude-4-5-sonnet | convert_pdf_to_md                       | 0.00       | 0.04           | 0.00
bedrock-claude-4-5-sonnet | hello_world                             | 1.00       | 0.00           | 0.00
gpt-4-1                   | convert_pdf_to_md                       | 1.00       | 0.25           | 1.00
gpt-4-1                   | hello_world                             | 1.00       | 0.53           | 0.67
gpt-4-1                   | filter_csv_employees_by_age_and_country | 0.00       | 0.34           | 0.30

This gives you a quick read on where things stand. For anything that looks wrong, the full detail is in the results/ directory.

What’s in the results directory

results/sam-local-eval-test/
├── report.html               # Visual HTML report
├── stats.json                # Aggregated scores for all models and test cases
├── gpt-4-1/
│   ├── results.json          # Per-run scores and LLM judge reasoning
│   ├── task_mappings.json
│   ├── hello_world/
│   │   ├── run_1/
│   │   ├── run_2/
│   │   └── run_3/
│   └── convert_pdf_to_md/
│       └── ...
└── bedrock-claude-4-5-sonnet/
└── ...

The HTML report is where you’ll spend most of your time. It shows benchmark run info, which models were tested, and LLM evaluation scores broken down by task category. Here’s what ours looked like:

The per-model results.json>/code> files go deeper: every run gets its own scores plus the full LLM judge reasoning as plain text. That reasoning is genuinely useful when a score surprises you.

Interpreting what the scores are telling you

The three scores serve different purposes and should be read together, not in isolation.

• Tool Match is binary and unforgiving. Either the agent called the expected tool or it didn’t. A score of 0.00 means the tool was never used across any run. For bedrock-claude-4-5-sonnet, both convert_pdf_to_md and filter_csv_employees_by_age_and_country scored 0.00 on tool match, which tells us immediately that the model failed to delegate to peer_MarkitdownAgent and failed to call extract_content_from_artifact. Something is broken at the instruction-following level for this model in this agent configuration.
• Response Match (ROUGE) is a weaker signal that can mislead. Notice that gpt-4-1 on hello_world scores 0.53 for response match but 0.67 for LLM eval. The agent responded with “Hello! How can I assist you today?” rather than “Hello! How can I help you today?” — the word “assist” vs “help” tanks the ROUGE score even though the response is perfectly correct. This is exactly why ROUGE alone is not enough: it measures word overlap, not meaning. Always cross-reference with LLM eval.
• LLM Evaluator gives you the most actionable signal, and the reasoning explains why. For gpt-4-1 on filter_csv_employees_by_age_and_country, the three runs scored 0.0, 0.6, and 0.3 respectively (average 0.30). The LLM judge’s reasoning on the 0.6 run explains it well:
  “The agent ultimately provides the correct answer (John Doe). However, the response is confusing and initially incorrect. The agent’s first sentence is factually wrong… it then contradicts itself…” That’s not a pass. The agent eventually got to the right answer but only after stating the wrong one. The LLM judge caught that nuance. ROUGE didn’t.

Low Response Match doesn’t always mean failure

A 0.25 response match on convert_pdf_to_md alongside a 1.00 LLM eval score is normal. The agent returned a confirmation message worded differently from the expected_response, but the LLM judge correctly recognized that it successfully completed the task. Set your expected_response to represent the spirit of the answer, and let LLM eval handle the quality judgment.

Using results to build confidence over time

A single eval run gives you a snapshot. Running evals consistently over time is what gives you confidence.

• After agent changes: any time you modify agent instructions, add a tool, or change how an agent delegates, re-run the suite before deploying. A drop in tool match scores is a reliable early warning that your instruction changes broke something in the agent’s reasoning. A drop in LLM eval scores points to response quality degradation.
• After model updates: models change under you. If your LLM provider pushes an update to a model you’re using in production, run your eval suite against it before cutting over. The bedrock-claude-4-5-sonnet results here are a good example of this: the model greets correctly (hello_world tool match 00) but completely fails at tool delegation and data processing. Without evals, you wouldn’t know that until a user complained.
• Tracking variance: the json file stores score distributions (min, Q1, median, Q3, max) for every test case, not just averages. If a test case scores 1.0, 1.0, 0.0 across three runs, the average is 0.67 but the 0.0 run is a real problem. Watch the distribution, not just the mean. A model that fails 1 in 3 times is not production-ready for that task.
• Building a baseline: run your suite when your agents are working well, and save those results as a baseline. Future runs can be compared against it. Scores drifting downward on tasks that used to pass consistently is a signal worth investigating, even if the average is still technically above your threshold.

What “good enough” looks like

For tool-dependent tasks, aim for tool match 1.00 consistently. For LLM eval, a score above 0.8 averaged across 3 runs is a reasonable production bar for most tasks, though you should set your own thresholds based on how critical each task is. For response match, treat anything below 0.5 as a prompt to check the LLM eval score before drawing conclusions.

What’s Next

Once you have a working suite, the natural next step is running it on a cadence: after every agent config change, after a model version update, or as part of your CI pipeline.

Solace Agent Mesh evaluations documentation covers all the configuration options in detail, including remote evaluation mode, artifact types, and full field references.

The example test cases shown in this post (hello_world, convert_pdf_to_md, filter_csv_employees_by_age_and_country, and the rest) are all available in the Solace Agent Mesh GitHub repo. Clone it, run the examples, and start adapting them for your own agents.

Start with hello_world – it’ll tell you more than you expect!

Your AI agent just gave a customer the wrong answer.

The model was accurate, but the data it was given was stale. This is the defining infrastructure problem of enterprise AI adoption, and it’s not getting enough attention. Teams spend months fine-tuning models, prompt-engineering system instructions, and evaluating LLM providers — then deploy agents that answer questions about accounts that updated yesterday, inventory levels that changed this morning, and orders that were cancelled an hour ago. The intelligence layer is fine—the data enrichment layer isn’t.

The root issue is architectural: AI agents are event consumers, not workflow triggers. They don’t initiate data flows — they need to be continuously informed by them. Enterprise AI deployments that ignore this distinction build enrichment layers optimized for the wrong model. Nearly every production agent will need live data from systems that weren’t designed to serve agents: Oracle ERP databases, IBM MQ queues, Salesforce records, SAP back-office systems, Snowflake warehouses, cloud event streams. The question isn’t whether you need real-time data enrichment for AI — it’s whether you’ll build it with the right tool for the job.

The iPaaS Assumption That Breaks for AI

Traditional integration platform as a service (iPaaS) was designed around a specific model: a human or business process initiates a workflow, data flows through transformation and routing steps, and a result lands somewhere useful. That model is valuable. It’s also the wrong model for how AI agents consume data.

AI agents don’t initiate workflows. They need to be continuously informed — subscribed to changes in the data that matters to them, receiving enrichment signals as those signals emerge, not when a human-initiated process happens to trigger them. An iPaaS workflow that fires when a sales rep submits a form is useful to that sales rep. It does nothing for an agent that needs to reason about current account state across every interaction, all day, without prompting.

The mismatch runs deeper than architecture. Consider what it actually costs to connect a new data source to an AI agent through a traditional iPaaS:

• A dedicated iPaaS environment or tenant, licensed at enterprise scale
• Workflow design, testing, and change management processes
• An integration team to build and maintain pipelines
• Governance overhead: versioning, approval chains, deployment windows
• Monitoring and on-call responsibility for every pipeline that feeds production agents

That overhead made sense when integrations served hundreds of human-initiated workflows. It’s economically punishing when you need to connect dozens of data sources to dozens of AI agents, and every new data source requires a new project, a new pipeline, and a new operational responsibility.

Micro-integrations: Purpose-Built Building Blocks

A micro-integration is a single-purpose connector process with one job: move data from one system to Solace Platform, or from Solace Platform to a target. No workflow engine. No approval chains. No visual designer. A developer can configure a new connector — a single YAML file with a handful of required fields — in an afternoon, without a visual canvas or a multi-step wizard. One config file, one JAR, one clear responsibility.

This narrowness is the feature. A micro-integration for Oracle CDC doesn’t need to know that you also have an IBM MQ integration running next to it. An agent subscribed to Salesforce change events doesn’t know or care that the same architecture is also streaming Microsoft SQL Server updates to a different agent. Each connector is independent, composable, and replaceable without touching anything else.

You still need an integration team. What changes is the size and nature of that team. When each connector has a single, readable YAML config and a clear runtime boundary, one engineer can own a dozen connectors without heroics. There’s no sprawling workflow graph to trace, no proprietary runtime to learn, no approval chain for a YAML key change. Micro-integrations don’t eliminate operational responsibility — they reduce the surface area per connector, which means the team that runs them can be meaningfully leaner.

The result is an integration model that scales horizontally the way AI agent deployments do — by adding focused components, not by expanding a centralized platform.

A Connector for Every Corner of the Enterprise

The breadth of what’s available matters as much as the architecture. A micro-integration ecosystem that covers only a handful of systems creates pressure to route everything else through iPaaS anyway, negating the simplification.

The current Solace micro-integration library spans the full enterprise data landscape across six categories:

• Database change data capture: Oracle CDC, IBM DB2, Microsoft SQL Server, MySQL, PostgreSQL, MongoDB — each streaming row-level changes the moment they commit, without polling.
• Enterprise messaging and applications: IBM MQ, TIBCO EMS, JMS, Salesforce, SAP ERP On-Premises, SAP S/4HANA — bridging the legacy broker estate and key business systems directly into modern event infrastructure.
• Cloud platforms and object storage: Amazon S3, Kinesis, SQS/SNS, MSK; Azure Blob Storage, Event Hubs, Service Bus, Cosmos DB; Google Cloud Storage, Pub/Sub.
• Kafka-compatible brokers: Apache Kafka, Confluent, Confluent Cloud, Aiven, Red Hat OpenShift Streams, Redpanda.
• Analytics and data platforms: Snowflake, Databricks, Microsoft Fabric, Apache Spark, KX Kdb+.
• Serverless and cloud functions: AWS Lambda, Azure Functions, Google Cloud Functions, Google Cloud Run. Plus OPC UA for industrial IoT.

An AI agent that needs to enrich a customer service case with data from Salesforce, SAP, and a PostgreSQL operational database isn’t looking at three separate iPaaS projects. It’s subscribing to three topics, each fed by a dedicated micro-integration running independently.

The Oracle CDC Example in Detail

Solace Micro-Integration for Oracle CDC illustrates the design philosophy concretely.

The micro-integration connects directly to Oracle’s redo log via LogMiner, reading committed changes as they occur — no polling, no periodic queries, no lag accumulating between check intervals. Configuration is a single YAML file: Oracle connection details, table name, Solace topic. Run the JAR. Oracle changes stream to consumers the moment they commit.

The design choice that eliminates a separate state store is that the CDC offset and schema history live in a last value queue (LVQ) on Solace Platform itself, making Solace Platform both the broker and the state store for change capture. For teams already running Solace Platform as core infrastructure, it’s a clean consolidation with no additional moving parts. For teams evaluating fresh, it’s a dependency to factor in — though one that pays off across every connector in the fleet, not just this one.

Every message from the Oracle CDC micro-integration carries structured metadata headers — cdc_table_name, cdc_operation, cdc_schema, cdc_scn — so AI agents can subscribe at the broker level to exactly the change events they care about, without parsing payloads. An agent that cares about account updates subscribes to oracle/accounts/update/v1. It doesn’t receive inserts. It doesn’t receive changes to tables it doesn’t need. This matters when LLM calls cost money and agent reasoning time is the scarce resource.

AI Agents Are Native Consumers of Event Streams

The deeper advantage isn’t operational simplicity — it’s that AI agents are architecturally better suited to event-driven data consumption than to request-driven pipelines.

An iPaaS workflow delivers data in response to a trigger: something happened upstream, so data moves. An AI agent enrichment model works differently: the agent is continuously subscribed to signals relevant to its domain, and those signals flow to it as they occur. The agent doesn’t poll. It doesn’t wait for a pipeline to fire. It accumulates context continuously, so that when it needs to act on a customer record or a support case, the enrichment data is already there.

Salesforce change events, SAP order status updates, Oracle account tier changes, Microsoft SQL Server inventory adjustments — each of these flows independently to whatever agents have subscribed. Add a new agent that cares about inventory? Subscribe it to the existing stream. The micro-integration that’s already running doesn’t change.

Don’t Rent Your Integrations…Own Them!

iPaaS platforms deliver breadth by abstracting away the underlying runtime — and that abstraction is the value, until it isn’t. When a connector doesn’t support a specific data type, a transformation requires logic the config layer can’t express, or an edge case falls outside what the platform anticipated, your options are limited: submit a feature request, wait for the next release, or pay for a professional services engagement.

Solace’s micro-integrations are open code built on Apache Camel and Spring Boot — two of the most widely deployed integration frameworks in the Java ecosystem, with active communities and independent roadmaps. When the standard Oracle CDC behavior needs adjusting for a specific downstream agent requirement, you modify the source. When a transformation needs logic that config can’t express, you write Camel routes. The full runtime is yours to extend, and the engineers who can do it aren’t rare.

Configuration ownership is where the practical gap with iPaaS is most pronounced. “Treat integrations like code” sounds like marketing copy until you’ve tried to diff a MuleSoft application export or review a Boomi process change in a pull request. iPaaS platforms serialize their artifacts as XML or proprietary binary formats — a moved shape, a renamed variable, a reordered step produces a diff that no human can meaningfully review. Rubber-stamp approvals in PR review aren’t a governance failure; they’re the only rational response to an unreadable artifact.

When a micro-integration configuration is a YAML file with human-readable keys and values, code review is actual code review. A change to the Oracle CDC batch size is one line in the diff. Rollback is a revert. Audit history is a git log. Our Micro-Integration Manager is built around a Git-backed config server by design — not as an aspiration, but because it’s the only model where configuration ownership means something.

The Right Tool for the Real-Time Enrichment Job: Micro-Integrations

I’m not saying micro-integrations can or should replace iPaaS everywhere. Complex orchestration, multi-system approval workflows, and business process automation are exactly what iPaaS platforms are built for. Use them there.

But for the specific problem of real-time data enrichment for AI agents — streaming current state from operational databases, enterprise applications, cloud platforms, and legacy brokers into an event infrastructure that agents can subscribe to — micro-integrations are a better way. Faster to deploy, cheaper to operate, and designed for how agents actually consume data. Micro-Integration Manager provides fleet-level visibility and operational tooling across all running micro-integrations, so the model scales beyond a handful of sources without losing observability. Adding a new data source is additive: one new config, one new JAR, no changes to anything already running.

The teams that build effective enterprise AI will be the ones who recognized early that agents are event consumers — and built their data enrichment layer to match. That means subscribing agents to live signals instead of polling pipelines, owning configuration as code instead of renting a workflow canvas, and scaling by adding connectors instead of expanding a centralized platform.

Explore the full micro-integration library at Solace Integration Hub

AI is everywhere right now. Fraud detection, risk analytics, customer service automation, software development — organizations across industries are actively deploying AI across the board. But there’s one category that’s quietly delivering some of the highest ROI with the fastest payback: AI Operations, commonly called AIOps.AIOps applies AI to IT operations by using intelligent agents to monitor infrastructure, diagnose issues, automate routine tasks, and surface insights that would otherwise require hours of manual investigation. The appeal of AIOps is simple: the problems are well-defined, the data is already there, and the inefficiencies being replaced are enormous. You don’t need to convince anyone there’s a problem. Every ops engineer already knows exactly how their day looks.

Managing a Large Messaging Estate Takes Real Effort

Large organizations commonly run substantial middleware estates. Event brokers route data between systems, and keeping them healthy at scale requires dedicated expertise. Solace brokers are extremely powerful, handling mission-critical data flows with high throughput and reliability, but with that power comes operational depth. Diagnosing an issue typically means checking the broker manager for current configuration, pulling historical metrics from Prometheus, searching Splunk for relevant logs, and cross-referencing Solace documentation alongside internal runbooks. That process can easily take an hour or two per ticket, and during a critical outage, every minute matters.

If you multiply that effort and “operational latency” across a team handling dozens of tickets a week, you’ll realize that a significant chunk of engineering capacity gets consumed by work that should be automatable. BAU requests, health checks, capacity reviews, and documentation lookups are the tasks that keep experienced middleware engineers stuck in reactive mode rather than focused on strategic initiatives.

What Solace Has Built

Solace has built a multi-agent AIOps solution on top of Solace Agent Mesh to address exactly this challenge. It’s available as part of a service pack called the Solace AIOps Accelerator.

Solace Agent Mesh is Solace’s open, vendor-neutral platform for building and deploying agentic AI systems at production scale. It provides the orchestration, observability, security, and connectivity infrastructure that agents need to work reliably together — without locking you into a specific LLM or cloud provider. It’s the foundation that makes building trustworthy multi-agent applications practical rather than experimental.

This AIOps solution is composed of several specialized agents, each focused on a specific operational domain:

• Issue Analyzer — the core diagnostic agent. It synthesizes current broker configuration, historical metrics, operational logs, institutional knowledge, and Solace documentation to produce a structured analysis with root cause assessment and recommended next steps. Every response includes detailed citations so engineers know exactly where the insights are coming from.
• Health Check Agent — generates comprehensive health check reports against your Solace brokers in minutes. This is work that Solace’s Professional Services team has traditionally done manually, requiring deep product expertise and significant time. The agent handles it automatically, surfacing configuration drift and best-practice violations without human intervention.
• Capacity and Trend Analysis Agent — monitors broker capacity and usage trends over time, generating reports that flag brokers approaching their limits before they become production incidents. The kind of analysis that prevents outages rather than simply responding to them.
• Prometheus, OpenSearch, JIRA, and RAG Agents — these work in the background to pull data from the monitoring and logging tools your team already relies on, and to surface relevant institutional knowledge at exactly the moment it’s needed.

One of the more compelling capabilities is the integration with ticketing platforms such as Atlassian JIRA. When a developer opens a support ticket, the solution automatically analyzes the issue and posts a comment with a root cause assessment and recommendations for the middleware team to review. This is human-in-the-loop by design. The ticket owner reviews an AI-generated response already enriched with current broker state, historical metrics, and institutional knowledge, then validates and acts. What used to take one to two hours now takes about five minutes.

The Features That Make an AI Ops Solution Production-Ready

Building a chatbot that can answer questions about Solace is the easy part. Building something an operations team can trust and rely on in production is a different challenge entirely. A few things make the difference.

• RAG and institutional knowledge. Every organization has knowledge locked in wiki pages and internal runbooks scattered across different systems. The solution ingests this institutional knowledge from multiple sources and makes it available to the AI agents at query time. This is what separates generic AI from something that actually understands your environment — your naming conventions, your custom configurations, your known issues, your escalation processes, and your SLAs across different environments.
• Citations. AI agents can and do hallucinate, which makes traceability non-negotiable. Every response from the Issue Analyzer includes citations pointing back to the specific documents, metrics, and configuration data that informed the analysis. Engineers don’t have to take the AI’s word for it. They can follow the reasoning, validate the sources, and build genuine trust in the outputs over time.
  

  Dashboard with Real-time Evals and Feedback on what’s lacking in each response.

• Continuous evaluation. Every AI response is automatically scored across multiple criteria: technical accuracy, completeness, relevance, clarity, and actionability. Prompts are categorized — greeting, informational, technical, analytical — and different scoring weights apply accordingly. This gives teams real visibility into how the system is performing and provides the data needed for continuous improvement, rather than relying on gut feel.
• Observability. Because the solution is built on Solace Agent Mesh, the underlying agent communication is event-driven and fully observable. You can see what agents are doing, trace how decisions are made, and monitor the system like any other production application.

See it in Action

Built to Fit Your Environment

No two Solace deployments look the same. The monitoring tools, the ticketing systems, the internal runbooks, the team structure — these vary significantly across organizations. This solution is designed to be customizable. Solace’s forward-deployed engineers work directly with customers to tailor the solution to their specific environment and requirements, integrating with the tools already in use, ingesting the institutional knowledge that matters most, and configuring agents to match how the team actually operates. The goal is demonstrable ROI early — not a six-month implementation before anyone sees value.

It’s All About The ROI

Most AI initiatives struggle with adoption because they solve problems that aren’t painful enough, or they require too much behavior change before delivering any value. AI Ops is different. The problems it targets — slow MTTR, repetitive BAU tasks, manual health checks — are real, well-understood, and easy to measure. That means ROI is trackable from day one. And beyond the efficiency gains, there’s an opportunity cost benefit that compounds over time: every hour an experienced middleware engineer isn’t writing up a ticket response or compiling a capacity report is an hour they can spend on work that actually requires their expertise.

We’ve had strong early interest from customers, and the feedback has been consistent: the depth of the analysis, the quality of the institutional knowledge integration, and the ticketing automation in particular have resonated with ops teams who have been waiting for something like this.

If you’re managing a large Solace deployment and want to see this in action, I’d be happy to walk you through a demo. Reach out — we’re actively working with customers to deploy this and would love to show you what’s possible.

Solace Agent Mesh is an open-source framework for building networks of modular, specialized AI agents. Agents communicate asynchronously via Solace Event Broker, share artifacts, and delegate tasks to each other through a standardized discovery protocol. Solace Agent Mesh handles the infrastructure, so you spend time on agent logic, not on wiring.At the heart of Solace Agent Mesh is the Orchestrator: an LLM-powered coordinator that receives tasks, reasons over the available agents in the mesh, and dynamically decides which agents to invoke and in what sequence. For exploratory, open-ended tasks: “analyze this dataset and give me a summary”, “draft a response to this customer email”, the Orchestrator is the right tool. It adapts, improvises, and handles ambiguity well. It is, by design, flexible. However, this same adaptability becomes a problem when we need the system to behave the same way every time.

The Challenges with Pure AI-orchestrated Flows in Production

The core issue is non-determinism. LLMs don’t execute a fixed function, they sample from a probability distribution. Temperature settings and prompt engineering can narrow the distribution, but they can’t eliminate it. For a conversational assistant, that’s acceptable. But it’s not for a process that must run the same steps in the same order every time.

There are additional failure modes that compound the problem:

• Observability gaps: When an orchestrator makes a wrong decision, e.g. assigning a task to the wrong agent, tracing why is difficult. The decision lived inside an LLM inference call. There is no explicit branch in code to point to, no line number, no deterministic condition that can be unit-tested.
• Brittleness under schema drift: If an upstream agent changes its output format slightly, a dynamic orchestrator may silently adapt by routing data to the wrong downstream agent without raising an error, because the LLM fills in the gap with a plausible but incorrect assumption.
• Compliance and audit requirements: Regulated industries require demonstrable proof that a process ran specific steps in a specific order. “The LLM decided the steps at runtime” is not an acceptable audit trail.
• Runaway costs: A poorly constrained orchestration loop where the LLM repeatedly re-delegates a task it cannot resolve will burn through tokens and time with no automatic circuit breaker.

None of this means the Orchestrator is the wrong approach. Outside the use cases it was designed for, prompt engineering cannot compensate for the absence of explicit control flow.

What Are Agent Mesh Workflows and How Do They Differ from the Orchestrator?

Solace Agent Mesh ships with two ways to coordinate agents: the Orchestrator and Workflows. They solve different problems.

• The Orchestrator is backed by a LLM and when it receives a task, it decides at runtime which agents to invoke, in what sequence, and how to assemble the result. It works well when the task is open-ended or the right path isn’t known in advance. The downside is that, and it isn’t guaranteed to be the same twice.
• A workflow is an explicit execution graph over your agents, code. You specify each step, the order they run, what data flows between them, and what happens when something goes wrong. The runtime follows your specification exactly: no inference, no interpretation.

Workflows move that control flow into configuration. The graph is explicit, version-controlled, and fully visible before the first message is sent.

The Orchestrator can invoke a workflow like any other agent in the mesh. Dynamic reasoning and deterministic execution aren’t mutually exclusive; they are to be used together to build the best suited solution.

A workflow is also not a step down in intelligence. The individual nodes in a workflow are still AI agents. They still reason, interpret, and generate. What’s constrained is the control flow and not the capability of the agents executing within it.

Why Production Agentic Systems Need Explicit Workflows

LLMs Are Probabilistic by Design

This isn’t a criticism of LLMs. It’s a statement of how they work. A language model doesn’t execute a deterministic function, it samples from a probability distribution over possible next tokens. Every inference call is a weighted roll of the dice, shaped by the model weights, the prompt, the context window, and a degree of inherent randomness that no temperature setting fully eliminates.

For a chat assistant, this is fine. For a system making routing decisions about live business processes, it means you cannot guarantee the same execution path for the same input. At low volumes that’s manageable. At production scale, even a 1% deviation rate becomes hundreds of failures per day which are difficult to trace and prove.

The Error Threshold Problem

Traditional software systems are built with a near-zero tolerance for logic errors in critical paths. The expectation is that the code executes the same logic, in the same order, every time.

Pure orchestrator-led agentic solutions operate at a fundamentally different error threshold. Even well-engineered LLM orchestration will exhibit:

• Reasoning drift: The model reasons slightly differently across calls, producing different delegation decisions for semantically equivalent inputs.
• Prompt sensitivity: Minor changes in upstream agent output phrasing can shift how the orchestrator interprets the situation and what it decides to do next.
• Model version sensitivity: A model update from your LLM provider can silently shift the orchestrator’s behaviour across your entire production fleet overnight.
• Context window effects: As conversation history or tool call results accumulate in the context, the model’s attention shifts, affecting decisions made later in a long chain.

None of these are bugs you can fix. They are properties of the system.

The Cost of a Wrong Step

The cost of a wrong execution significantly depends on the context and use case. For low-stakes, read-only tasks like summarizing a document, drafting a response for human review, the orchestrator’s flexibility is worth the trade-off. If it takes a slightly different path, the worst outcome is a suboptimal answer that a human catches.

For tasks with side effects, the equation is different:

• A workflow that triggers a payment, modifies a database record, or fires an external API call cannot be “mostly correct.”
• A compliance process that must execute specific checks in a specific order cannot rely on an LLM to remember that order reliably.
• An incident response pipeline that escalates based on severity cannot afford to occasionally skip the severity classification step because the orchestrator inferred it was unnecessary.

These are not edge cases. They are the operational reality of running agentic systems at production scale, in domains where outcomes matter.

Explicit Workflows Are the Engineering Answer

Explicit workflows don’t reduce agent intelligence; they constrain the structure around it. Which agent runs, on what data, in what order, and with what fallback if it fails: all that lives in configuration you can read and test before it touches production. This is how reliable systems have always been built.

The Workflow Execution Model: Node Types and Data Flow

A Solace Agent Mesh workflow is a list of nodes. Each node declares its dependencies via the property depends_on. Nodes with no unsatisfied dependencies run immediately in parallel if there are multiple.

Node Types

There are 5 node types in a workflow:

Node	What does it do
Agent	Calls a named agent, passes input, captures output
Switch	Evaluates conditions top-to-bottom, routes to the first match
Map	Runs a node once per item in an array, parallel by default
Loop	Repeats a node until a condition is false
Workflow	Calls another workflow as a sub-step

Passing Data Between Nodes

Data flows through template expressions using {{...}} syntax:

# From workflow input
input:
  order_id: "{{workflow.input.order_id}}"

# From a previous node's output
input:
  validated: "{{validate_order.output.result}}"

Two operators cover edge cases:

• coalesce: Returns the first non-null value from a list. Useful when a node is conditionally skipped.
• concat: Joins values into a string.

A Note on Parallelism

Parallelism in your workflows can be achieved by declaring the dependencies between different nodes accurately. Two nodes with no depends_on relationship between them will run concurrently without any extra configuration. The workflow engine resolves the execution order from the dependency graph at runtime.

Building Your First Workflow: Request for Quote

The Use Case

When a customer sends a request for quote (RFQ), a few things need to happen in the right order:

1. Validate the request
2. Check stock availability and lead times
3. Calculate pricing based on volume and customer tier
4. Route to auto approval or sales review based on total value or out of stock items
5. Generate and deliver the quote.

None of these steps are optional, and the sequence matters. This is a very simple but critical scenario which cannot be left to the LLM’s planning. If the LLM occasionally skips the stock availability check or auto-approves a high value quote, it will lead to operational losses.

The Agents

The workflow is composed of the following agents:

Agent	Responsibility
RFQValidator	Parses the incoming request, validates required fields, extracts line items
InventoryAgent	Checks stock availability and lead times for each line item
PricingAgent	Calculates price based on volume, margin rules, and customer tier
QuoteGeneratorAgent	Assembles and formats the final quote document
SalesEscalationAgent	Escalates to the sales team based on pre-defined conditions

The validator runs first. Once that clears, inventory and pricing checks run in parallel as they don’t depend on each other but only on the validated RFQ. The quote generator waits for both before assembling the output. In case pre-defined conditions like high value orders or out of stock items, the SalesEscalationAgent flags the request to the sales team for manual handling.

What This Demonstrates

A few things worth pointing out in this configuration:

• Parallelism is implicit: check_inventory and calculate_pricing both declare depends_on: [validate] but have no dependency on each other. The workflow engine runs them concurrently without any extra configuration.
• The switch node enforces business rules: The routing decision to auto-approve or escalate for manual approval is encoded in conditions you can read and test, not inferred by an LLM at runtime. In this example, escalation is triggered if the total value is above €50k, or any line item out of stock.
• coalesce handles the branching output: Since only one of generate_quote or escalate_to_sales will run, the output mapping uses coalesce to pick whichever produced a result.

Triggering Workflows

When a Solace Agent Mesh workflow starts, it registers itself as an agent on the event mesh and publishes an agent card the same way any other agent does. From that point on, anything that can invoke an agent can invoke the workflow. No special integration, no separate endpoint to wire up.

Workflows are agents

The RFQProcessingWorkflow from the previous section becomes discoverable by name the moment it’s running. It can be triggered directly from the WebUI chat gateway along with the Orchestrator and REST gateway. Another workflow can invoke it as a sub-step using a workflow node. The entry point is always the same: send a message to the workflow’s address on the broker with a payload matching its input_schema.

Event-Driven Invocation

Because Solace Agent Mesh distributes data in an event-driven manner via Solace Event Broker, workflow invocations are events. An upstream system can publish an event to the broker, and the workflow picks it up.

For the RFQ use case, this means a sales portal can publish a rfq-received event the moment a customer submits a request, and the workflow processes it without the portal knowing anything about Solace Agent Mesh’s internal structure.

Invoking from the Orchestrator

The Orchestrator discovers workflows through their agent cards, including the skills you define. If a user says “process this RFQ”, the Orchestrator can identify RFQProcessingWorkflow as the right tool and invoke it directly by passing the structured input it needs.

This is where the two components fit together. The Orchestrator handles the ambiguous front-end interactions like understanding intent, gathering missing fields, and confirming with the user. Once it has everything it needs, it hands off to the workflow, which executes the rest deterministically.

Vibe Coding a workflow

Writing a workflow from scratch is not quite complex, but getting the final version might take a few iterations. Solace Agent Mesh’s vibe coding support shortens this iterative process considerably.

Agent Mesh provides native support for vibe coding by connecting your coding assistant to the Agent Mesh codebase and documentation via a Context7 MCP integration.

Setting Up

1. Follow the MCP installation instructions for your IDE to connect your coding assistant to Context7 using the MCP server.
2. Verify the integration by asking the coding assistant:

Use the solacelabs/solace-agent-mesh context7 library when answering questions in this chat session.

Using Vibe Coding

1. Following our previous example, use the below prompt in your coding assistant:

   I want to create a new workflow based on a Request-For-Quote use case. The workflow should have the following agents:
        - RFQValidator : Parses and validates the incoming RFQ. Extracts line items, checks required fields, and determines the customer tier.                                                                              
        - InventoryAgent : Checks stock availability and lead times for each line item. Runs in parallel with PricingAgent after validation.                                                                                
        - PricingAgent : Calculates quote price based on line items, volume, and customer tier. Runs in parallel with InventoryAgent after validation.                                                                      
        - QuoteGeneratorAgent : Assembles the final quote document when auto-approval conditions are met.                                                                                                                   
        - SalesEscalationAgent : Handles RFQs that exceed the auto-approval threshold or have stock issues.                                                                                                                 
   
   The workflow : RFQProcessingWorkflow should orchestrate the full RFQ pipeline : validate → (inventory ∥ pricing) → route → generate_quote | escalate
   

2. The coding assistant generates the below draft for your review:
   
3. You can review and iterate through the configuration to make sure that it meets your expectations.

What to Watch For

While vibe coding handles structure well, it handles details less reliably. A few things to double check before running generated workflows:

• Schema fields and structure
• Switch conditions
• Dependencies
• Error handling

Start with the core flow, test it, then add-in error handling and retry once the happy path works. While vibe coding is most useful to get the structure right quickly, the nuances and fine tuning are still the critical parts for you to focus on.

Conclusion and Key Takeaways

• The Orchestrator and Workflows solve different problems. The Orchestrator is the right tool for open-ended, exploratory tasks where the path isn’t known upfront. Workflows are for processes where the steps, sequence, and decision rules are known and too critical to leave to runtime inference.
• LLM non-determinism is not a bug you can fix. Reasoning drift, prompt sensitivity, and model version changes; these are basic properties of how LLMs work. At production scale, even a small deviation rate compounds into operational failures.
• Explicit control flow belongs in configuration, not in an inference call. Solace Agent Mesh workflows define every step, branch, and dependency in YAML. The result is a process you can read, test, version-control, and audit before it is deployed to production.
• Five node types cover common patterns. Agent invokes a specialist, switches for conditional routing, map for parallel collection processing, loop for polling, and workflow for composing larger pipelines from smaller ones.
• Implicit Parallelism. Two nodes with no dependency between them run concurrently without any extra configuration. The execution order is resolved from the dependency graph at runtime.
• Workflows register as agents. Once running, a workflow is discoverable and invokable by name, by the Orchestrator, by a REST gateway, by another workflow, or by any upstream system publishing an event to the Solace broker. No special wiring is required.
• Error handling is first-class. Per-node retry strategies with exponential backoff, exit handlers for cleanup and alerting, configurable fail-fast behaviour, and hard timeout limits are all built into the configuration model.

Next Steps and Resources

• Documentations: Workflow Documentation
• Vibe coding: Vibe coding in Solace Agent Mesh
• Have a question? Ask in Community

• Star the Solace Agent Mesh Github Repository
• Try out the open-source Solace Agent Mesh: Open-source Solace Agent Mesh
• Contact us to schedule a demo: Schedule a Solace Agent Mesh Demo
• Ask to run a Solace Agent Mesh pilot: Run a pilot

This post is based on a system that was built and demonstrated in a recent episode of Solace Office Hours.

---

“Can I get a quote for this?”

Seems like such a simple question…a buyer specifies what they need, and a supplier or seller responds with availability, pricing, and a delivery window.

But generating that quote requires getting product details, checking stock levels in an ERP, calculating lead times, and pricing out shipping and other logistics costs. Each of these steps requires checking a different system (some internal, some external), with its own data contract, turnaround times and failure scenarios.

The traditional way to resolve this problem is sequential: a sales rep or some automation workflow can route a form, check multiple systems, send email requests and wait for vendor responses, assemble the results and send out a quote which may be stale by the time it is read.

On top of the process delays are situations like vendors replying with partial quotes, ambiguity in specs, unavailable supplier and missed deadlines. Many times, procurement or sales teams end up doing the coordination work manually, defeating the purpose of automation entirely.

Why Traditional Automation isn’t Enough

The first choice when automating an RFQ process is to choose familiar tools: a workflow engine to sequence the steps, API integrations to connect the systems. These approaches work well in bounded and predictable processes. Unfortunately, an RFQ process is neither of those.

Workflow Engines Create a Sequence that is not Needed

A typical RFQ process has the following steps :

• Check the Product Information Management (PIM) system and wait for the request to complete
• Check the inventory and pricing, waits again
• Then query shipping and wait again.

The three lookups are independent, but their execution is serial. On a slow ERP day, that step could drag on.

Data Living in too Many Places

Most large enterprises are inherently multi-system. And each system has its own “master” data. In an RFQ process, the initial requirement might come from an ERP, vendor communications happen over emails or portals, pricing rules are governed by personal relationships and spreadsheets, approval history lives in a ticketing system and finally compliance checks reference a separate vendor management database.

Error Handling Evolving into a special Case

Most workflow-based automations assume the happy path. When an API call fails, the ERP times out or, takes too long, retries or failures are the only mitigation options. Partial results, cached fallbacks etc. require conditional branching that makes the workflow definition unmaintainable in the long run.

Extending the Process Requires Modifying It

If a regulatory compliance check or a supplier credit scoring service needs to be added to the quote flow, a traditional integration script has to be extended and re-tested as a whole. There’s no mechanism for adding a new participant without touching the existing one.

Why RFQ Is a Strong Candidate for an Agentic Solution

Not every business process or automation workflow benefit from an agentic solution. The overhead of LLM inference, the non-determinism of agent reasoning, need for a high error threshold and the complexity of multi-agent coordination all carry real costs.

Before committing to an agentic solution, always ask this question: does this process really need AI and agents or is it a workflow problem dressed up as an AI problem.

The RFQ process has several characteristics that align clearly with how agentic solutions work:

Multi-Step Reasoning Across Ambiguous Inputs

An RFQ is not a single transaction. It is a sequence of dependent decisions, each of which may require interpreting context from previous steps. Parsing an incoming requirement, assessing vendor fit, handling partial or non-conforming responses, normalizing heterogeneous quote formats, and producing a ranked recommendation all require reasoning, not just routing.

The inputs are also structurally ambiguous. A buyer might submit a requirement in natural language. A vendor might reply by email with a PDF attachment that does not match the original or expected format. An approval stakeholder might ask a clarifying question mid-process. These are some edge cases that simply cannot be pre-coded for; they are the nature of the process itself.

Parallel and independent sub-tasks

A complete quote requires product information, live inventory availability, and shipping costs. Product catalog and inventory lookups are independent of each other and can run in parallel. Shipping rate calculation depends on both – product dimensions from the catalog and origin warehouse from the inventory check.
The result is a two-phase execution: independent tasks run concurrently, and the shipping lookup follows when both are complete. This is a significant latency improvement over running all the three sequentially.

Approvals from a human in the loop

Agentic pipelines can pause at defined checkpoints for a human to review and confirm before continuing. For high-value quotes, this is a feature rather than a workaround, and the pipeline handles it without needing a special execution mode.

The table below shows a high level comparison of agentic and traditional alternatives :

Approach	Parallelism	Error Handling	Extensibility
Manual or human coordinated	None	Ad-hoc	High Effort
BPM / workflow engines	Limited (branching)	Script-based	Moderate
Custom ETL / Integrations	None by default	Error queues	Pre-integration rework
Agentic	Native fan-out	Agent-level retry and fallbacks	Swap agents independently

The Agentic AI-Powered RFQ

The designed solution runs on Solace Agent Mesh, which provides both the agent framework and the event-driven messaging fabric.
Agents communicate with each other via the Solace Event Broker which means they are decoupled by design. An agent publishes to an event topic; the broker routes it; the receiving agent processes it asynchronously.

The agent topology uses two levels of orchestration which is a very conscious choice.

A single flat orchestrator with knowledge of product lookup, ERP queries, and shipping calculations would become a maintenance problem fast. The domain-specific logic would accumulate in one place.

By separating a dedicated RFQ sub-orchestrator, there is a clean boundary: the main orchestrator handles routing across business domains, and the RFQ sub-orchestrator owns the quote-assembly logic specifically.

Agent roles

Orchestrator

The main orchestrator is the system’s entry point. It receives incoming requests from the web UI or any other integration surface including events, identifies the appropriate downstream handler, and returns the final assembled response.
For RFQ requests, it delegates immediately to the RFQ sub-orchestrator. The main orchestrator doesn’t change when the RFQ workflow evolves.
This separation pays off in a multi-domain deployment. If the same Agent Mesh handles order management, inventory alerts, and RFQ workflows, the main orchestrator routes each to the right sub-orchestrator and without any of them knowing about each other.

RFQ Sub-Orchestrator

The RFQ sub-orchestrator owns the quote-assembly workflow. It receives the structured request from the main orchestrator, fans out to the specialist agents in parallel where applicable, collects their responses, and consolidates a complete quote to return upstream.
Business rules that span multiple data sources belong here: input validations, minimum order quantity checks, escalation flows amongst others.

 Product Information Management Agent

The PIM agent is a RAG-based agent that queries the Product Information Management (PIM) system. Given a product identifier or a natural-language description from the RFQ, it retrieves product specifications, upsell and cross selling opportunities, product dimensions. The agent handles embedding-based retrieval from the product catalog index and returns a structured response to the sub-orchestrator.

SAP Joule Agent

The SAP Joule agent is the most integration-intensive component in the system. Solace Agent Mesh connects to an SAP Joule agent via the A2A protocol, which in turn accesses SAP S/4HANA (or other SAP systems) for real operational data: Available-to-Promise (ATP) quantities, current stocking levels, warehouse location, and confirmed lead times.
The SAP Joule agent translates the sub-orchestrator’s request into an SAP Joule agent query, parses the structured response, and surfaces the fields the RFQ workflow needs.

Shipping Agent

The shipping agent connects to a shipping integrator, a service that aggregates rates and delivery windows across multiple carriers. Given an origin warehouse, a destination, a weight or volume estimate, and a required delivery date, it returns a list of carrier options.

It runs after the PIM and SAP Joule agents complete, because it depends on both: product weight and dimensions come from the PIM response, and the origin warehouse comes from the SAP Joule response. The sub-orchestrator assembles these inputs and dispatches them to the shipping agent as a single request once Phase 1 is done.
Shipping APIs are often the most variable in terms of response time, so the total quote latency is bounded by the slower of Phase 1 (PIM and SAP Joule in parallel) plus the shipping lookup.

How Agent Mesh connects it all

Individual agents are only as useful as the infrastructure that connects them. Solace Agent Mesh provides the event-driven backbone that makes multi-agent coordination reliable, observable, and scalable.

Agent discovery and routing

Agents in the mesh publish their capabilities as standard A2A cards. When inventory data is required, the sub-orchestrator identifies the SAP Joule agent based on its A2A Agent Card and  publishes a request to the SAP Joule agent which acts on it and sends a response back. Solace Agent Mesh allows all of the agents to be scaled, restarted, or swapped without reconfiguring the rest of the system.

Async-first by design

RFQ timelines are measured in hours and sometimes days, not milliseconds. Solace Agent Mesh is built for durable, asynchronous message delivery. An agent can publish a request and resume other work while waiting for a response. If the ERP is slow, the message waits in the queue rather than blocking a thread. This gives Solace Agent Mesh fault tolerance – if an agent crashes mid-process, the message is not lost and the agent will periodically retry.

Auditability

Every agent interaction is an event on the mesh. This means the full decision trail, which agent made which decision, when, and based on what input, is captured as a sequence of messages. Replaying the event stream for a specific RFQ gives you a complete audit log without any additional instrumentation.

Extensibility without disruption

Adding a new process, for example: an export and sanctions check agent on AWS Bedrock, means simply deploying a new agent and registering it with the Orchestrator.
No existing agent needs to be modified or redeployed. This is the practical realization of what the earlier section described as the limitation of traditional integration: in a mesh, you extend the process by adding participants, not by modifying the existing flow.

Key Takeaways

• RFQ is a structurally complex process, not a simple workflow. The multi-system, multi-party, exception-heavy nature of quote generation makes it a poor fit for rule-based automation and a strong fit for an agentic solution.
• Not every process needs agents. Before reaching for an agentic solution, validate that the process genuinely requires multi-step reasoning, handles ambiguous inputs, or has frequent unpredictable exceptions. RFQ passes this test on all three counts.
• Two-level orchestration keeps concerns separated. A main orchestrator handles cross-domain routing; the RFQ sub-orchestrator owns quote-assembly logic. Each agent can evolve independently, and this boundary is what makes the system maintainable at scale.
• Structured fan-out reduces latency without sacrificing accuracy.The total quote time is bounded by the slowest individual lookup, not the sum of all lookups in sequence.
• Human approval is a mandatory requirement. Agentic pipelines pause and resume cleanly at approval checkpoints. For high-value procurement decisions, this is a governance requirement, not a limitation to work around.

Appendix and Resources

• Solace Agent Mesh: https://solace.com/products/agent-mesh/
• Introduction to SAP Business Agent Foundation: https://community.sap.com/t5/technology-blog-posts-by-sap/ai-business-agents-and-the-evolution-of-business-automation-join-the/ba-p/13614232
• Solace Office Hours Episode: Solace Office Hours – “Jan” 2026 – Agent Mesh + SAP Joule

Next Steps

• Star the Solace Agent Mesh GitHub Repository
• Try out the open-source Solace Agent Mesh: Open-source Solace Agent Mesh
• Contact us to schedule a demo: Schedule a Solace Agent Mesh Demo
• Ask to run a Solace Agent Mesh pilot: Run a pilot

In the high-stakes world of financial services, the difference between success and missed opportunity often comes down to the speed at which data moves across systems, and microseconds matter.In this post, we’re excited to share groundbreaking results from a collaboration between AWS and Solace that demonstrates how by leveraging AWS Outposts racks with accelerated networking Amazon Elastic Compute Cloud (Amazon EC2) instances (bmn-sf2e) & Solace  Event Brokers, financial institutions can achieve unprecedented performance for their most demanding workloads.

The Importance of Latency and Performance in Capital Markets

In capital markets, latency and throughput are key metrics for trading systems that contributes to firm’s profitability. Whether it’s delivering market data to trading algorithms or routing orders to execution venues, the speed and predictability of data movement directly impact profitability and regulatory compliance. Firms in equities, fixed income, and derivatives trading all rely on lightning-fast dissemination of prices, quotes, and trades to maintain a competitive edge. Even microbursts of latency can result in missed opportunities, stale data, or increased exposure to market risk.

Modern trading infrastructures are built around market data distribution and order flow processing—both of which demand consistent, ultra-low-latency performance. Market data publishers must fan out millions of updates per second across trading systems, while order management systems must ensure every order, cancel, and acknowledgment moves across the network with microsecond precision.

Traditional data center deployments achieve this with purpose-built hardware and finely tuned networks, but as firms modernize, the challenge becomes: how do you retain that deterministic performance while embracing cloud agility and scale?

Solace Event Brokers on AWS Outposts Bare Metal: Bridging Low Latency and Cloud Agility

AWS Outposts racks with accelerated networking Amazon EC2 instances (bmn-sf2e) bring AWS services and infrastructure directly into customers’ data centers or colocation sites, minimizing round-trip latency to while maintaining a consistent operational model with the AWS Regions (Outposts is part of the AWS Cloud). This is where Solace Event Broker excels. Deployed on AWS Outposts racks with accelerated networking Amazon EC2 instances, Solace Event Brokers can deliver high throughput data distribution with the low and predictable latency demanded by trading workloads.

Solace Event Broker’s event mesh architecture enables real-time data distribution across trading floors, co-location facilities, and cloud-based analytics systems, all while maintaining guaranteed delivery and high throughput. With AWS Outposts, firms can deploy Solace brokers physically close to trading venues for deterministic latency, and easily extend event streams to Solace Brokers deployed in AWS Regions for machine learning, risk analysis, or trade surveillance—without compromising performance or control.

This hybrid model empowers capital markets firms to modernize their trading infrastructure—preserving the deterministic performance of on-prem environments while benefiting from AWS’s ease of provisioning and security and operational consistency with AWS Regions. By running Solace Event Brokers on AWS Outposts racks with accelerated networking Amazon EC2 instances. Trading firms achieve the best of both worlds: ultra-low-latency event distribution at the edge, and seamless integration with cloud-native analytics services to accelerate innovation.

Topic Routing in Solace

In capital markets, market data distribution involves broadcasting an immense volume of real-time updates — price ticks, quotes, trades, depth-of-book changes — to thousands of consuming systems simultaneously. Achieving this at scale demands more than raw network speed; it requires an intelligent messaging layer that can route information precisely where it’s needed, without unnecessary duplication or processing overhead. This is where the Solace Event Broker’s Topic Routing becomes a game-changer.

Solace uses a hierarchical topic structure to dynamically route messages based on their subject matter — for example:
MARKETDATA/NYSE/IBM/TRADES
MARKETDATA/NASDAQ/AAPL/QUOTES
MARKETDATA/FX/EURUSD/DEPTH

Producers publish data once to these topics, and Solace’s event broker efficiently delivers each message only to subscribers that have expressed interest, using topic filters such as MARKETDATA/NYSE/> delivers all NYSE quotes or MARKETDATA/*/IBM/> delivers all IBM market data. This eliminates redundant data flow, minimizes network utilization, and ensures that each consumer receives only the data it requires — all while maintaining  low latency and high message fanout performance.

When deployed on AWS Outposts racks with accelerated networking Amazon EC2 instances, Solace Event Brokers can leverage dedicated network interfaces to route millions of topic-based messages per second locally within the trading environment. This setup delivers wire-speed performance and deterministic latency, enabling trading systems, order gateways, and analytics platforms to receive market data updates in real time without jitter or congestion.

Outposts also ensures that this low-latency infrastructure is fully managed and integrated with AWS services, easily extend event streams to AWS Regions. For example, the same market data published to MARKETDATA/NYSE/IBM/TRADES on-premises can be delivered locally or selectively forwarded to a Solace Broker in the parent AWS Region for consumption for use cases such as historical analytics, real-time dashboards, or AI-driven anomaly detection, while keeping the latency-critical trading workloads on-premises.

Test Environment

The results shown on this post were achieved by performing a set of tests ran under the following environment:

AWS Outpost

AWS Outpost racks with 13 bmn-sf2e instances:

• Operating System: RHEL 9.6
• 32 Core x2 CPUs Intel(R) Sapphire Rapids Lake 3.9 GHz turbo
• 1 TB RAM
• 4 x 4 TB NVMe
• 4 x AMD Solarflare X2522 NI, 8×25 Gbps (200 Gbps Total)

Solace Configuration

Solace Event Broker deployed as a Podman container:

• SolOS: 10.25.0.87
• Tier: 200K connections
• High-Availability Deployment
• Podman 5.4.0
• TLS 1.2 x509v3 Certificate

Test Clients Configuration

For simulating traffic from client applications, we used Solace’s testing tool SDKPerf using the C API. The test cases were managed and run with the help of Ansible, making sure they were evenly distributed across multiple accelerated networking EC2 instances located on the same AWS Outpost as the instances hosting the Solace Brokers:

• 10 Amazon EC2 bmn-sf2.metal instances
• 10 publishers spread across 10 nodes
• 100 subscribers/consumer across 10 nodes
• 10 Solace topics = Fan-out of 10
• Each client/process was pinned to a different CPU
• Each client was connected to the broker using a secure tcps channel

Environment Optimization

Although we achieved very good results with an ‘Out of the Box’ environment, we performed the following optimizations to achieve the impressive results presented in the following sections:

Podman Rootful Containers & Host Network.

Although, running a Solace software broker as a rootless container is fully supported, running as a Rootful container delivers the best performance.

Regarding networking, the containers were configured to use HOST network mode (over BRIDGED), which allows the container to use the host network stack instead of having to manage virtual interfaces and NATs.

Open Onload Kernel Bypass

One of the greatest advantages of accelerated networking Amazon EC2 instances is that they are available as bare metal instances, reducing latency impact inherent to virtualization layers. In our testing we made sure to take advantage of this by performing Kernel Bypassing using Solarflare’s  Open Onload drivers.

Same NUMA for CPUs & NIC

To take optimization to the extreme, in our testing we made sure that the CPUs and NIC Interfaces being used by the Solace Software Broker were in the same NUMA node to avoid slow remote memory access across sockets.

Latency testing were performed using the network interface ens8f0

Test Methodology

The tests were run using direct messaging – designed for use with high-rate or very low-latency message messaging use cases that can tolerate occasional message loss. The most common use case in trading systems is market data distribution.

For direct  messaging, payload sizes were used, and the following measurements were taken:

• Throughput; both ingress and egress
• Latency; Average, 99^th percentile and standard deviation

To ensure test legitimacy, each test was run for at least 3 times during 20 minutes, a valid Direct Messaging test is when publishers and subscribers are capable of maintaining target ingress & egress message rates and no message discard is registered.

Performance Results

Solace’s direct messaging can provide high-rate throughput and very low-latency because it does not require acknowledgments between publishers and consumers, also messages are only maintained in memory and never persisted to disk.

Direct Messaging Throughput

When measuring throughput while using Solace’s Direct Messaging QoS on a Software Broker deployed on general purpose VMs, the limiting factor is usually the NIC bandwidth (10 or 25 Gbps are common VM capacities).

Msg Size (Bytes)	Ingress (Msgs/Sec)	Egress (Msgs/Sec)	Ingress Bandwidth (Mbps)	Egress Bandwidth (Mbps)
100	2,700,000	27,000,000	2,059.94	20,599.37
512	1,100,000	11,000,000	4,296.88	42,968.75
1,024	800,000	8,000,000	6,250.00	62,500.00
5,120	180,000	1,800,000	7,031.25	70,312.50

If we analyze the results obtained when testing Direct Messaging, we can see that when deploying a Solace broker on a super high performant  bare metal EC2 instance like bmn-sf2.metal, we can push the broker its limits to achieve distribution of millions of messages per second.

Direct Messaging Latency

To establish a baseline, RTT was measured using SDKPerf — Solace’s purpose-built benchmarking tool with a single message a few times. The end-to-end round trip from publisher → broker → subscriber was measured at approximately 70 microseconds on average, indicating roughly 35µs from publisher to broker and 35µs from broker to subscriber.

We then published up to 20,000 msgs/second and had a latency probe measure the latency of messages of the application. As you can see from this table, the broker’s own processing overhead is minimal — the latency observed is driven almost entirely by the underlying network and the broker adds minimal overhead as throughput increases.

Msg Size (Bytes)	Average (µs)	99th Percentile (µs)	StdDev (µs)
100	72.10	80.00	7.20
512	70.50	76.00	6.80
1,024	70.30	78.00	6.40
5,120	79.90	88.00	7.80

Conclusion

By combining AWS Outposts racks with accelerated networking Amazon EC2 instances with Solace Event Brokers, we’ve created a solution that delivers the ultra-low latency and high throughput required for mission-critical financial applications—all while maintaining the operational consistency and scalability benefits of AWS cloud infrastructure.

---

AWS Contributors