Enhancing Production Operations with FastMCP and AI Agents

It is 6:30 PM on a Friday, and you are staring at 50 events trapped in a Dead Letter Queue (DLQ). Each one represents stuck funds or an end customer waiting on payment confirmation. In event-driven microservice architectures, diagnosing why each flow broke down usually means cross-referencing five distinct data models and untangling asynchronous calls across hours or days.

With the advent of AI agents and the Model Context Protocol (MCP), we can slash these recovery times dramatically. The goal is never to let an LLM run wild across production, but rather to equip it with deterministic tools while keeping the human analyst firmly in the driver’s seat.

The Nightmare of Forensic Triage

The real bottleneck in these operational incidents is not event volume, but cognitive overhead. To understand why a single transaction failed, an analyst must reconstruct its entire lifecycle across disjointed tooling:

• Inspect trace spans in CloudWatch or Datadog to uncover silent exceptions.
• Run ad-hoc SQL queries in Athena or read replicas to reconcile database states.
• Probe internal APIs using Postman or cURL to check whether downstream subscribers ever processed the message.

Multiply this workflow across every single message in the queue. Manual triage burns entire workdays, and mental fatigue takes a toll: missing a single nested field in a giant JSON payload or skipping one service check leads to flawed conclusions and inflates the Mean Time to Resolution (MTTR).

The First Attempt: Traditional Scripting

When recurring failures occur, our initial engineering impulse is to write a Python or Bash script. It is quick to assemble and works reliably as long as failure conditions remain identical.

However, in a production DLQ, those 50 trapped events frequently stem from 5 or 6 distinct root causes (downstream timeouts, inconsistent intermediate states, malformed schemas). Maintaining standalone scripts for every subtle variant forces teams into manual batching, code divergence, and ultimately defeats the speed we set out to achieve.

The Solution: FastMCP + Agent Skills

By implementing a server with FastMCP, we expose tools that AI clients like Cursor or Windsurf can seamlessly orchestrate. The MCP server encapsulates deterministic business logic against our internal APIs, while the language model contributes contextual reasoning to analyze discrepancies on a case-by-case basis.

Execution Architecture and Security Guardrails

Because this operational workflow directly touches production data, the architecture adheres to two foundational principles:

1. Authentication via Environment Variables: Access tokens for internal APIs or databases are injected directly into the MCP server process through local environment variables (OPERATIONS_API_TOKEN), ensuring credentials are never exposed to the LLM or chat transcripts.
2. Human-in-the-Loop via Stdio: The MCP server runs locally, communicating over stdin/stdout with the IDE (Cursor / Windsurf). The analyst retains complete visibility over proposed actions and must explicitly approve every mutating tool invocation (patch_transaction) within the interface.

Implementing the MCP Server

In this implementation, we expose two dedicated tools: a read tool to inspect transactions and a write tool to apply corrective patches. Notice that strictly typing the HTTP response payload is unnecessary: the model parses and interprets the returned dictionary without friction. Conversely, mutating inputs are strictly typed with Pydantic to act as robust safety guardrails:

import os
from decimal import Decimal
from typing import Optional
from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field

# Initialize FastMCP server
mcp = FastMCP("transactions-mcp")

# Client consumes credentials injected via environment variables
auth_token = os.environ.get("OPERATIONS_API_TOKEN")
client = TransactionsClient(token=auth_token)

class PatchTransactionRequest(BaseModel):
    id: str = Field(description="Transaction unique identifier")
    status: Optional[str] = Field(None, description="New status (COMPLETED, CANCELLED)")
    amount: Optional[Decimal] = Field(None, description="Transaction amount")
    sent_at: Optional[str] = Field(None, description="ISO timestamp when the event was dispatched")

@mcp.tool()
async def get_transaction(id: str) -> dict:
    """
    Retrieves the transaction details across microservices.
    """
    return await client.get_transaction_by_id(id)

@mcp.tool()
async def patch_transaction(request: PatchTransactionRequest):
    """
    Applies a corrective patch to a transaction.
    Operation is idempotent and creates an immutable audit trail entry (analyst ID, timestamp, diff).
    """
    return await client.patch_transaction(request)

if __name__ == "__main__":
    mcp.run()

Design Note on Idempotency: The patch_transaction endpoint must be strictly idempotent. If an analyst confirms a patch or a network timeout triggers a client retry, applying the exact same payload twice must not produce duplicated side effects. Furthermore, the backend automatically writes an immutable audit log linking the analyst session, timestamp, and field diffs.

Connecting the Server to the IDE with uv

To connect our local server to Cursor or Windsurf, configure the corresponding mcp.json file. Running the process via uv provides the cleanest and most reproducible setup, handling the virtual environment on the fly with instantaneous stdio startup:

{
  "mcpServers": {
    "transactions-mcp": {
      "command": "uv",
      "args": ["run", "mcp"],
      "env": {
        "OPERATIONS_API_TOKEN": "sec_live_ops_8f3a92bc..."
      }
    }
  }
}

Documenting Tools: The Skill

A common question is: Do we really need a Markdown Skill if modern IDEs automatically inspect MCP schemas?

Under the hood, tools like Cursor, Claude Desktop, and Windsurf invoke tools/list on the MCP protocol to automatically ingest Pydantic types and docstrings directly from Python. For trivial operations, this schema discovery is sufficient.

However, in mission-critical operations, a Markdown Skill provides two indispensable advantages:

1. Business Context over Primitive Types: While the MCP schema merely specifies that status is a str, the Skill explains the operational ramifications of transitioning an entity to COMPLETED, or why CANCELLED is prohibited once a ledger entry exists.
2. Operational Heuristics: It instructs the agent on evaluation order, when to hold off on data mutations, and how to interpret ambiguous responses without polluting backend code with bloated docstrings.

# MCP Skill: Transaction Operations Guide

This is an explanation for the `transactions-mcp` toolset in order to work with transactions.

## **MCP Tool: get_transaction**
- **Tool ID:** `get_transaction`
- **Description:** Retrieves the details of a specific transaction across microservices.
- **Inputs:**
  - `id` (string, required): Transaction unique identifier.
- **Output:**
  - `dict`: Transaction details including status, amount, timestamps, etc.

## **MCP Tool: patch_transaction**
- **Tool ID:** `patch_transaction`
- **Description:** Applies a corrective patch to a transaction, updating fields to the given values.
- **Inputs:**
  - `id` (string, required): Transaction unique identifier.
  - `status` (string, optional): New status (COMPLETED, CANCELLED).
  - `amount` (Decimal, optional): New amount.
  - `sent_at` (string, optional): Timestamp when the event was sent.
- **Output:**
  - Acknowledgement of the update.

## **Operational Heuristics & Business Rules**
- **Legitimate Errors (Do NOT patch):** Failures such as `INSUFFICIENT_FUNDS`, `CARD_EXPIRED` or `FRAUD_BLOCKED` are terminal business outcomes, not technical glitches. Do not patch or retry these transactions; flag them to the analyst as legitimate rejects.
- **Technical Inconsistencies (Patch candidates):** If ledger charges exist but the status remained `PENDING`, or a `COMPLETED` transaction has no dispatch timestamp, these represent message loss or event timeouts that require state reconciliation.

Natural Language Orchestration: The Workflow

Finally, we define the operational workflow. This document completely supersedes fragile one-off scripts: instead of hardcoding execution paths, we express core operational procedures in natural language. The agent assesses runtime conditions and devises targeted action plans:

# Transaction analyst workflow

You are a transaction analyst specialized in payment operations. Use the available tools to analyze and resolve transaction issues.

## Execution plan

For the given list of transaction ids, perform the following steps:

1. Retrieve transaction details using the `get_transaction` tool.
2. Verify and plan patches based on transaction statuses:
   - If status is "PENDING" but amount is present: the dispatch event was missed. Prepare a patch to update status to "COMPLETED".
   - If status is "COMPLETED" but `sent_at` is null: the event was lost. Prepare a patch with status "COMPLETED" and `sent_at` set to the current ISO timestamp.
3. Present the proposed action plan to the analyst for confirmation.
4. Once authorized, execute `patch_transaction` for each transaction.
5. Verify the updated state by invoking `get_transaction` once again.

Live Interaction: Reasoning and Confirmation

When you hand an agent a batch of transaction IDs, execution is far from an opaque black box. Guided by the workflow, the model first reviews the runtime state and presents a structured proposal:

Agent:
“I have analyzed the 3 provided transactions:

• • tx_90123: Status is PENDING, but the EUR 150.00 charge was already captured in the ledger (confirmation event was never dispatched). Proposed patch: set status to COMPLETED.*
• • tx_90124: Status is COMPLETED, but sent_at is empty (lost event). Proposed patch: populate sent_at with 2026-09-26T18:32:00Z.*
• • tx_90125: Legitimate failure due to insufficient funds. No operational intervention required.*

Would you like me to execute patch_transaction for tx_90123 and tx_90124?“

At this juncture, the human analyst simply clicks Approve in the IDE interface to trigger the patches in seconds—or denies execution if unexpected domain nuances appear.

Results: Balancing Incident Inflow and Outflow

The true operational bottleneck during incidents is the structural asymmetry between inflow (the rate at which failures land in the DLQ or backlog) and outflow (the team’s capacity to diagnose, reconcile across databases, and mitigate them).

By pairing local FastMCP servers with declarative natural language workflows:

• Balanced Inflow vs. Outflow: Incident batch triaging that once consumed days of fragmented context-switching is now resolved in a couple of hours. Operational outflow increased significantly without expanding on-call headcount.
• Continuous Control and Auditability: Running over local stdio within the analyst’s IDE means every single state mutation demands human sign-off. We eliminate the risk of destructive hallucinations without sacrificing LLM reasoning speed.
• Zero-Friction Adaptability: When an unforeseen edge case surfaces in production, adjusting the operational heuristic takes 30 seconds inside the Markdown prompt, completely avoiding the overhead of refactoring, testing, and shipping traditional automation scripts.