Architecting a Multicloud Event Sourcing Plane for Hexagonal Microservices

State synchronization across distributed multicloud microservices often degenerates into a tangled web of distributed transactions and dual-write anti-patterns. When an Amazon Web Services hosted command service updates a core domain entity, failing to immediately propagate that state change to a Microsoft Azure hosted query projection leaves the system fundamentally inconsistent. This architectural flaw leads to phantom reads, corrupted user experiences, and severe debugging nightmares where distributed traces vanish across cloud provider boundaries. The definitive resolution to this fragility is implementing a multicloud Event Sourcing and Command Query Responsibility Segregation (CQRS) plane. By leveraging AWS EventBridge and Azure Event Grid as a unified enterprise service bus, and structuring the application payload using Hexagonal Architecture, engineering teams decouple state mutation from read projections entirely. This strategy ensures deterministic state reconstruction, guarantees eventual consistency across distinct cloud ecosystems, and enforces strict domain boundaries necessary for massive production scale.

Prerequisites

Executing this multicloud event-driven architecture requires Terraform version 1.7.0 or higher. You must configure the AWS provider (version 5.40.0+) and the AzureRM provider (version 3.90.0+). The application routing logic necessitates Python 3.12 utilizing the boto3 (1.34+) and azure-cosmos (4.5+) libraries. A rigorous understanding of Domain-Driven Design principles, specifically aggregate roots, domain events, and bounded contexts, is mandatory. Engineers must possess advanced familiarity with AWS EventBridge API Destinations, IAM role assumption mechanisms, and Azure Event Grid custom topics to successfully establish the secure, federated message topology.

Step-by-Step

Step 1: Provisioning the Federated Multicloud Event Bus

Establishing a reliable communication channel between isolated cloud providers requires an infrastructure layer that natively handles retries, authentication, and payload routing without introducing intermediary compute nodes. We accomplish this by configuring AWS EventBridge to act as the primary ingress for domain events and utilizing an EventBridge API Destination to securely forward these payloads directly to an Azure Event Grid custom topic. This topology completely eliminates the need for custom Lambda forwarders, thereby reducing operational overhead and cold start latency. We deploy this configuration using Terraform to ensure deterministic infrastructure states. The AWS EventBridge connection resource securely manages the Azure Event Grid Shared Access Signature key, injecting it into the authorization headers dynamically. By binding an EventBridge rule to this API destination, we filter and route only specific bounded context events, preventing unnecessary egress costs and ensuring Azure only receives payloads strictly relevant to its query projections.

# multicloud_bus/main.tf
variable "azure_event_grid_endpoint" {
  type        = string
  description = "The HTTPS endpoint for the Azure Event Grid Custom Topic"
}

variable "azure_event_grid_sas_key" {
  type        = string
  sensitive   = true
  description = "The SAS key for Azure Event Grid authentication"
}

resource "aws_cloudwatch_event_connection" "azure_federation" {
  name               = "azure-event-grid-connection"
  description        = "Secure connection to Azure Event Grid"
  authorization_type = "API_KEY"

  auth_parameters {
    api_key {
      key   = "aeg-sas-key"
      value = var.azure_event_grid_sas_key
    }
  }
}

resource "aws_cloudwatch_event_api_destination" "azure_target" {
  name                             = "azure-event-grid-destination"
  description                      = "Forward domain events to Azure"
  invocation_endpoint              = var.azure_event_grid_endpoint
  http_method                      = "POST"
  invocation_rate_limit_per_second = 300
  connection_arn                   = aws_cloudwatch_event_connection.azure_federation.arn
}

resource "aws_cloudwatch_event_rule" "logistics_events" {
  name        = "capture-logistics-domain-events"
  description = "Capture all shipment manifest events"
  event_pattern = jsonencode({
    source = ["corp.logistics.shipment"]
  })
}

resource "aws_cloudwatch_event_target" "azure_forwarder" {
  rule      = aws_cloudwatch_event_rule.logistics_events.name
  target_id = "SendToAzureEventGrid"
  arn       = aws_cloudwatch_event_api_destination.azure_target.arn
}

This infrastructure efficiently routes raw bytes from AWS to Azure seamlessly. How do we guarantee the structural integrity and semantic meaning of these events as they traverse these distinct architectural boundaries?

Step 2: Enforcing Domain Event Contracts via Hexagonal Ports

We guarantee structural integrity by enforcing a strict schema registry and utilizing Hexagonal Architecture to isolate the event publishing mechanism from the core domain logic. In a pure DDD implementation, the core domain must remain blissfully unaware of cloud provider specifics. We achieve this dependency inversion by defining a precise domain event contract using Python dataclasses and establishing an abstract protocol (the port). The infrastructure layer then implements this protocol (the adapter), translating the abstract domain event into a concrete boto3 EventBridge payload formatted to the CloudEvents specification. This separation of concerns ensures that if the enterprise decides to migrate the message broker from EventBridge to Apache Kafka, the core domain logic remains untouched. The adapter is strictly responsible for handling AWS specific error handling, retries, and batching constraints, ensuring the command service remains highly cohesive and loosely coupled.

# domain/events.py
import json
import logging
from typing import Protocol, List
from dataclasses import dataclass, asdict
from datetime import datetime, timezone

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

@dataclass(frozen=True)
class ShipmentDispatchedEvent:
    aggregate_id: str
    tracking_number: str
    carrier_code: str
    sequence_number: int
    occurred_on: str = datetime.now(timezone.utc).isoformat()

class EventPublisherPort(Protocol):
    def publish(self, events: List[ShipmentDispatchedEvent]) -> None:
        pass

# infrastructure/adapters.py
import boto3
from botocore.exceptions import ClientError

class EventBridgeAdapter:
    def __init__(self, event_bus_name: str):
        self.client = boto3.client('events')
        self.event_bus_name = event_bus_name

    def publish(self, events: List[ShipmentDispatchedEvent]) -> None:
        entries = []
        for event in events:
            entries.append({
                'Time': datetime.fromisoformat(event.occurred_on),
                'Source': 'corp.logistics.shipment',
                'DetailType': 'ShipmentDispatched',
                'Detail': json.dumps(asdict(event)),
                'EventBusName': self.event_bus_name
            })

        try:
            response = self.client.put_events(Entries=entries)
            if response.get('FailedEntryCount', 0) > 0:
                logger.error(f"Failed to publish events: {response}")
                raise RuntimeError("Partial failure during event publishing.")
            logger.info(f"Successfully published {len(events)} events to AWS EventBridge.")
        except ClientError as error:
            logger.error(f"AWS API Error: {error}")
            raise

This decoupled publishing mechanism securely delivers payloads to the Azure boundary. What happens when the Azure consumer receives these events out of order due to network latency or Event Grid retry storms?

Step 3: Idempotent Materialized View Generation in Azure

When the Azure consumer receives events out of order, the system must rely on strict sequence versioning and idempotent projection logic to maintain state integrity. In a distributed multicloud system, chronological sorting is unreliable due to clock drift and asynchronous delivery guarantees. Instead, we implement Optimistic Concurrency Control using Azure Cosmos DB. The read projection handler functions as the consumer, intercepting the payload from Event Grid. It reads the current materialized view from Cosmos DB and compares the incoming event's sequence number against the stored document version. If the incoming event possesses a lower or equal sequence number, it is recognized as a stale or duplicate event and is safely discarded, ensuring idempotency. If the sequence is valid, the handler applies the state mutation and persists the updated document back to Cosmos DB utilizing the _etag property to prevent race conditions during concurrent updates. This pattern ensures that the Azure-hosted query API always serves eventual, yet mathematically correct, state reconstructions regardless of network anomalies.

# projection/handler.py
import os
import logging
from typing import Any, Dict
from azure.cosmos import CosmosClient, exceptions

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

def get_cosmos_container():
    client = CosmosClient(
        os.environ["COSMOS_DB_ENDPOINT"], 
        credential=os.environ["COSMOS_DB_KEY"]
    )
    database = client.get_database_client("LogisticsQueryDB")
    return database.get_container_client("ShipmentProjections")

def process_event_grid_payload(event: Dict[str, Any]) -> None:
    container = get_cosmos_container()
    event_type = event.get("eventType")
    event_data = event.get("data", {})

    if event_type != "ShipmentDispatched":
        logger.info("Ignoring irrelevant event type.")
        return

    aggregate_id = event_data.get("aggregate_id")
    incoming_sequence = event_data.get("sequence_number")

    try:
        # Fetch the current state to validate sequences
        current_state = container.read_item(
            item=aggregate_id, 
            partition_key=aggregate_id
        )
        current_sequence = current_state.get("version", 0)

        if incoming_sequence <= current_sequence:
            logger.warning(f"Discarding stale event. Incoming: {incoming_sequence}, Current: {current_sequence}")
            return

    except exceptions.CosmosResourceNotFoundError:
        # Document does not exist yet, proceed with creation
        current_state = {"id": aggregate_id, "partition_key": aggregate_id}

    # Apply state mutation
    current_state["tracking_number"] = event_data.get("tracking_number")
    current_state["carrier_code"] = event_data.get("carrier_code")
    current_state["version"] = incoming_sequence
    current_state["last_updated"] = event_data.get("occurred_on")

    try:
        # Upsert with Optimistic Concurrency Control using ETag if it exists
        if "_etag" in current_state:
            container.replace_item(
                item=aggregate_id, 
                body=current_state, 
                etag=current_state["_etag"], 
                match_condition=exceptions.MatchConditions.IfNotModified
            )
        else:
            container.create_item(body=current_state)

        logger.info(f"Successfully projected view for aggregate {aggregate_id}")

    except exceptions.CosmosPreconditionFailedError:
        logger.error("Concurrency conflict detected. Another process updated the view.")
        raise RuntimeError("ETag mismatch during projection materialization.")

Common Troubleshooting

When federating EventBridge to Event Grid, engineers frequently encounter 401 Unauthorized errors in the AWS CloudWatch logs for the API Destination target. Verify that the Azure Event Grid SAS key injected into the AWS EventBridge connection resource has not expired. The key must be explicitly rotated, and the Terraform workspace applied, before the Azure expiration policy invalidates the token. Furthermore, verify the API connection sets the header key precisely as aeg-sas-key; standard HTTP Authorization headers will be rejected by Azure.

During the execution of the projection logic, encountering azure.cosmos.exceptions.CosmosPreconditionFailedError is an expected behavior during high-throughput concurrent processing of the same aggregate root. This exception indicates the optimistic concurrency lock prevented a race condition. Do not treat this as a fatal failure. Ensure your Azure Function handler is configured to retry on failure; the subsequent execution will fetch the updated _etag and process sequentially.

Finally, if events appear to be missing entirely in the Azure Cosmos DB projection, inspect the AWS EventBridge Dead Letter Queue. Azure Event Grid enforces a strict 1MB maximum payload limit per event. If the command service attaches heavy, uncompressed payloads to the domain event, AWS will fail to forward the message and shunt it to the dead letter queue.

Conclusion

Implementing a multicloud event sourcing architecture fundamentally transforms brittle synchronous APIs into highly resilient, decoupled systems. By strictly adhering to Hexagonal Architecture and utilizing robust managed buses like EventBridge and Event Grid, we isolate vendor dependencies and guarantee structural integrity across cloud providers. The idempotent projection logic ensures that query models remain eventually consistent despite the inherent chaos of distributed networks. For advanced scaling, integrate OpenTelemetry across both providers to visualize the complete transaction lifecycle, mapping the exact latency from the AWS command entry point to the final Azure materialization.

References

Evans, E. (2004). Domain-driven design: Tackling complexity in the heart of software. Addison-Wesley Professional.

Fowler, M. (2011). CQRS. MartinFowler.com. https://martinfowler.com/bliki/CQRS.html

Stopford, B. (2018). Designing event-driven systems: Concepts and patterns for streaming services with Apache Kafka. O'Reilly Media.

Vernon, V. (2013). Implementing domain-driven design. Addison-Wesley Professional.