Most microservice diagrams put stateless compute in front of a database and call the database the source of truth. That split is easy to reason about until the database becomes the bottleneck, the network hop sits on every request, and failover is something the data tier does while the app tier just retries. Microsoft Service Fabric, the cluster OS that has hosted Azure SQL Database, Cosmos DB components, Event Hubs, and many first-party services, took the opposite default for a class of workloads: let a stateful service own partitioned, replicated in-memory (and disk-backed) collections, and let the runtime move primary replicas when nodes die. The unit of scale is a partition of the service, not a connection pool to a remote store.

Reliable collections are not "Redis on the box"

Service Fabric stateful services expose reliable dictionaries and queues that look like local data structures. Writes go through a replication protocol to replica sets: a primary applies the operation, secondaries confirm, then the write acknowledges. Reads can be restricted to the primary for strong consistency or served from secondaries when the service allows it. The programming model is closer to a partitioned, replicated log plus a state machine than to an embedded cache. If you treat the dictionary as a cache that can be dropped, you will. If you treat it as the system of record, you must size partitions, back up, and test failovers the way you would for a database — because that is what you built.

Partitioning is key-based. A poor key (one tenant, one date, one hot player) puts all traffic on one primary. Fabric will not magically split a partition under load the way some cloud databases split ranges. You choose the partition count at service creation for many classic deployments, which makes early capacity planning a product decision, not a later optimization. Secondary replicas exist for availability; they also exist so a failover can be a promotion rather than a cold rebuild from off-cluster storage.

Lifecycle, upgrades, and the ways this bites

Fabric's cluster manager places replicas with constraints (fault domains, upgrade domains) so a rack loss or a rolling upgrade does not kill every copy of a partition at once. Rolling upgrades of stateful services are the sharp edge. During an upgrade, primaries move, replication pauses or slows, and in-flight transactions must drain. Teams that deploy as if the service were stateless — instant instance replacement, no replica quorum awareness — cause availability blips that look like "the cluster is flaky" in dashboards.

Another failure mode is unbounded reliable-collection growth. In-memory state that never expires becomes a swap storm, then a failover storm, because the replica can no longer rebuild in time. Compaction, truncation of queues, and explicit archival to blob storage are part of the design, not extras. Backup and restore are similarly non-optional: replica sets protect against node loss, not against a bad deploy that writes garbage into every primary.

The reason Azure's own services used this model is locality. If the state machine for a shard of Event Hubs or a slice of a control plane lives on the same nodes that serve it, you avoid a second clustered database for that hot path. The cost is operational complexity that a managed SQL or Cosmos account would have hidden. Mid-size teams should steal the idea — partition by key, replicate a log, fail over the primary — only when an external database cannot meet latency, and they should be honest about owning a database thereafter.

What you can borrow

  • Collocate a replicated state machine with the service that mutates it when cross-network database hops dominate your p99.
  • Pick partition keys that spread load; treat partition count as a capacity contract you will live with.
  • Run upgrades and node drains as first-class tests: primaries must move without dropping quorum.
  • Cap in-cluster state size and archive; replica rebuild time is your real RTO.
  • Do not confuse replica redundancy with backup. Quorum will happily replicate a corrupt write.