In a conventional app, a "secret" is rotated if it leaks. In crypto custody, the secret is the asset. There is no chargeback if an attacker who has the private key broadcasts a transaction. Coinbase's custody and cold-storage practices — described in engineering and security posts over the years — lean on hardware security modules, geographically split key material, quorum approvals, and key-generation ceremonies that look more like nuclear surety than like openssl genrsa. The software is important. The human protocol is load-bearing.

Ceremonies are distributed systems with clipboards

A key ceremony is a scripted procedure: who is in the room, which devices are air-gapped, how entropy is sampled, how shares are produced (Shamir or HSM-native), how the ceremony is filmed or logged, how the HSM is initialized so no single engineer walks out with a working key. The point is not theater. It is to make undetected substitution hard and to produce an audit trail a regulator and an incident reviewer can believe.

Hot wallets still exist because customers withdraw. They hold limited inventory, with automated policies and alarms. Cold storage holds the bulk, with delayed, multi-party withdraw paths. The bridge between them is where people try to be clever and where most designs fail: a "convenience" hot top-up that bypasses quorum.

HSMs are not magic; policies are

An HSM can refuse to sign unless a quorum of smartcards or a policy engine agrees. That policy — amount, destination allowlists, time delays — is the real product. If policy lives in a database that the same app can rewrite, you have a software key with extra latency. Coinbase-scale custody invests in making policy changes as ceremonial as key gen: two-person rules, hardware-enforced constraints, and monitoring that treats a new destination as a threat until proven otherwise.

Address generation and deposit detection still need online systems. Those can be watch-only. The signing keys stay offline until a withdrawal ritual. Confusing watch-only infrastructure with signing infrastructure is how a blockchain indexer breach becomes a theft narrative even when keys were safe — or how a signing box gets networked because someone wanted richer logs.

Failure modes of custody engineering

The concrete failure is a hot wallet with an unbounded drain path because the policy engine timed out and the code failed open. Mid-size steal: fail closed, cap inventory, and alert on velocity.

Operational gotcha: ceremony leftovers — a laptop that was used for air-gapped gen and then rejoined the corporate LAN, or a printed share in a desk. The ceremony's last step is destroying or securing intermediates. Another is geographic redundancy of shares that still sit in one provider's "different regions" with the same insider risk. True split is legal and physical, not only AZ-id. Software updates to HSM firmware and to the code that builds unsigned transactions can change what you think you signed; hash the payload on an independent display. If you run any crypto keys, even for a game, do not put them in environment variables. Most companies are not Coinbase; they should still copy the idea of inventory caps and two-person production access. Test restore from backup shares on a schedule. An untested cold backup is a story you tell after the fire. Do not let customer support tools craft withdrawals. Support tickets are a social engineering API.

What you can borrow

  • Treat signing keys as requiring a scripted ceremony, quorum, and fail-closed policy, not a config value.
  • Split hot inventory from cold storage with a delayed, reviewed bridge.
  • Keep watch-only indexing off the boxes that can sign.
  • Practice recovery of key shares; an untested backup is not a backup.