← Back to the shelf
Appearance

Dispatches / a short note

concept

Mogged by microservices

Managing unimaginable amouts of debts flawlessly.

Dude managing 2800 and more microservices… just how… https://monzo.com/blog/how-we-run-migrations-across-2800-microservices

Gist from the article:

  • centralizing migration to one team for maximum coordination
  • avoid decentralizing migrations to owners - it stalls deployments indefinitely. centralizing it through a queue and jobs (with team notification enabled) streamlines it a lot better, takes less time and scales better (while also having retries available).
  • slow rollout, and quick rollback
  • not everything should be automated (80/20), rest case by case basis.
  • wrap up old sdk interfaces with an in-house wrapper sdk, use tools to prevent people using old sdk directly - anything should be invoked through in-house wrapper. Also, in-house wrapper can switch bw old sdk and new sdk based on config (easier rollback without redeploy).
  • targetting least critical first (labels) for rollout, then working through the critical systems later.
  • using rollback rules in argocd (through prometheus metrics - RPC errors, crashes / panics, database errors)
  • use centralized config store (that serivices talk to directly, not configmap or secrets) for enabling / turning off features.
  • stripping out legacy code.

him flexing again: https://x.com/JackKleeman/status/1190354757308862468

On contrary, here’s some positive views on monoliths: https://dev.to/codenameone/is-it-time-to-go-back-to-the-monolith-3eok