Dispatches / a short note
Mogged by microservices
Managing unimaginable amouts of debts flawlessly.
Dude managing 2800 and more microservices… just how… https://monzo.com/blog/how-we-run-migrations-across-2800-microservices
Gist from the article:
- centralizing migration to one team for maximum coordination
- avoid decentralizing migrations to owners - it stalls deployments indefinitely. centralizing it through a queue and jobs (with team notification enabled) streamlines it a lot better, takes less time and scales better (while also having retries available).
- slow rollout, and quick rollback
- not everything should be automated (80/20), rest case by case basis.
- wrap up old sdk interfaces with an in-house wrapper sdk, use tools to prevent people using old sdk directly - anything should be invoked through in-house wrapper. Also, in-house wrapper can switch bw old sdk and new sdk based on config (easier rollback without redeploy).
- targetting least critical first (labels) for rollout, then working through the critical systems later.
- using rollback rules in argocd (through prometheus metrics - RPC errors, crashes / panics, database errors)
- use centralized config store (that serivices talk to directly, not configmap or secrets) for enabling / turning off features.
- stripping out legacy code.
him flexing again: https://x.com/JackKleeman/status/1190354757308862468
On contrary, here’s some positive views on monoliths: https://dev.to/codenameone/is-it-time-to-go-back-to-the-monolith-3eok