Building high quality services at Uber
Day one: The ironfist trial
- Build a simple proxy service
- Deploy it to 50 machines
- Do not go down, ever.
Just "productionize" it
- Logging info, warn and error.
- Logging to a distributed Queue
- Logging to an error reporting service
Just "productionize" it
- Monitor everything
- Monitor occurances of warnings
- Monitor volume of traffic
- Monitor volume of business objects / events
Just "productionize" it
- Manage config files
- Deal with command line overwrites
- Support multiple datacenters
- Allow remote overwrites
Just "productionize" it
- Log uncaught exceptions
- Do a graceful shutdown
- Abort and dump core.
Just "productionize" it
- Run a control port for every service
- Expose a profiler
- Expose heapdumps
Just "productionize" it
Pager duty and nagios
- When a health check fails nagios knows
- When a dashboard spikes nagios knows
- When nagios knows, pager duty wakes you up
Getting started quickly
- How do you scaffold a new app
- How do you setup CI
- How do you get it deployed
Potter workflows
Local workflow
- Check for conflicts
- Scaffold out some files
- Create a local git repository
- Ensure tests pass and you can work
Potter workflows
Shared workflow
- Push to a gitolite remote
- Setup documentation and wiki
- Setup jenkins CI
Potter workflows
EC2 workflow
- Setup your app in puppet
- deploy your app to your own EC2
Potter workflows
production workflow
- Setup sentry
- Setup graphite
- Integrate with deployment system
Potter plugins
You can write your own workflow
Just write a plugin that implements your own
commands
Thanks.
Oh, and we're hiring:
jakev@uber.com