calum.org:~#

Enjoyable development

Tags: programming linux java aws soa metrics agile graphite logs

Added: 2014-05-02T11:50:29

Enjoyable development

I feel very fortunate to be working where I'm working at the moment. We've reached a really nice spot in our development techniques, methods, tools, etc.

I work in a small part of a large corporation writing services to support our products. The services are used by millions of people a day all around the world.

I thought I'd share what makes it so enjoyable.

We don't have many layers of management - we're treated like responsible adults, and given pretty much all the flexibility we want.
We have an open forum once a week where anyone can suggest anything that needs changing.
We work in an Agile way.
We can deploy when we want - we don't need sign-off from anyone.
We run in a Service Oriented Architecture. Many small services, all consuming from each other via REST.
We get to choose what software/libraries we want. Need a DB? MySQL? Mongo? DynamoDB? Pick the best one for the problem.
We like tests. Not mindless "I need to cover every single code path" tests, but useful ones.
We're pragmatic, not slaves to dogma.
We practice Continuous Integration. We have Jenkins servers running the tests for every commit, with screens around the office showing when tests fail.
We deploy many times a day - lots of small changes. This makes it easy to identify what caused a problem.
It is very easy for us to roll back in the event of a problem.
We have our own in-house commandline deployment tool which manages baking and deploying.

blah deploy servicename prod
is all it takes. No authentication needed. There shouldn't be any code committed that isn't deployable, and if there is, it's the fault of the committer, not the deployer.
We don't need onerous RFCs to deploy. We have our deployment system set up to raise and close Standard Changes automatically. A deployment now only takes 3 minutes, rather than the "long" 15 minutes of our previous system.
We can check out code, modify, commit, create a release and have it deployed and running in prod within 15 minutes.
We are moving to Amazon's AWS.
We use Amazon's Autoscaling to replace servers in the event of a problem with a server.
We love metrics. All of our services give us lots of juicy metrics via Graphite. We have more screens around the office showing beautiful graphs.
We devs have SSH access to all the servers. Very useful for problem diagnosis, or even modifying a config and restarting a service.
We use Seyren to notify us when metrics exceed threshholds.
We have a decent centralised logging system (Logstash + Kibana)
We have an in-house Git-based system for AWS configuration. If you, say, need to add a new port to a load-balancer, or increase the number of instances for a service, you simply check out the config, make the change, and push, and in a few minutes, the change will be applied across the environment.
We keep all our service configs in Git too.
We have Hack Days, sometimes even Hack Weeks. This is when we stop doing "normal" work, and scratch ideas that have been itching us.
We use Linux everywhere - desktops, servers.

Git -> Code -> Push -> Run Tests -> Create release (RPM) -> Bake AMI with RPM and Config -> Deploy -> Relax.

All of this makes for a very pleasant way to develop.

posted by Calum on 2014-04-24T20:30 under

Add a comment

Your IP:
Please enter 5903794 here: