FiveRuns Blog

On Rails production performance and monitoring

Posts
No comments

Beyond Continuous Integration: Continuous Information

Steve recently posted about the tight feedback loops that we have in place on the FiveRuns development team. He described our tightest loop as being our morning standup meetings. I’m going take you further, though, into the workdays (and nights) between our daily standups. Here’s a peek at the tools and agile practices we use to stay constantly informed of the product status from hour to hour and minute to minute.

10:15am

Daily standup meeting over, I grab another cup of coffee from the kitchen. I head back to my desk, put in my earbuds, fire up iTunes, svn up, and get cranking on Feature X.

Takeaways:

  1. coffee from the kitchen – As a group, our collective level of caffeination is well above normal. The result? High-bandwidth conversations and fast context-switching.
  2. earbuds, iTunes – Most of the time, we’re heads-down, music on, focused on our individual sets of issues in Mingle. We don’t, in general, employ pair programming. Very commonly, though, one of us will need a second pair of eyes for a code read or an impromptu whiteboard design session; we grab whomever we need, focus on the task at hand, and move on.
  3. svn up – Most of our code is in subversion. Bruce and Adam have also been dropping some recent source into an in-house git repo.
11:28am

Feature X completed. Everything rakes green, so I svn commit with a descriptive error message. Ding! A Growl notification from Pyro lets me know that there are some new Campfire messages. I Apple-Tab to Pyro and see..

Bweep! New email comes in from subversion containing the same content.

Takeaways:

  1. rakes green – Our dev process is test-driven. We write test/spec tests and, by convention, svn up and rake before we commit our changes to the svn repository.
  2. Growl, Pyro, Campfire – Campfire is second to none for simple, functional group chat. We make it a point to always be online in the ‘Development’ room. In addition to it being our primary means of communicating with Adam (our one telecommuting team member), it’s integrated with our subversion repository through a post-commit hook and is constantly updated with svn commit messages and source diffs. Pyro is a standalone Mac interface to Campfire.
  3. email notification – We also get subversion commit notifications via email. Since I’m on Campfire all the time, I almost never look at these, opting instead to funnel them into a folder via a Mail.app rule.
11:30am

Ding! Pyro again. Another message.

This time, the message is from CruiseControl; I broke the build.

Bweep! Email from CruiseControl? You betcha.

I get an IM from Mike: “Dude, you broke the build.” Thanks, Mike.

Eric leans to the side at his desk and says to the room, “Somebody broke the build.”

“Yeah, that’s me,” I say. “I’m on it.”

Takeaways:

  1. CruiseControl – This continuous integration solution keeps an eye on the subversion repository and will rebuild and retest whenever a project is modified. It, too, is integrated with Campfire.
  2. Email – Of course, CC sends any notices to email as well. For a long while, I was sending these, unread, into a Mail folder, just like the subversion messages. At one point, however, our CruiseControl/Campfire integration layer broke down, and I failed to notice when I (or someone else) had broken the build.
  3. Mail.app rules to the rescue! I deleted the rule which sent the CruiseControl messages into a folder, and instead wrote 2 new rules:

    Email subjects with colored backgrounds FTW.
  4. IM – Yep, we’re on Instant Messaging, too. Adium or iChat, mostly.
11:32

I had forgotten to add a file to the repository. I realize this and check it into subversion. Ding! Bweep!

Cool. I move the issue’s status in Mingle to ‘Ready for Standup Review’ and move on to the next one in my queue.

Takeaways:

  1. Mingle, ‘Ready for Standup Review’ – Communicating my progress via Mingle keeps everyone (both inside and outside the dev team) in the loop. At tomorrow morning’s standup, I’ll report that I finished up Feature X. We’ll then move the issue’s status to ‘Complete’ and show it off next week to the rest of the company at the Friday Sprint Review.

And…that’s it. We all have each others’ cell phone numbers as well, should anything mission-critical (like a production outage) rear its head. In a nutshell, we stay continuously abreast of product changes by hooking all of our core infrastructure (SVN, git, CruiseControl) into a variety of communication channels (Campfire, email, IM) while using top-quality tools and practices (Pyro + Growl, Mail.app, Adium/iChat, direct human interaction) to monitor those channels. Do the same, and you’ll see team productivity benefits in no time.

Bookmark and Share
Continued Discussion

No comments have been added yet.

Contribute

Continue the conversation and share your thoughts. A name is required. Your e-mail address will not be displayed on the site. Textile formatting may be used in your comments (but will not be rendered in the live comment preview).

→ Posted by You on November 20, 2008 at 04:09 PM

Flickr

FiveRuns tagged photos on Flickr.

  • FiveRuns at the SF Ruby User Group
  • FiveRuns at the SF Ruby User Group
  • Al
  • Mindy
  • FiveRuns at the SF Ruby User Group
  • The office
  • FiveRuns at the SF Ruby User Group
  • Rachel

See more FiveRuns tagged photos…

Other Categories

Entries are also organized under the following general topic categories.