The "Oh Shit" Moment

You know that point when you realise you forgot something crucial, but it's too late to do anything about it? That's the "Oh shit" moment.

Last week, one of the guys at GitHub experienced just such a moment, when he accidently dropped their production database. A lot of people made fun of GitHub for this (after their data had been restored). We didn't laugh. Because we had just done something similar ourselves.

In our down time between client projects, we've been working on our own web application (more to come on that subject). It's been in private beta for a little while now, and some of our friends have been trying it out. We've been using it extensively ourselves. While trying out a new feature, we accidently deleted the entire production database. And we didn't have a backup.

What was the cause of this royal screwup? I'll get to that, but first I'd like to apologise to our friends who lost the data they had put in while helping us try out our new service. We fucked up. I'm so, so sorry!

What happened

We host our app on Heroku, and we have two environments there: the production (beta) environment and our staging (testing) environment. We were going to reset our staging database to get some new data in there by running heroku rake db:reset.

When you have multiple app environments on Heroku you specify which one you want to run a command against by appending --app appname to your command. If you leave that out, Heroku tries to guess which one you meant by seeing if any of your environments have the same name as the local directory you're in. In our case the production environment was named in the form of "appname" and the staging environment was "appname-staging". Our local directory was also named in the form "appname", which of course meant that the rake db:reset command ran against our production environment and reset the production database.

Now, we should of course have had a backup. But in our small private beta we just hadn't taken the time to set that up yet. We thought other things were more important. Now we know better.

What we are doing about it

We've taken two important steps to make sure that this doesn't happen again.

  1. We now have backups that get run automatically.
  2. We've changed the name of our production environment to the form of "appname-production" to make sure that we don't accidently run commands against it when we forget to add the --app flag.

If you were affected by our mistake, please accept my apologies. I am truly, very sorry.

Oh shit!


Build, RubyConf and Dave Hoover

Today we are 4 people at the office, half of the crew. Jimmy and Johannes, our designers, left on Monday for Build which is a design conference in Belfast. And yesterday CJ and Jonas flew to RubyConf in New Orleans were they will be giving the talk: The Front End Testing Frontier. CJ will also give a talk about Socialist Software Development.

On Tuesday we had a short but very appreciated visit by Dave Hoover, who flew by Gothenburg to see us and join Got.rb in the evening. Early on Wednesday he continued his journey by train to Malmö and Øredev Developer Conference.

A quiet day at the office.


The Elabs Warzone

Elabs has it’s own warzone – we call it the ping pong room.

When you walk past the warzone, and a game is in action, your ears will be filled by screams and shouts, both of agony and in triumph.

If you dare enter the warzone you will witness two Elabs employees in action. The ball moves at the speed of light and spins and twists like a crazy fly waiting for the inevitable outcome: the smash, followed by a cry of triumph and one of agony. This is deadly serious.

The ball lies in the hand, fingers gently squeezing it. The gaze between the two opponents is intense, both passing back and forth, like two wolves baring their fangs and showing of their muscles. Ready to jump at each other at any second. Then the ball is released, wet from perspiration, and in action. Soon another cry of triumph and one of defeat.

The warzone is calm and quiet, the game is over. We have a winner and a loser, returning to their computer for some more pair programming, both knowing that the next game can change everything. The current hierarchy is just temporary among the Elabs wolves.

Johannes in action


Speaking About Testing and Socialism at Mountain.rb and RubyConf

After attending and organizing it is finally time for the next step, to speak at a conference, two actually, with three talks. Dive in head first. CJ will be speaking at Mountain.rb and RubyConf.

First: Mountain.rb in Boulder, Colorado, October 6-8
The Talk: The Front End Testing Frontier

While most Ruby developers are very familiar with testing their code, frontend and JavaScript-testing is still a new frontier for many. This talk will show you how to easily write and run JavaScript integration tests with Capybara and Cucumber, and unit tests with Evergreen and Jasmine. The goal is to get you excited about frontend testing, and point you in the right direction to get started yourself!

Second and Third: RubyConf in New Orleans, November 11-13
Talk #1: The Front End Testing Frontier, extended, with co-presenter Jonas Nicklas
Talk #2: Socialist Software Development

Socialism is often portrayed as pure evil by US media (hello Fox News), yet many socialist countries are ranked as some of the best countries in the world (Newsweek). So maybe it's not all bad? If you look at job listings for software developers, it seem like a lot of companies are looking for "programmer rock stars", "coding ninjas", etc. There is a romantic notion about the ultra productive independent super developer. This talk examines software development from a socialist perspective.

Might we get better results and provide more value if we set aside our egos and work together?

CJ Kihlbom is speaking at Mountain.rb


Inside Information

A new side of Elabs is about to be revealed. You will be granted access to our office, to our thoughts, to the very core of our work. The intention is to share our everyday life.

First we want to welcome our newest addition, the Stockholm crew: Peter Marklund, Ingemar Edsborn and Dennis Rogenius. Ingemar and Dennis have worked with us before and we are very happy to announce them as official Elabs employees! Peter and CJ have known each other for several years. You will soon get to know them better, and the old Gothenburg crew, through a series of personal interviews which will be posted here on the blog.

The Stockholm crew is currently in the Gothenburg office for a two week long Elabs bootcamp. We will share knowledge, get to know each other and have fun. Pair programming, talking and playing ping pong. Soon we will hopefully recruit a fourth crew member as well. Our Stockholm headquarter will be located in our customer Streamio's facilities on Sveavägen. We are very happy to be sharing space with them!

We would also like to thank our client Alessandro, from Tinet, who was here last week working with us. Thank you for great pastries, fun conversations and productive work. Now he is back in a sunny Sardinia, Italy. We are just a little bit envious.

We hope you will enjoy the coming inside information!