Blog


Apr

Steve Jobs on what's important in the development of a product


You know, one of the things that really hurt Apple was after I left John Sculley got a very serious disease. It’s the disease of thinking that a really great idea is 90% of the work. And if you just tell all these other people “here’s this great idea,” then of course they can go off and make it happen.

And the problem with that is that there’s just a tremendous amount of craftsmanship in between a great idea and a great product. And as you evolve that great idea, it changes and grows. It never comes out like it starts because you learn a lot more as you get into the subtleties of it. And you also find there are tremendous tradeoffs that you have to make. There are just certain things you can’t make electrons do. There are certain things you can’t make plastic do. Or glass do. Or factories do. Or robots do.

Designing a product is keeping five thousand things in your brain and fitting them all together in new and different ways to get what you want. And every day you discover something new that is a new problem or a new opportunity to fit these things together a little differently.

And it’s that process that is the magic.

— Steve Jobs, Triumph of the Nerds

Via 37signals and CNN Fortune Tech

Mar

Handle secret credentials in Ruby On Rails


This blog post aims to lay out a simple and concrete strategy for handling sensitive data in your Ruby On Rails applications, and to explain the importance of such a strategy.

Never, ever check them into source control

Even if your project is closed source and your trusted colleagues are the only ones with access, you never know when a freelancer or consultant might be joining the project. Even if that never occurs, how do you keep track of all the locations where that repository is checked out? Who knows on how many hard drives your company's credit card transaction secret API key might be stored. What happens when someone with a weak login password forgets their laptop on the bus or at the airport?

Also note that it's not always as simple as removing secrets after the fact, especially with version control. It's usually impossible to do this without drastically changing your entire project's history!

Do it right

For a long time, we've been using YAML files to store our application configuration. It's easy to manage and can be configured for different Rails environments. These YAML files could look like the following:

config/app.yml:

development: &defaults
  awesomeness_score: 3
  host: "localhost:3000"
  s3_bucket: "example-development-us"

production:
  <<: *defaults
  host: "example.com"
  s3_bucket: "example-production-us"

test:
  <<: *defaults

config/app_secret.yml.example:

  development: &defaults
  aws_access_key_id: ""
  aws_secret_access_key_id: ""

production:
  <<: *defaults

test:
  <<: *defaults

config/app_secret.yml:

development: &defaults
  aws_access_key_id: "ACTUAL-ID-WOULD-GO-HERE"
  aws_secret_access_key_id: "ACTUAL-SECRET-WOULD-GO-HERE"

production:
  <<: *defaults

test:
  <<: *defaults

Only the first two files would be checked in to source control, and the application's README would instruct developers to cp config/app_secret.yml.example config/app_secret.yml and fill in the gaps from the company keychain.

To make sure we never check in the secrets by mistake, we ignore the app_secret.yml file:

.gitignore:

# ...
/config/app_secret.yml

We then use the econfig gem written by Jonas Nicklas to easily merge them together:

Gemfile

# ...
gem "econfig", require: "econfig/rails"

config/application.rb

# ...
module YourApp
  extend Econfig::Shortcut
  # ...
end

Now we can access any configuration variable and secret credential:

YourApp.host # => "localhost:3000"
YourApp.aws_secret_access_key_id # => "ACTUAL-SECRET-WOULD-GO-HERE"

Deploy

When you deploy the application, you must manually manage the secrets on the server(s).

Capistrano

If you deploy with Capistrano, you'll want to place the app_secret.yml in your /shared folder. Once that's done, it can be copied to each release with symlink task:

deploy.rb

# ...
namespace :config do
  desc "Symlink application config files."
  task :symlink do
    run "ln -s {#{shared_path},#{release_path}}/config/app_secret.yml"  
  end
end

after "deploy", "config:symlink"

Heroku

If you're deploying your application where you don't have file access, such as Heroku, you're better off storing this kind of information in ENV. The econfig gem has built in support for this and a few other storage backends, but that's another blog post.

Conclusion

With this method, we now have a clear separation of sensitive and non-sensitive data. There's no risk of checking in any sensitive data, since we have only one place to put it all and it's hidden from source control. Data access within the application hasn't changed, and we no longer have to concern ourselves with how sensitive it is.

We can now be sure that giving access to a repository does not imply giving access to other systems.

Epilogue

If you have any feedback on how the blog post can be improved, or if you spot any errors, please let me know by posting a comment below!

Mar

How to write a bug report


This is a blog post I've wanted to write for a really long time. Every couple of days I get an issue reported to one of my open-source projects which looks something like this:

Capybara doesn't work

It's totally broken. I tried everything. Why doesn't it work? Please fix it for me!

Okay, maybe it's not quite that bad. As you might guess, such a bug report is completely useless, it's just a burden to everyone involved in the project to have to tackle this kind of stuff. Worse than that, the person who posted this isn't going to get the help they need, and their possibly very legitimate issue is not going to get fixed.

Every time I receive one of these I wish there were some kind of source I could point these people to, so they might learn how to contribute with insight to open-source projects, instead of being a burden. This is that source.

Closed source

This stuff only applies to open-source, free as in beer and speech projects. Closed source projects are a different game, though some of this advice may still be useful.

The holy grail of bug report templates

This is where you should start. I can't remember which project I first saw this template on, so unfortunately I can't give credit where it's due, suffice to say I didn't invent it and am not claiming credit for it.

Answer these three very simple questions:

  • What did you do?
  • What did you expect to happen?
  • What happened instead?

Just by answering these three questions, you're already doing better than most bug reports out there.

If you encounter an error, any kind of error, make sure to include a stack trace. Those are immensely useful. Also be sure that the stack trace is complete, some frameworks such as Rails or test frameworks like RSpec and Cucumber tend to filter the stack trace. In RSpec and Cucumber, run your tests with -b to get a full stack trace. If you can't get a stack trace, make sure to expand upon the third question as much as possible.

A bug report isn't a question

If you're asking why something broke, if you're wondering what the cause of a particular behaviour is, you aren't actually writing a bug report, you're asking a question.

Asking questions is good, and I encourage you to reach out when you need help and have exhausted all other options. Asking questions on an issue tracker however is bad. Issues need to be closed, they need to be resolved; a question cannot really be resolved. It can be answered, but then that answer needs to be acknowledged and accepted and so on. There is a lot of process involved in closing each issue and it takes time away from the maintainers to focus on more important things.

Most open source project have a mailing list, use that for asking questions. If there isn't one, use StackOverflow or a more generic mailing list. If you've posted to one of these and you didn't get an answer in two weeks or so, investigate again and see if you can't find the answer yourself. If you can find it, make sure to answer your own question in the forum you asked it, so that others who have the same question in the future may find your answer through the almighty Google. If all of that's failed, that's when you're allowed to open an issue for a question.

When you send a bug report, you should be able to at least speculate at what the cause might be. Saying feature X broke doesn't help at all. Most open source projects of decent size are used by thousands of people, they have massive, comprehensive test suites. The most basic functionality in the project very likely isn't broken. So even if you have a legitimate issue, it likely happened under a specific set of circumstances. You need to pare away possible causes until you are left with the most minimal set of conditions needed to replicate the issue. Which brings us to:

Replication

While it isn't strictly necessary for a bug report, it's immensely helpful to have some way to replicate the issue. It's especially helpful to have an executable way of replicating an issue.

I generally dislike sample applications, such as Rails applications for this. There are too many possible areas where breakage could occur, and it takes too long for me to understand how all the pieces fit together. If you need a complete sample application, you likely don't understand the issue well enough yet to send a good bug report. Pare it down further until you truly understand what components cause the problem.

The absolute best way is of course to send a failing test case. You definitely don't need to actually fix the problem, though of course we appreciate it if you do. Understanding a code base can be troublesome, so it's cool if you don't want to spend the time to understand how best to resolve an issue. But if you do spend the time to write a test case, you reduce the time it takes the maintainers to solve your problem by an order of magnitude. Seriously.

A well formatted email

If you're sending a message to a mailing list, be aware that reading code or stack traces inline in an email message is quite painful. Either keep the code samples very short or link to a pastie/gist or something. Reading long code samples without syntax highlighting and possibly with automatically wrapped lines is quite painful.

Debugging

Debugging is a central part of software development. You cannot get by as a programmer without being good at debugging. Use the skills you use to solve problems in your applications when you encounter what you perceive as bugs in open source projects. Debugging really is a simple process:

  1. Isolate the cause
  2. Fix it

Where 99.8%¹ of the time is spent on point 1. If you don't do the first point, you are effectively asking someone else to do most of your work for you. For free.

Be polite

The people who write open-source projects do not owe you anything. They've invested a lot of time, which you get for free, but they are under no obligation to keep providing that time to you, or to provide more of their time just because you demand it.

The most common form of this I see are the emails I get to my personal email account from people asking for help. Asking me for help in a private forum such as email is asking for a handout. It's a form of begging. Please don't do it. I hate getting these emails, and I hate having to reply that I don't answer questions sent to me privately.

Some people believe that instead, you owe authors for the software they provide, but I don't really believe that either. I think the only thing you do owe them is to treat them with the same respect and politeness that you would extend to anyone else.

So no one owes each other anything really, and that's cool.

Don't be disappointed by the way, if it takes a while for your issue to get a response. It's cool if you ping the author after a couple of weeks if nothing has happened. Sometimes one can lose motivation, or become engrossed in something else.

Summary

Be polite. Be explicit. Be as knowledgeable about the causes of your issue as you can be. Answer these following questions:

  • What did you do?
  • What did you expect to happen?
  • What happened instead?

¹ totally made up number

Mar

Streamio Marketing Site - the Design Process


I will explain my typical process when designing a marketing site by showing you examples from my work with Streamio.

Why Streamio? Because they are a fun company to work with, easy to cooperate with and they give clear and good feedback.

First I got a mockup from Streamio with some thoughts and wishes. It's actually rare that you get a mockup this good from the customer:

The mockup from Streamio

The mockup from Streamio, which I'm very impressed with. Sooner or later they won't need my help anymore :).

An important thing with a marketing site is to get the user's attention. This is done by giving enough information in a short amount of time, so that the user gets interested and keeps on reading further down. This means that we need to scale down the information, put it in a natural order and keep it simple and clear.

It's quite known how much, or little, we read on the internet. Studies show that we read very little, I myself is a typical example. We browse sites very fast, and it doesn’t take many seconds before we jump to the next one. This is something we have to think about when designing web sites, we only have a couple of seconds to catch the users attention, not much more exposure then a logo. Some interesting articles on the subject are UXMYTHS - People read on the web, How Users Read on the Web and E-Mail Newsletters: Increasing Usability.

The first thing I do in the design process is to look through the information on the customer's mockup. The second step is to make my own interpretations. My first thoughts on the Streamio mockup was that there was too much information and the headlines were unclear. On the following drawings I have scaled down the information and changed the layout in the at the top:

Draw with pencil on paper

I love to draw with pencil on paper - a great way to get your thoughts down at the beginning of the process.

Close-up on Streamio's mockup

Close-up on Streamio's mockup.

Final drawing of the design

My final drawing of the design, which I showed Streamio.

What's left is:

  • a short headline about Streamio: they help you upload and publish videos online.
  • then a picture of a computer and two devices with their video platform or video, to show that it works on all modern platforms.
  • followed by 3 short blurbs about their advantages.
  • the 3 blurbs are then put into one sentence to make it even more clear.
  • a prominent Call-to-Action button makes it easy for the user to try the service.
  • finally there is a price tag with the minimum price.

When I'm done I show the customer the new drawings. This time Streamio was happy with the result, so then I continue with the next step: the real designing.

I design directly in the web browser, which is most efficient today. This way you work closer to the end result and you don't have to work in two steps, like when you're using Photoshop. I do additional graphical elements in Photoshop, but I constantly look at the end result in the web browser. Another great advantage with working in the web browser is that you have access to A LOT of fonts from @font-face services, such as Typekit. Today you can do a lot with the design using html and css3, which I'm sure you already know, but if you want to read more about it you can take a look at Smashing Magazine or NetTuts.

Two fonts

Two fonts used on Streamio's site. The first one is the popular serif FF Tisa which I think is great! It is used for all the content on the site. The second one: FF DIN Pro Condensed, is used for the logo.

First layout

This is my first layout after the drawings. Here I have made the basic layout and placed the content, and made some screenshots and additional graphics. But it is a bit plain and boring.

Streamio likes a very sharp and glossy style, and I won't say no to that. A lot of sites can't pull this of, but I think it gives a great vibe and harmony to this site and it works with Streamio's kind of service.

Some screenshots from the final site:

Final site 1

Final site 2

Final site 3

I really like the last page because it's only built with three images. It's a fun challenge to try to use as few images as possible. This keeps the loading time for the site short, or rather compensate for custom fonts.

Streamio launched their site on March 7th. It looks a bit different from my final version because they combined some of the final design with some of the earlier to create what they want.

Take a look at Streamio.com.

In my next blogpost I will write about the design process behind the actual video tool which David at Streamio has developed and I've made the design for.

If you have any comments or questions you can text me on twitter: @leuchovius, thanks!

Jul

Continuous Integration Testing for Ruby on Rails with Integrity


Doing test-driven development usually means you have a lot of tests in a project. While this is almost entirely a good thing, running the thousands of Cucumber features and RSpec examples in a large project takes a couple of minutes. If you run your entire test suite every time you commit this will easily eat up a large chunk of your day. Offloading some of this to a continuous integration server will allow you to save time by running your tests asynchronously, in addition to its other benefits.

At eLabs we usually run our unit tests locally—as well as the Cucumber feature for the story we're currently working on—before checking in. Then we let our CI server run the rest of our Cucumber features and notify us if something goes wrong. Here's the setup we use:

Integrity

At eLabs we've looked at a number of different CI servers, such as CruiseControl.rb and Run Code Run, but our favorite by far is Integrity.

Screenshot of our Integrity site

Integrity suits us perfectly. It fetches our code from our private GitHub repositories, can run any testing command and notify us in a variety of ways such as email and Campfire. It also has a very nice and clean interface. Its one major shortcoming is its complete lack of error reporting. If there's something wrong with your setup it will silently fail, which makes troubleshooting a nightmare. Hopefully the instructions below will help you avoid some of the pitfalls.

Installation

We installed Integrity on a server running Mac OS X and Passenger under Apache. Here's a quick guide.

First we installed the gem:

$ sudo gem install integrity

Then set it up in your chosen directory using the --passenger option:

$ integrity install --passenger /Library/WebServer/Sites/integrity

Next, set up a virtual host in Apache, pointing its DocumentRoot to the public folder in your Integrity installation.

DocumentRoot "/Library/WebServer/Sites/integrity/public"

One absolutely crucial step that we missed at first is to make sure that the system user that runs the Integrity passenger processes has git in its PATH. The simplest way to do this is to set the PATH in the virtual host configuration:

SetEnv PATH /opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

After configuring Apache you have to configure Integrity by editing config.yml in the root directory of your Integrity installation. We used SQLite for the database (couldn't get it to work with MySQL). If you want to use a hash password for the admin user, here's a simple way to get the SHA1 of a password:

$ ruby -r 'digest/sha1' -e 'puts Digest::SHA1.hexdigest("password")'

The final step is to create the database:

$ integrity migrate_db

You should now be able to log in to your Integrity site and add your projects.

Setting Up a Project

The most important part of setting up a project for CI is the build command. This is the command that Integrity runs to test your app, and it can be anything that exits with a status of 0 when successful. We use a simple rake task that prepares our project by copying a database.yml file and runs RSpec and Cucumber tests.

namespace :ci do
  task :copy_yml do
    system("cp #{Rails.root}/config/database.yml.ci #{Rails.root}/config/database.yml")
  end

  desc "Prepare for CI and run entire test suite"
  task :build => ['ci:copy_yml', 'db:migrate', 'spec', 'features'] do
  end
end

With that committed to our repository (along with a database.yml.ci file) we add the project to Integrity. The important parts here are the Git repository and Build script settings.

Add a project to Integrity

You must also make sure that the Integrity user can access your repository on GitHub. There are a couple of different ways you can do this, but we created a separate free GitHub account that we add as a collaborator to our projects.

After you add the project you should be able to request a manual build from the Integrity web interface. Note that the build is done synchronously—so you'll have to wait a while—but if the build succeeds you're ready to set up the Post-Receive hook for GitHub to have Integrity run your tests whenever you push your code to GitHub.

GitHub Post-Receive URL settings

Go to your project's page on GitHub and click the Admin link in the top menu, and then Service Hooks in the sub menu. Enter the push URL for your Integrity project as Post-Receive URL. The URL has the following format:

http://username:password@hostname/project-name/push

After you've updated the settings, click the Test Hook link and Integrity should start a new build. If that works, you're all set for having automated builds on every push to GitHub.

Notifiers

While Integrity's interface is nice, you probably don't want to visit your Integrity site after every commit to check the status of your build. The point of asynchronous tests after all is to get notified when somethings goes wrong. Integrity has a bunch of different notifiers you can use. We use the ones for email and Campfire. Find more and installation instructions on the Integrity site.

In addition to Integrity's own notifiers we also use CCMenu, a Mac OS X Menu extra built for showing CruiseControl build status. It works with Integrity as well with the gem integritray.

We also use GitHub's Campfire service hook that posts a message to our Campfire room every time someone pushes new code. This makes it very easy to keep track of what other people in the company are working on.

Campfire screenshot

Not having to wait for our entire test suite to run before each commit saves us a lot of time. But we can still feel confident knowing that Integrity has our backs and will alert us if something goes wrong.