Blog

Mar

How to write a bug report


This is a blog post I've wanted to write for a really long time. Every couple of days I get an issue reported to one of my open-source projects which looks something like this:

Capybara doesn't work

It's totally broken. I tried everything. Why doesn't it work? Please fix it for me!

Okay, maybe it's not quite that bad. As you might guess, such a bug report is completely useless, it's just a burden to everyone involved in the project to have to tackle this kind of stuff. Worse than that, the person who posted this isn't going to get the help they need, and their possibly very legitimate issue is not going to get fixed.

Every time I receive one of these I wish there were some kind of source I could point these people to, so they might learn how to contribute with insight to open-source projects, instead of being a burden. This is that source.

Closed source

This stuff only applies to open-source, free as in beer and speech projects. Closed source projects are a different game, though some of this advice may still be useful.

The holy grail of bug report templates

This is where you should start. I can't remember which project I first saw this template on, so unfortunately I can't give credit where it's due, suffice to say I didn't invent it and am not claiming credit for it.

Answer these three very simple questions:

  • What did you do?
  • What did you expect to happen?
  • What happened instead?

Just by answering these three questions, you're already doing better than most bug reports out there.

If you encounter an error, any kind of error, make sure to include a stack trace. Those are immensely useful. Also be sure that the stack trace is complete, some frameworks such as Rails or test frameworks like RSpec and Cucumber tend to filter the stack trace. In RSpec and Cucumber, run your tests with -b to get a full stack trace. If you can't get a stack trace, make sure to expand upon the third question as much as possible.

A bug report isn't a question

If you're asking why something broke, if you're wondering what the cause of a particular behaviour is, you aren't actually writing a bug report, you're asking a question.

Asking questions is good, and I encourage you to reach out when you need help and have exhausted all other options. Asking questions on an issue tracker however is bad. Issues need to be closed, they need to be resolved; a question cannot really be resolved. It can be answered, but then that answer needs to be acknowledged and accepted and so on. There is a lot of process involved in closing each issue and it takes time away from the maintainers to focus on more important things.

Most open source project have a mailing list, use that for asking questions. If there isn't one, use StackOverflow or a more generic mailing list. If you've posted to one of these and you didn't get an answer in two weeks or so, investigate again and see if you can't find the answer yourself. If you can find it, make sure to answer your own question in the forum you asked it, so that others who have the same question in the future may find your answer through the almighty Google. If all of that's failed, that's when you're allowed to open an issue for a question.

When you send a bug report, you should be able to at least speculate at what the cause might be. Saying feature X broke doesn't help at all. Most open source projects of decent size are used by thousands of people, they have massive, comprehensive test suites. The most basic functionality in the project very likely isn't broken. So even if you have a legitimate issue, it likely happened under a specific set of circumstances. You need to pare away possible causes until you are left with the most minimal set of conditions needed to replicate the issue. Which brings us to:

Replication

While it isn't strictly necessary for a bug report, it's immensely helpful to have some way to replicate the issue. It's especially helpful to have an executable way of replicating an issue.

I generally dislike sample applications, such as Rails applications for this. There are too many possible areas where breakage could occur, and it takes too long for me to understand how all the pieces fit together. If you need a complete sample application, you likely don't understand the issue well enough yet to send a good bug report. Pare it down further until you truly understand what components cause the problem.

The absolute best way is of course to send a failing test case. You definitely don't need to actually fix the problem, though of course we appreciate it if you do. Understanding a code base can be troublesome, so it's cool if you don't want to spend the time to understand how best to resolve an issue. But if you do spend the time to write a test case, you reduce the time it takes the maintainers to solve your problem by an order of magnitude. Seriously.

A well formatted email

If you're sending a message to a mailing list, be aware that reading code or stack traces inline in an email message is quite painful. Either keep the code samples very short or link to a pastie/gist or something. Reading long code samples without syntax highlighting and possibly with automatically wrapped lines is quite painful.

Debugging

Debugging is a central part of software development. You cannot get by as a programmer without being good at debugging. Use the skills you use to solve problems in your applications when you encounter what you perceive as bugs in open source projects. Debugging really is a simple process:

  1. Isolate the cause
  2. Fix it

Where 99.8%¹ of the time is spent on point 1. If you don't do the first point, you are effectively asking someone else to do most of your work for you. For free.

Be polite

The people who write open-source projects do not owe you anything. They've invested a lot of time, which you get for free, but they are under no obligation to keep providing that time to you, or to provide more of their time just because you demand it.

The most common form of this I see are the emails I get to my personal email account from people asking for help. Asking me for help in a private forum such as email is asking for a handout. It's a form of begging. Please don't do it. I hate getting these emails, and I hate having to reply that I don't answer questions sent to me privately.

Some people believe that instead, you owe authors for the software they provide, but I don't really believe that either. I think the only thing you do owe them is to treat them with the same respect and politeness that you would extend to anyone else.

So no one owes each other anything really, and that's cool.

Don't be disappointed by the way, if it takes a while for your issue to get a response. It's cool if you ping the author after a couple of weeks if nothing has happened. Sometimes one can lose motivation, or become engrossed in something else.

Summary

Be polite. Be explicit. Be as knowledgeable about the causes of your issue as you can be. Answer these following questions:

  • What did you do?
  • What did you expect to happen?
  • What happened instead?

¹ totally made up number

blog comments powered by Disqus