In the airport after Lean Kanban Central Europe I was talking with Benjamin Mitchell and Jabe Bloom about one useful metric that I get from Kanbanery which is the task type distribution graph. I told them that after building over a hundred web apps with the same teams, hygiene I can look at a task type distribution pie chart and know in an instant whether the team is doing hallway tests. If they are, then the feature to bug ratio is less than one. If they aren’t, then there are 2-3 bugs per feature.
This team does hallway tests
This team doesn’t do hallway tests
Until we started incorporating the Kanban Method into our process, I was pretty happy with 2-3 bugs per feature. They were usually minor bugs, like display issues in IE or a misspelled word. After all, we use TDD and have cross-team code reviews and the guys are really experienced and professional, so it seemed we’d reached a sufficiently high level of quality. But then we started limiting our WIP and optimizing the system to improve flow. Suddenly, it became clear that these bugs were work, no matter how small, and they hung over the developers’ heads while they were trying to code, blocking items in the testing column and creating a bottleneck.
“The testing column is full. We need more testers,” the team said to me one day, just two weeks into our first experiment with limiting WIP, back in 2008.
“We could do that,” I answered, “but what else might we do? Let’s get a few options on the table before we start increasing costs.”
It turned out that almost all the bugs were IE issues. Why? Because web developers don’t like IE, so they all check their work in Chrome and Safari before marking it ready to test. What if they didn’t? A few of the bugs were just silly stuff, like a form missing an input button (how’s that supposed to work?) or typos. How could we keep those out of testing? Any second pair of eyes would catch those kind of things. The conclusion — get a Windows laptop and have someone else take a quick look at each feature before I goes to testing, like a sanity check. That’s called a hallway test, because you grab some poor slob from the hallway, just trying to get back to his desk from a coffee break, and make him sanity check your feature.
It’s not as easy as it sounds. Here are some hallway testing anti-patterns I’ve seen that you should look out for.
The Projectionist – In this scenario, the programmer grabs you and asks for a hallway test. Then he plops into his chair, grabs the mouse, and says, “Look at this. I click here, fill in this form, click SUBMIT, and then the results load here. Look good? Okay, thanks!” This doesn’t really tell the programmer anything about what a real user might do, or whether the implementation makes sense to someone who didn’t create it, or if the validation would catch inputs that he didn’t consider. He could have done this without the hallway tester and it would go the same way, without learning anything.
The Director – Now you get to sit at the computer and even touch the keyboard, but you don’t get to think, because there’s a voice in your ear and a looming presence over your shoulder as the programmer says, “Okay now click here, fill in that form and click that submit button there and see, there are your results. Look good! Okay, get out my chair, dude!” This isn’t much better. A hallway test done this way removes any possibility for misinterpretation of confusing UX or dealing with any of the inexplicable things that actual software users do when no one is watching them.
The Critic – So now you have a voice and real agency, but you’re an idiot. Sorry. You plop in the chair and try to figure out what to do. You click somewhere that makes sense and see a form. That’s easy. You fill in your name where it says to when a voice makes you jump. “Not your full name! There’s another line for your last name!” Yikes, now you see it and you keep filling in the form. You reach for the mouse, glide to the lower right-hand corner and stop, there’s no SUBMIT button. Isn’t that where it should be? You hesitate, getting nervous because you can sense the dissatisfaction your poor performance is causing to your hyper-attentive audience, but finally you find a submit text link just to the right of the form heading. You click and breath a sigh of relief. In a moment of boldness you stammer, “I was looking for the submit button down here.” You immediately regret your words, because now you learn that there’s a perfectly reasonable explanation for the misplaced submit link. It’s SUPPOSED to be there. I guess you just didn’t apply yourself. Oh well, thanks for the hallway test! In this scenario, the programmer might have learned something valuable, if he’d wanted to, but he didn’t. He wanted to get to the next task. He’ll see these issues again, but they’ll be in the form of bug tickets and user complaints.
Ideally, a hallway tester gets no prompting, no explanations. They try to use a feature as they see fit, and they are free to get confused, lost, and do whatever dumb things make sense to them within their own context. The programmer is watching, peacefully, quietly, re-assessing their own assumptions based on what they see with the goal of making any changes necessary to make the feature as easy to use as possible for the next guy, knowing that when the end user is in the same position as their hallway tester, the programmer won’t be there to help.
Comments on this entry are closed.