Mob Debugging

Here at Silverfin, we encourage pair programming for knowledge sharing, instant code reviews, and overall better code quality.

There have been occasions of mob programming as well when we got together on-site or during a retreat. Mob programming is like pair programming, but with a whole team that works on the same thing, at the same time, in the same space, and at the same computer.

But how about mob debugging?

Setting the stage

We've been plagued by "out of memory" issues (OOMs) that were hard to debug for a while. Our documentation has a "HOWTO: debug out of memory issues," but it does not seem to suffice as many people still do not feel comfortable handling these.

This seemed like a perfect problem to experiment with mob programming debugging.

The set-up:

• all "experts" that know how to fix these can only observe and not talk
• we need one voluntary driver, with hands on the keyboard
• everyone else is a "navigator", helping the driver steer to the correct solution.

In general, it was a great way of assessing our (perceived) shared knowledge. It was a lot of fun with many learnings. And it differed depending on the role.

The driver

Most of all, the driver had fun experiencing a more active kind of programming complete with interaction. As the driver, there was a lot of listening and looking at other people's thoughts. Trying to understand how people think through an issue and understand how different skillsets and codebase knowledge interact.

Being the driver is having a lot of very excited people asking you to try a solution. It can be overwhelming if there's no consensus, so there's a bit of moderation involved.

Navigators

In general, it was a great way of shining a light into the nadirs of knowledge we have as an institution, as well as a way of assessing the myths and legends that may come up.

In our case, in particular, it's clear that the shared understanding of how to do something was way off.

It might have been due to the context and setting. Still, the solution was often staggered by being distracted after every paragraph read in the HOWTO. Though you could argue that this says something about the actual documentation.

Experts

One thing we didn't do from the start, but would do next time: the experts should move to another room, being "invisible". Even when silent, navigators will be looking at your facial expressions, sighs, and giggles, which can make them feel uneasy.

Maybe one of the most exciting learnings was for the experts. Many misunderstandings came forth from different mental models about infrastructure and apps and jargon in general.

Not everything obvious to person A will be evident to person B. That sounds like a given, but when writing documentation or HOWTOs, it's easy to assume things incorrectly and leave a lot of blind spots.

Conclusion

The driver and navigators got immersed into the world of OOMs, and I'm sure they can now debug and fix them appropriately.

All relevant dashboards, monitors, and documentation were updated to reflect the general mental model of our app and OOMs.

It was a fun endeavor and one I think is worth repeating on a future retreat. :-)