Experimenting in a Sociotechnical System

At Honeycomb, we have a company value: Everything is an Experiment. Sometimes people take that too literally, and try to make every UI change an A/B test, every process adjustment measurable by some number.

The scientific method is fantastic for debugging—and completely inadequate for the unique sociotechnical systems we each work in. Yet we do experiment (probe), and we can do this rigorously (for a more appropriate definition of “rigor”). How does this work?

We need different ways to frame experiments, choose the ones worth our time, conduct them, and recognize the outcome. We need new words to replace “failure” and “success.”

In complexity, everything is an experiment if we look at the result. That doesn’t mean objective measurement. It does mean careful observation and curiosity. Let’s talk about how we do this.