How I Used A / B Testing to Hack My Kids

“I wonder how we can get them to sleep more.”

This simple thought, expressed by my wife, not even a question, became a challenge to me. My engineer mind took this as a problem to be solved, and when a software developer sees a problem, they devise tests. Luckily, I knew the perfect system for testing out some ideas in a controlled and measurable setting. And with twins, testing would be even easier. Welcome to parenting, A/B testing style.

A/B testing is used all over the web. You likely encounter it dozens, if not hundreds of times a day, without even noticing it. All the big tech companies do it, using it as a tool to test the performance of ideas and measure them.

Google is famous for testing 41 shades of blue for search results. Designers allegedly couldn’t decide which of two shades to use, so they tested 41 in total to see which led to the more users clicking on the results.

Facebook tests different experiences within the feed constantly. Amazon even changes around the buy buttons and cart layouts fairly often. You may notice these if you ever log in from a new computer or see a friend using a site that looks subtly different from yours.

A/B testing is used to test one or more “treatments” or experiments over a “control” or the existing experience. A metric is measured, usually based on a user action such as a click through or “conversion” with a baseline against the control.

For the Google example, they might test the likelihood of users clicking through to at least one result with the different shade. After a statistically significant period of time, often a week or two, whichever experience has a better rate will be chosen as the winner and becomes the new control.

Where this gets really complicated is when multiple experiments are run at the same time or when the percentage of users is not equally split. Here a complicated knowledge of statistics is needed. Or the use of any of the many powerful testing tools available. At Audible and Amazon, we test experiences like this all the time. It’s the best way to see how users actually behave as often what users say they will do and what they do can be slightly different.

At Audible and Amazon, we test experiences like this all the time. It’s the best way to see how users actually behave, as often, what users say they will do and what they do can be slightly different.

Charting results

In any experiment, accurate measurement and data tracking are critical. Often a success metric is chosen due to the availability of data or measurability. You don’t want to be trying to measure something that takes longer to measure than it does to change the test or test input. Luckily measuring sleep is about as easy as it gets.

When they wake up at night, we just write it down. This is exactly what we’ve been doing since the day they were born as the nurses at the hospital instructed us. We’ve gone through several notebooks already, but it’s so easy to track. For this, we even started importing the data into a spreadsheet to see the impact more visually.

Next was a secret whispered about in the dark corners of parent blogs around the web and passed from parent to fellow parent, at least in my office, gripe water. Ok maybe it isn’t that much of a secret, but it took us a while to try it. Supposedly this mix of herbs and spices, as opposed to KFC’s blend, would settle stomachs from reflux and gas, especially overnight, resulting in longer sleep.

After a week of testing, we found it did actually help with reflux, especially spit ups, and though we didn’t track individual burps or farts, seemed to reduce them as well. The length of sleep was not impacted much, though. We did see a small increase on average, between 20 and 30 minutes, but again this may have been natural increases due to age.

No reflux equals a happy baby

Last, we tested keeping them awake longer during the day. Our hypothesis was that they would therefore, be more tired at night and would sleep longer as a result. This may have been slightly true, we saw minor increases in length of sleep, but we didn’t account for the stress and exhaustion it would cause by keeping them awake and making them unhappy. It also took significantly longer to get them to settle down and sleep at night as they were overtired and fussy. The lesson for testing: don’t sacrifice other metrics for a small gain in one.

This also shows the importance of the test, measure, iterate process. Though several of the methods didn’t show large improvements, put together they may. By using the treatment as the control when it outperforms the control, small improvements get stacked. By continuing to try new things quickly and moving on, it’s easy to come up with new ideas to try. You don’t need to move the mountain, just move little handfuls of dirt over a long time. With this approach to parenting, the boys can continuously grow as well. And with luck, so will our sanity, well-being, and lives as parents.

This piece was originally published on Dad On The Run.