Using data for product development by Steven Sinofsky (Andreessen Horowitz)

Want to know how successful companies like Microsoft got to where they are today? By using data in smart, innovative ways.

In this interview with Editor-in-Chief of ReadWrite, Owen Thomas, Steven Sinofsky draws on his experience at Microsoft and Andreessen Horowitz to tell you how to use data to drive product development. This fun and engaging interview will teach you how to use data, what pitfalls to be aware of and how to align customer support and product development.

Strategize to Materialize: using data in product development

Editor-in-Chief of ReadWrite, Owen Thomas, interviews Steven Sinofsky (board partner at Andreessen Horowitz) about everything from his time at Microsoft to the importance of data in product development.

Owen: When you were at Microsoft creating things like Word and Excel, how did you first start extracting data from your software product?

Steven: Back in the early days, we would develop products by writing all our ideas down on a whiteboard and then yelling at each other. Whoever yelled the loudest for the longest, would get their product made. I called it Testosterone Based Development. It was a disaster.

Without any data, you end up developing products for yourself. But software developers aren’t the target audience for things like word processors, so we decided to learn from how our customers actually used the product. Unfortunately, with complex products like Word, you can’t just ask people how they use them because your customers don’t know how to answer that question in a way that is helpful.

To collect useful data, we started an experiment. We made a special version of Word that had all of these measurement points in it. Because this was the 80s and no one had internet, we would go to people’s offices, install our special version and then a month later go back and collect the data.

Once we had hard data, we started generating reports on the usage of popular features to decide what to develop next. We realized that people were not using our products the way we thought they would.

Owen: In the 90’s when Microsoft bought Hotmail, how did that change your operations?

Steven: All of a sudden we had to become an Internet-based organization and that changed everything. I’d never even seen a data center before! It was challenging, but it opened up a whole world of new possibilities.

For example, one of our developers on Word was having trouble finding bugs in the software. When he saw how Hotmail was finding bugs, he realized that every time someone’s Word crashed he could send that data directly to Microsoft. This instant data access meant he could debug things right when they crashed and not six months down the line.

Before that, if your Word crashed and you phoned up the support team, you literally had to read out the crash report over the phone and someone on our end had to log that manually.

As soon as we turned this crash report on, we start getting 10s of millions of these reports streaming into Microsoft. And while that was extremely humbling from a development point of view, it was also extremely useful because we had so much more information to work with.

It also allowed us to react quicker to problems. For example, one of our vendors released an update that didn’t work and because of these crash reports we knew as soon as it happened. So we picked up the phone, told them about it and they were able to fix it.

Instant data access literally revolutionized the way we did product development. We knew what features you used, how they performed and how that differed by region and user type. We could then use that information to drive product development and quality.

By watching the flows, we could optimize the usage pattern and make the products better. At the time, it was incredibly novel, but it fundamentally changed the way we thought about our products and how we developed them.

Owen: Your blog is called Learning by Shipping. What do you mean by that phrase?

Steven: Learning by shipping means that all of your up front plans really don’t matter until you actually get your product into people’s hands and they use it.

That’s the incredibly difficult challenge that everyone who builds software faces. You want to be done with your product, but you also want it to be right. But it can’t be right until people use your product that’s not yet done.

You also have to reevaluate what “done” even means. You’re never done in software development anymore. You used to go to a store and buy a CD with some software and expect it to ‘just work’. It’s not like that anymore. People expect products to change and adapt to their evolving needs in addition to being robust and working and all the rest of it.

Owen: You’ve learned a lot about how to use data. What have you learned about the misuse of data?

Steven: One of the things that happened at Microsoft was that if you didn’t like someone’s idea you’d say: Where is the data to back that up? Of course, they didn’t have any yet because it was just an idea. Unfortunately, we got into this cycle where if you didn’t have data to prove it was a good idea it just didn’t get built. Which grinds everything to a halt and stops you from innovating.

When I moved over to the Windows team they were obsessed with A/B testing. Every meeting ended with: Let’s just do two things and see which one works better. There was a weird stigma around using your intuition. Which doesn’t make sense, because Apple were releasing products that no one had ever heard of let alone asked for and consumers were loving it. Somebody with intuition was coming up with those ideas.

When you lack a big vision, you start optimizing small little things. I think too much data, or too much reliance on data, can make you so focused on tiny details that you lose sight of the big picture.

The challenge with data is that it looks like it’s right. Even when it isn’t. You have to be thinking critically about what the data tells you all the time. If you’re paid to be a product manager you have to have a vision and then execute on it. Use data along the way, absolutely, but don’t let it drive your direction.

Owen: Is Twitter the new version of focus groups?

Steven: It depends on your business. If the people who use Twitter and the people who use your product are similar, then yes, it’s a great source of data. But as your user base expands there’s going to be a large section of them that aren’t on Twitter and you’re going to have to find a new avenue of talking to those guys.

As a product manager, you’re going to have a lot of different sources of data, and they’re not always going to agree with each other. The key is to learn how to manage all those different types of feedback and prioritize effectively.

The more data you have, the more you need to rely on intuition because your data won’t give you a definitive answer.

Owen: How do you feel about this trend towards putting customer support and product development together?

Steven: Customer support should be a major source of data for anyone in product development.

We were doing this at Windows even back when support was all on the phone or in person. The best thing we did was to take all that customer support data and set up an internal website that was accessible to everyone on the company.

Any time you wanted to know something about your product, you could go to this site and do your own deep dive and analysis on what we had.

Owen: You recently joined the board of ProductHunt, which is a very early stage product discovery site. What does it mean that products are evolving with their users from day 1?

Steven: The best thing about ProductHunt is that it’s an environment where the developers can get positive feedback about what they’ve built.

Instead of hearing about the 9 million features your product is missing, you can start conversations with people about what made you create that product in the first place. You can also get validation for your idea through upvotes and links.

ProductHunt creates a much smaller community than something like the app store where you can get eyeballs on your app to see if people will be interested in it. It gives you both the qualitative and the quantitative feedback that you need when you’re first developing a product from people who understand and are interested in product development.