Ronny Kohavi and I started the Experimentation Platform Team at Microsoft in 2006.  This team is focused on making data driven decisions easier by buidling a platfrom,  a platform for running online controlled experiements and analyzing the data correctly.   You can find out more about the team at http://exp-platform.com/

On this team, we work very hard to do things right.  We've assembled a team of some of the best people in the industry with years of experience in experimentation, data mining, statistics and software engineering.  There are many companies out there focused on A/B testing, split testing, multi-variable testing (sometimes called multi-variate testing which is actually something else).  These companies promise you the world making bold claims about their techniques.  But you have to be careful about these wonderful claims . . . if something sounds too good to be true it probably is. 

We are taking the approach of publishing our methods, lessons learned and best practices, not in marketing white papers but peer reviewed journal articals.  For example:

Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. Ron Kohavi, Randal M. Henne, Dan Sommerfield

Appears in KDD 2007.

In this paper we talk about real nuts and bolts issue of running online experiements.  There are a couple of tricky steps and we do our best to detail what we have learned.  We also talk about the highest paid person’s opinion the meaning of the term Hippo or HiPPO.  It was because of this cute little term that I had to clear England and Dublin customs with a suitcase full of little rubber Hippos with our team logo on the back.  I'm sure the customs folks were wondering "what kind of freak did we just let in the country?" ;-)

We also realize that at least half the battle of moving towards data-driven decisions is cultural.  People fundamentally believe that they can come to the right answer about building customer facing software by thinking hard and formal design process.  All data points to the fact that people's intuition is actually quite poor at these problems.  So we can build tools and techniques but we also have to get people to use them.  And I think that is fun too.  That's why I love my job.  I'm working on great scientific and engineering problems with some of the smartest people on the planet.  I also get to work with teams to help them or even better try to convince them to run experiments. 

Now we are expanding our development to Microsoft Ireland Research.  I'm going to be running a development effort there.  If you are interested in experimentation, data mining and data driven decisions check out our jobs at http://http://www.joinmicrosofteurope.com/

See me on linkedin