While split or multivariate testing is an extremely powerful tool, it should not be deployed as the sole method driving improvements to your online business. If testing is applied as the only way to determine the right path forward the outcome is certain to be less successful over the long term.
At the usability professional's association a few weeks ago in Portland, I chatted with Beverly Tseng @ Ebay and she recorded the essence on the whiteboard:
My premise is that testing is much more productive when it is aimed at hypothesis testing because the cumuluative outcomes of multiple tests create a body of knowledge about your business, your users, and their interaction. The next point in the white board is about triangulation. Analytics can provide strong statistically meaningful results, but making sure that rationalize a "why" is important. The general pattern here is that you can identify in the usability lab or other high resolution, low volume observation scenario and quantify with testing. Alternatively, you may observe in analytics and engage with customers in lab testing or other inquiry to understand.
My premise is that testing is much more productive when it is aimed at hypothesis testing because the cumuluative outcomes of multiple tests create a body of knowledge about your business, your users, and their interaction. The next point in the white board is about triangulation. Analytics can provide strong statistically meaningful results, but making sure that rationalize a "why" is important. The general pattern here is that you can identify in the usability lab or other high resolution, low volume observation scenario and quantify with testing. Alternatively, you may observe in analytics and engage with customers in lab testing or other inquiry to understand.
