There’s a lovely article about the movie GroundHog Day. The article talks about a lot of issues in the movie, but ends with the wonderful quote:
A/B testing is like sandpaper. You can use it to smooth out details, but you can’t actually create anything with it.
This thought reminds me of Don Norman’s comment that one of the risks for the field of CHI is that we become so focused on analysis that we never actually create anything new.