His “list of things I’ve discovered over the years” is particularly good. Here are a few of the things that resonated with me:
The more you can work with someone to help refine their question the easier it will be to answer Conducting a randomized controlled experiment via an A/B test is always better than analyzing historical data Metrics are crucial to the story a company tells itself; it is essential to honestly and rigorously define them Good experimental design is difficult; don't allow a great testing framework to let you get lazy with it Data science (A/B testing, etc.) can help you how to optimize for a particular outcome, but it will never tell you which particular outcome to optimize for Always seek to record and attain data in its rawest form, whether you're instrumenting something yourself or retrieving it from an API
I highly recommend reading the whole post.