Here is a good post from Munjal Shah , the CEO and co-founder of like.com, where he talks about trying to get their heads around the metrics of their new site after launch:
“Launching a new site is like becoming the owner of a brand Russian nuclear power plant. You have a ton of dials with labels you can’t read. The only dial you can read is the amount of electricity (in our case revenue) and the temperature of the nuclear core (in our case number of clicks you are sending to merchants). ”
That’s a nice analogy!
In their case, they used a form of A/B testing to better understand the impact of the changes they were making.
What are you doing?
Whatever it is, make sure you base your decisions on facts not opinions.