Sometime around the early 90's I started noticing that major companies like Microsoft and Oracle started adding provisions to the EULA's stating that the licenses forbade users from publishing benchmarks about the product. Evil Big Brother implications aside, I now am beginning to at least see why they think it is a good idea.

The fact of the matter is benchmarking today's web based computer software is almost impossible. In past years, the software was easier to isolate and the number of variables could be controlled so that a useful metric could be produced for buyers to be able to compare two similar products. Today however, when software is increasingly being deployed on the web, there are too many moving parts that have to be taken into consideration. In any modern web application web servers, database servers, third party APIs and the network all have to be factored in when benchmarking an application. Any one of these not functioning correctly could cause your benchmarks to report false data.