First, and foremost, it seems clear to me that testing all games with maximum in-game settings - as we do currently - does not reflect how we may run the game at home. For instance some games look appalling without AA but others aren't too badly affected. So for the former game you would compromise some detail settings so you can have AA on but for the latter you would crank everything else up and leave AA off. The perfect example of this is Call of Duty 2, which still manages to give the latest and greatest cards a severe workout because the game is very texture heavy - and we run our tests with textures on maximum - but I'd happily compromise on texture levels to use AA with this game as I feel it needs it.
Therefore, I propose playing each game until I find what I believe are the settings most needed for a satisfying gaming experience, and which settings are less crucial. Then, I will come up with a high, middle, and low group of settings for each game and we will apply these settings when we run our timedemos. Therefore, when we test low to mid range hardware we can use the lower settings group to more accurately reflect how the card would be used by most people. This way, you still get your easy to read graphs, testing is consistent across different makes of card, and I don't have to sit at work playing computer games all day.
On top of this, there is one particular development of recent years that has been completely neglected in our testing; transparency anti-aliasing. Whereas, normal anti-aliasing only affects the edges of 3D objects, transparency AA is applied to the actual surface texture. This means that where partially transparent textures are used to create the illusion of a solid object (most commonly used for leaves on trees and wire fences) the edges of the internal objects are anti-aliased. The best way to show this is with an example.
Each section of the fence in this scene is actually just one large texture so, when normal AA is applied to the scene, none of the edges of the ironwork are anti-aliased. With transparency AA, these edges are smoothed out creating a far more realistic scene. With more and more games incorporating realistic foliage, the use of transparency AA will become increasingly important and it's something we shall be testing from now on.
So, that's what I think, now I want to hear what you think. Is there something I've missed? Do you like it the way it is? Or do you want to see a wholesale change to our methodology? Log onto the forums and let me know. After all, it's for your benefit.