Benchmarks are overrated.

The first thing that you think of when you hear about benchmarks is probably CPU or GPU tests.... well, at least if you're a PC gamer.  This is an age old tool that we have used to compare similar products or even see where the older ones stack up to the latest and greatest.

For a longtime benchmarks have used video and processor intense tests to decide how each one handles what they throw at it.  Unfortunately most of the tests are not real world examples of what we see.  Sometimes the tests reveal strengths and weaknesses that you may never run in to.  If I only played WoW, but wanted to have a good experience then I would obviously look for scores that related more to me.  I play a lot of modern shooters as well, so I'd want to know what product gave me the best performance in that category.  We use these synthetic tests and expect them to represent the actual capability of these cards.

A lot of these tests don't even push the cards to the fullest.  They don't stress them to the point that they could be.  They do push them and test them, but they are rarely working in a real environment.  A fresh install of Windows with nothing but a benchmark application open isn't how most gamers run their set ups.  I know that I've got Steam open almost all of the time.  I've also got a copy of Ventrilo running.  In the case of playing BF3 I've got Origin AND Chrome open (Chrome can become quite the memory hog with a couple of extensions running).  If I'm playing WoW you can bet I've got Winamp on and I may have Chrome open to check out WoWHead and look up spots from transmog gear or whatnot.

The fact remains that we are still using this tired old method to tell us what is performing where.  While it is nice to know that the last generation is better than this one, I'm not sure it is as important as it once was.  There are many new features available now that we could have never dreamed of.  Realistic water, amazing smoke effects, the facial expressions, it's all very cool.  While the video cards might be "faster" does that mean that they are really doing much more?  nVidia has gone so far as to use the extra power on their GPUs to power a physics engine built into their drivers.  That's pretty cool and I'm not sure we have a benchmark to see what the offloading of physics can do to the overall experience.  Not to mention how much more in depth the experience can became just because of the way the physics is handled.  I can tell you right now that playing some games, especially those built on the Unreal Engine, can be drastically changed just by adding that extra physics power.

While I understand the importance of these tests, I'd like to see some headway made or some more standard tests developed to actually see what kind of REAL WORLD performance they give.  We have seen some rather disappointing numbers from these new sets of CPUs released, but are they really disappointing?  Maybe we just don't know how to measure the gain of their performance yet.  Once we start using more and more cores to push forward or the software gets better I am sure we will elevate these chips in terms of dominance.  Even dual core chips faced some problems at release because we just didn't know how to measure what it was they were capable of.