The Emperor wears no clothes: An(other) editorial

[pHaestus]

Yesterday, Futuremark released an analysis of the NVIDIA driver “optimizations” we talked about last weekend. If you look at their statements, the recurring theme is that the drivers that were sent out with the GF FX5900 Ultra in its release:

DETECT WHEN 3DMARK SHADERS ARE CALLED FOR IN THE GAME

and then either

REPLACE THOSE SHADERS WITH THEIR OWN FASTER ONES or

USE TIMERS TO CLEAR BACKBUFFERS ONLY IN THE MOST EFFICIENT PLACES.

NIVIDIA was quick to respond:

“Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer… We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."

Am I the only one that is NOT convinced by this? Let me get this straight: YOUR drivers check for shaders by filename and then replace them with ones that don’t render properly but much faster and also start timing the demo so that you can avoid costly back-buffer clearing. And you have the audacity to blame Futuremark? I would assume if you were part of the beta test program then you would have signed some documents to pledge not to do BS like this...

We have seen similar “deny all responsibility” tactics used before in our community with great success. Just ask the banned folks over at www.xtremesystems.org forums about how much satisfaction they got from HardOCP’s apology when they pointed out they were changing benchmarks. Oh, wait, it never happened. Instead they got comments like you guys are just out to get us because we are so great and you are jealous of our manly muscles etc etc. Same exact PR tactic NVIDIA is using now. And most likely it’ll work again.

I checked out www.anandtech.com and www.hardocp.com and www.tomshardware.com and they are NOT coming to the same conclusion that I am. They are basically taking another page from the NVIDIA PR report and running with it: synthetic benchmarks are useless. Here’s something else from NVIDIA:

“This is obvious (that Futuremark is trying to make them look bad) since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.”

Unfortunately, they are avoiding the real issue. This isn’t about an unrepresentative test that might favor one card over another. I personally thought 3DMARK03 was bunk when it came out because it so markedly favored ATI cards. I got 11k 3dmarks in ’01 using a GF3 while 9500s got maybe 12k. In 3dmark03 I got 1250 while the 9500 got 3x that. How can this be representative for current buying decisions? But I digress. See! It’s easy and fun to change the subject. If I were getting free hardware I could just end it right there and go right back to scamming kids out of their money by calling $500 cards “must have” and “values”. Nah, instead I think I will return to THE POINT. The issue here is that NVIDIA KNOWINGLY AND WILLINGLY CHEATED WITH THEIR DRIVERS SO THEIR CARD WOULD LOOK FAST AT LAUNCH. And this absolutely kills every ounce of credibility they have in other benchmarks as well. If they took the time to optimize for 3DMark, what is to say they haven’t done the same thing for Unreal Tournament 2003 or Doom3? Did anyone else think it was fishy that they bundled the Doom3 benchmarks with the GFFX in the first place? How hard is it to cheat in these benchmarks compared to 3DMARK? Easier I would guess since there are no developer tools for you to use to check up on them. The image quality is so subjective and the timedemos go so fast that it is hard to know if what is being rendered in the demos is absolutely the same quality as what is being observed “in game”. We are forced to trust that manufacturers aren’t artificially cheating to make them look good. NVIDIA can't be trusted on THAT one obviously.

We are talking about people selling you a $500 card and using its performance to justify the cost. We are talking about a 24% boost in 3DMark03 that resulted from their cheats. We are told to take the other benchmarks (the most important one SUPPLIED by NVIDIA) as gospel instead. If their card is so clearly faster, then why did they have to cheat to compete in 3dmark? I usually think of the drama in the hardware community as being like a soap opera, but in this case I think fairy tale is more appropriate (the emperor wears no clothes!).

[ymboc]

Good Editorial.

It seemed to be getting awful quiet with no significant site really addressing the reported 3dmark detection schemes to my satisfaction... Most seemed to just gloss it over, ignore it or just write it off completely as futuremark pro-beta group propaganda -- which admittedly it may be in part.

I seem to be getting more and more dissappointed with the older established hardware sites out there... I thought AT would have at least addressed the futuremark report a little better.

Nice to read what I'm sure many of the more objective people were already thinking.

[Zhentar]

Another interesting column. But, I see one flaw in it- you've assumed Futuremark is being honest!

But that flaw just proves your point- You cannot trust any company in this industry, and thats definately a bad thing.

[bigben2k]

The PC industry is pretty crooked, and I speak from experience, both in retail and distribution.

This whole Graphics card thing has really turned me off the latest and greatest. the bottom line for me is that I'm not willing to spend more than $200 for a VC.

I can see clearly now that we're passed the performance point where the VC used to hold us back: we're in a phase where the VC can enhance a few of our gaming experiences, and that's just outside of what I do on a PC.

Just like the hard drive warranty fiasco, where the mainstream HDD's warranties fell to one year, we may very well be going through a market segmentation. Sure, we've always had the basic SiS or Trident cards, but now we're looking at three levels:
-the basic "it works" card
-the mid-level user (me)
-the gamer

Now we ought to get review sites to acknowledge this segmentation, so that the manufacturers can acknowledge it. Then again, we could always go on and pick up the crumbs they leave, such as me picking up an ATI Radeon 9700 pro. I have no problem buying a mature product, where all the drivers are tweaked out, and bugs/kinks worked out, since it's obvious that the manufacturers are only putting the gamers on the bleeding technological edge.

[pHaestus]

Well my decision regarding video cards is (at the moment anyway) to hold tight with my GF3 and not buy another card until one that will run HL2 and Doom3 with all the eyecandy is out and has dropped in the 150-200 range. This will probably mean that I have to wait a while after those games are released to buy them. I am assuming that they will be the same games nonetheless

I have been having a lot of fun playing older games anyway of late. It's easier to find a bunch of people in our chat with the hardware needed to run UT or Q3 or MOHAA than it is to find 5-6 people with 9700 pros who want to play a game.

[Zhentar]

This is from Anandtech:
"This is not to say the 5900 cards are perfect, either. Although the 5900 (NV35) chipset is not completely finalize, a slew of reviews have accused the video card giant of purposely inflating benchmarks by hard coding some acceleration into the chipset that inaccurately takes advantage of weaknesses in benchmark suites. While the evidence looks compelling, we cannot speculate this early on what is going on. We will most likely see a different “final” product and driver set that does not contain the same problems."

nVidia's got them, hook, line, and sinker.... I'm not thinking their really that gullible either.

[ezlid]

All benchmarks are flawed! It doesn't matter who developed the test, it will have some feature or code that will just run better or worse for a particular CPU or VC. There is no getting around that fact. When a manufacturer exploits that, is it really cheating? No, it's called marketing. Everyone does it. In PCs, autos, trucks, oil companies and the list goes on. We do it when we look for jobs, too. As consumers we have to make the intelligent choice when it comes spending OUR money by comparision shopping.

And remember - No one is selling a product because they like you! The bottom line is they want your money and we willingly give it to them because we don't or won't do the research.

As far a NVIDIA optimizing Futuremark, I don't have a problem as long as they TELL me that they modify the benchmark on the fly to work better. At least then I know that I am dealing with apples and oranges. I wonder if they do the same thing for off the shelf games too? That would impress me. If it is only for benchmarks, shame on them for not being open!

The bottom line for me is I don't trust benchmarks because they are never real-world. I would rather see testing that bypasses benches. Play the games, transfer files. whatever. Do the things normal users do and provide your impressions. Everyday users don't run benchmarks to work or play on their PC's, do they? So why should testing be any different?

Done ranting for now.

[pHaestus]

Sure there are plenty of ways to transform data and produce the same final image. I don't have a problem with a company optimizing drivers in this way; it's a valid way to get the most performance out of their hardware. That seems to be what ATI did in the 3dmark03 demo. But it is pretty clear that what NVIDIA did was something different: they changed the output to get better scores.

What would happen if I were testing heatsinks and told you that a $10 aluminum oem cooler was just as good as an Alpha PAL 8045? Well you probably wouldnt believe me, so I would post complete timestamped temperature logs from my diode reader. Ok why spend more money on the Alpha then right? Well what if it turned out that I neglected to mention that I was running the CPU at 2V and 2.5 GHz on the alpha and at 1.47 and 1.5V on the oem? That bit of info might have influenced your buying decision right? Same sort of deal. The performance values are inflated by doing shady things whose only purpose were to mislead.

[Blackeagle]

This whole 3Dmarks thing causes me to wonder about several things.
I've read a large number of people saying just what pHaestus did about the latest 3Dmark "03", "...was bunk". And for much the same reason as pHaestus gave. How can the 3D03 marks give a Nvidia card that scored quite closely with a Ati card in "01" only a score 30% of the Ati card when it was in the 90% range before? Given the fact that many people do use this benchmark as a major determining factor when buying a card, how could you not be upset if you were in nVidia's situation? I think nVidia was right to claim right off that the newest marks was BS. They have now, seems to me, to have shot themselves in the foot by altering their product to run the Marks. Tends to discredit what they started out saying.

The whole thing is a cluster ****.

Deciding on a card now, for me, is going to be based on what I read in forums regarding user experiance with them. Not based on speed, but on dependablity & other side issues. Why be concerned with the speed anymore? The frame rates, run by all the now leading cards, are above move rates.

Perhaps that last part is why these companys are trying so hard to make sure they look good running these benches. They themselves relize that it's getting harder to tell the differances in the cards just by reguler users day to day use.

All rather twisted and strange to watch.

[pHaestus]

Well the scores were so low for NVIDIA cards because they weren't DX9 compliant so the majority of the tests failed to run (and got 0 for score). The ATI cards with comparable OGL/DX7 and 8 performance to NVIDIA cards got much better scores because they ran every test.

[Blackeagle]

Thanks pHaestus,

But doesn't that tend to support the idea that the benchmark is biased?:shrug:

If it's set up to allow Ati's older cards to run then why didn't they allow for nVidia's as well?:shrug:

What are your thoughts on the quality of this benchmark pHaestus (beyond it's being bunk)? What should be done differantly to make for a fair(er) benchmark?

I'll gladly admit this is confusing.

Only thing I can see out of this, is go down and check out some running systems at some shops before buying. I've done so in the past for reasurance if nothing else. Then buy what ever is best priced that does what I want.

[pHaestus]

It's biased towards DX9 cards since it's a DX7,8, and 9 benchmark. Not necessarily purposely designed for ATI, and I am sure older cards w/o DX9 drivers from other brands do badly as well. As far as my opinion on the benchmark as a whole, I dunno. I actually find it fun to try and tweak my system to the max and reach a new goal by overclocking my existing hardware a bit further. It's a fun diversion from time to time. That to me has value. Am I going to base my buying decision only on 3dmark03? No way. The questions I have usually are from my previous experiences:

"Did ATI ever get truly stable drivers?"
"Did NVIDIA ever get GOOD image quality?"
"Is this card going to just ****ing blow me away compared to my GF3?"

That's pretty much it. My first step in buying a card is to determine my budget. If a card runs much hotter than another one then account for aftermarket cooling needs too. Then I would go check the rage3d forums before buying any ATI card and verify its driver goodness. I would ask around for honest opinions about 2d and 3d IQ regarding nvidia cards since I have been burned badly by that before. And THEN I would look at benchmarks from all the big hardware sites and see if I could get them to converge to some meaningful decision.

I am about 99% likely to buy an ATI card next though. I have been really happy with the Radeon 7000 in my HTPC and the mobility 7500 in my notebook. And I HATE companies trying to rip off consumers. Hate it.

[bigben2k]

FYI, build 330 of 3DMark03 should be "on the level".

[taoofbean]

What in the hell is up with [H], Anand and Tom's? How come they have not yet publicly shirked their fanboyism and made a stand on this issue? I just don't get it. I read all of these sites on occasion but this is just blatantly wrong in my mind. Ack, my head hurts with how poor it is that such blatant lies could be allowed to persist without derision.

[pHaestus]

Well it seems that Nvidia gave a NV35 to test at all the press representant that were invited at the Nvision 2 weeks ago (Cannes, France).

from here: http://www.beyond3d.com/forum/viewtopic.php?t=5776

There apparently are a lot more fringe benefits to being webmaster at one of the "big and trusted" sites than I ever imagined. Perks large enough to make one think twice about making an uproar? Who can say. What's that Ween song?

"Don't shit where you eat, my friend"

[gundog]

When assessing the Neutrality of a review:

A good start is to count the amount of advertising space a particular vendor has on the site in question, particularly in the graphics cards section.

[ymboc]

in related news...

Has anyone else read the latest BS from cetain formerly respected hardwares sites regarding futuremark's benchmarks, the futuremark-nvidia legal fiasco and cetain editorial oppinions...?

Unbelievable...

I humbly ask that anyone evaluating computer hardware to stop publishing the game demo scores taken from 3DMark03. The media is greatly responsible for giving FutureMark its power in the industry and it is my thought that if we work collectively, we can shift that power to a more responsible set of real world utilities that I suggest will become a reality this year.

Humbly my ass... Later in a thread in their forums, responding to a reviewer who said he deleted his 3dmark03 scores from his website in response to that request, Mr Humble himself had this to say...

First off, I did not suggest you go back and delete your scores, although I think it is a great idea and you should be patted on the back for not relying on them. I think for sure they should not be posted from this point out.

To me, it's sickening to see the sheep, err mice follow the pied piper so blindly...

[Phant0m51]

Futuremark Statement on NVIDIA and 3dMark 2003 - 7:07 pm EST - ben6
HardOCP has a statement from Futuremark on 3dmark 2003 and NVIDIA's apparent "cheating". Click the link below for the whole story. Here's a snippet:

Dear Member of the Press:
Following is a joint statement from Futuremark and NVIDIA relating to 3DMark03 and optimizations.

Futuremark Statement

For the first time in 6 months, as a result of Futuremark's White Paper on May 23rd, 2003, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark.

Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat .

I just read this at Rage3d.com That just blows my mind!