Pro/Forums - View Single Post - The Emperor wears no clothes: An(other) editorial

pHaestus · 05-24-2003, 11:55 AM

Yesterday, Futuremark released an analysis of the NVIDIA driver “optimizations” we talked about last weekend. If you look at their statements, the recurring theme is that the drivers that were sent out with the GF FX5900 Ultra in its release:

DETECT WHEN 3DMARK SHADERS ARE CALLED FOR IN THE GAME

and then either

REPLACE THOSE SHADERS WITH THEIR OWN FASTER ONES or

USE TIMERS TO CLEAR BACKBUFFERS ONLY IN THE MOST EFFICIENT PLACES.

NIVIDIA was quick to respond:

“Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer… We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."

Am I the only one that is NOT convinced by this? Let me get this straight: YOUR drivers check for shaders by filename and then replace them with ones that don’t render properly but much faster and also start timing the demo so that you can avoid costly back-buffer clearing. And you have the audacity to blame Futuremark? I would assume if you were part of the beta test program then you would have signed some documents to pledge not to do BS like this...

We have seen similar “deny all responsibility” tactics used before in our community with great success. Just ask the banned folks over at www.xtremesystems.org forums about how much satisfaction they got from HardOCP’s apology when they pointed out they were changing benchmarks. Oh, wait, it never happened. Instead they got comments like you guys are just out to get us because we are so great and you are jealous of our manly muscles etc etc. Same exact PR tactic NVIDIA is using now. And most likely it’ll work again.

I checked out www.anandtech.com and www.hardocp.com and www.tomshardware.com and they are NOT coming to the same conclusion that I am. They are basically taking another page from the NVIDIA PR report and running with it: synthetic benchmarks are useless. Here’s something else from NVIDIA:

“This is obvious (that Futuremark is trying to make them look bad) since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.”

Unfortunately, they are avoiding the real issue. This isn’t about an unrepresentative test that might favor one card over another. I personally thought 3DMARK03 was bunk when it came out because it so markedly favored ATI cards. I got 11k 3dmarks in ’01 using a GF3 while 9500s got maybe 12k. In 3dmark03 I got 1250 while the 9500 got 3x that. How can this be representative for current buying decisions? But I digress. See! It’s easy and fun to change the subject. If I were getting free hardware I could just end it right there and go right back to scamming kids out of their money by calling $500 cards “must have” and “values”. Nah, instead I think I will return to THE POINT. The issue here is that NVIDIA KNOWINGLY AND WILLINGLY CHEATED WITH THEIR DRIVERS SO THEIR CARD WOULD LOOK FAST AT LAUNCH. And this absolutely kills every ounce of credibility they have in other benchmarks as well. If they took the time to optimize for 3DMark, what is to say they haven’t done the same thing for Unreal Tournament 2003 or Doom3? Did anyone else think it was fishy that they bundled the Doom3 benchmarks with the GFFX in the first place? How hard is it to cheat in these benchmarks compared to 3DMARK? Easier I would guess since there are no developer tools for you to use to check up on them. The image quality is so subjective and the timedemos go so fast that it is hard to know if what is being rendered in the demos is absolutely the same quality as what is being observed “in game”. We are forced to trust that manufacturers aren’t artificially cheating to make them look good. NVIDIA can't be trusted on THAT one obviously.

We are talking about people selling you a $500 card and using its performance to justify the cost. We are talking about a 24% boost in 3DMark03 that resulted from their cheats. We are told to take the other benchmarks (the most important one SUPPLIED by NVIDIA) as gospel instead. If their card is so clearly faster, then why did they have to cheat to compete in 3dmark? I usually think of the drama in the hardware community as being like a soap opera, but in this case I think fairy tale is more appropriate (the emperor wears no clothes!).

05-24-2003, 11:55 AM	#1
pHaestus Big Player Making Big Money Join Date: Aug 2001 Location: irc.lostgeek.com #procooling.com Posts: 4,782	The Emperor wears no clothes: An(other) editorial Yesterday, Futuremark released an analysis of the NVIDIA driver “optimizations” we talked about last weekend. If you look at their statements, the recurring theme is that the drivers that were sent out with the GF FX5900 Ultra in its release: DETECT WHEN 3DMARK SHADERS ARE CALLED FOR IN THE GAME and then either REPLACE THOSE SHADERS WITH THEIR OWN FASTER ONES or USE TIMERS TO CLEAR BACKBUFFERS ONLY IN THE MOST EFFICIENT PLACES. NIVIDIA was quick to respond: “Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer… We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad." Am I the only one that is NOT convinced by this? Let me get this straight: YOUR drivers check for shaders by filename and then replace them with ones that don’t render properly but much faster and also start timing the demo so that you can avoid costly back-buffer clearing. And you have the audacity to blame Futuremark? I would assume if you were part of the beta test program then you would have signed some documents to pledge not to do BS like this... We have seen similar “deny all responsibility” tactics used before in our community with great success. Just ask the banned folks over at www.xtremesystems.org forums about how much satisfaction they got from HardOCP’s apology when they pointed out they were changing benchmarks. Oh, wait, it never happened. Instead they got comments like you guys are just out to get us because we are so great and you are jealous of our manly muscles etc etc. Same exact PR tactic NVIDIA is using now. And most likely it’ll work again. I checked out www.anandtech.com and www.hardocp.com and www.tomshardware.com and they are NOT coming to the same conclusion that I am. They are basically taking another page from the NVIDIA PR report and running with it: synthetic benchmarks are useless. Here’s something else from NVIDIA: “This is obvious (that Futuremark is trying to make them look bad) since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.” Unfortunately, they are avoiding the real issue. This isn’t about an unrepresentative test that might favor one card over another. I personally thought 3DMARK03 was bunk when it came out because it so markedly favored ATI cards. I got 11k 3dmarks in ’01 using a GF3 while 9500s got maybe 12k. In 3dmark03 I got 1250 while the 9500 got 3x that. How can this be representative for current buying decisions? But I digress. See! It’s easy and fun to change the subject. If I were getting free hardware I could just end it right there and go right back to scamming kids out of their money by calling $500 cards “must have” and “values”. Nah, instead I think I will return to THE POINT. The issue here is that NVIDIA KNOWINGLY AND WILLINGLY CHEATED WITH THEIR DRIVERS SO THEIR CARD WOULD LOOK FAST AT LAUNCH. And this absolutely kills every ounce of credibility they have in other benchmarks as well. If they took the time to optimize for 3DMark, what is to say they haven’t done the same thing for Unreal Tournament 2003 or Doom3? Did anyone else think it was fishy that they bundled the Doom3 benchmarks with the GFFX in the first place? How hard is it to cheat in these benchmarks compared to 3DMARK? Easier I would guess since there are no developer tools for you to use to check up on them. The image quality is so subjective and the timedemos go so fast that it is hard to know if what is being rendered in the demos is absolutely the same quality as what is being observed “in game”. We are forced to trust that manufacturers aren’t artificially cheating to make them look good. NVIDIA can't be trusted on THAT one obviously. We are talking about people selling you a $500 card and using its performance to justify the cost. We are talking about a 24% boost in 3DMark03 that resulted from their cheats. We are told to take the other benchmarks (the most important one SUPPLIED by NVIDIA) as gospel instead. If their card is so clearly faster, then why did they have to cheat to compete in 3dmark? I usually think of the drama in the hardware community as being like a soap opera, but in this case I think fairy tale is more appropriate (the emperor wears no clothes!).