Scientific Integrity and Ethics

[Lothar5150]

Over the last several months much has been said regarding the integrity and motivation of various personalities who frequent ProCooling. Given the recent scientific fraud made by Professor Hwang Woo-suk at Seoul National University it thought it was an appropriate time to address this issue.

Many observers of the forums will note that any data produced by “industry” are meet with immediate suspicion. However, all “third party” testing is readily accepted on its face. The obvious reason for the skepticism surrounding “industry” testing is the perceived bias and motivation to fudge data in order to demonstrate greater performance, equating to greater sales. However there is little discussion regarding the motivations or bias of so called third parties.

I think Professor Hwang Woo-suk clearly demonstrated that even academic, non-industry scientist can have motivations to fudge data and/or out right lie about results and commit academic fraud. This is the reason that science requires the duplicity of experiments and peer review.

Clearly money can be a motivation to lie about data but hubris can also motivate scientist to lie about data. The involvement of money in scientific endeavors should not presume a lack of integrity and the pursuit of science in an academic setting should not presume purity. Perhaps we should consider that the data which industry produces is as valid as any produced by “third parties” and assume all scientific claims to be made in good faith to be true and only call in to suspicion claims with can not be reproduced.

[gmat]

Little background: I've been tempted with the scientific / research path, i had the opportunity to join national research centers. But the pay was a misery... still some friends went that way (they had 'The Faith' if you get what i mean). I've taken the 'industry' path.
I've been working for some high profile private sector research labs, and i know for having been there that the research is no less serious or scientifically sound than in the 'public / official' sector. Actually the motivations may be different, scientists in the industry work to enhance the products / knowledge of their own company instead of enhancing the knowledge of general public. This does not make them different in their scientific integrity in my opinion, because if they fudge their results it will be soon uncovered by failing products or a financial crash (which is a kind of 'hardcore' peer review)
The difference *may* reside in public perception of their research work. In the industry most research is kept secret, and tech / scientists are not the ones who communicate, this is the task of the marketing dept. So public perception of private sector research is (generally) distorted by that filter. This makes the perception of private sector scientific integrity distorted as well.
This is only a general statement. Some companies make their research papers public, but do they submit them to the peer review process ?
All i can say is as long as there is peer review, the frauds will be rejected by the system, just like Pr Hwang Woo-suk.

[Cathar]

There are many good reasons to be suspicious of manufacturer dependent tests. Many of those reasons have nothing at all to do with the honesty and integrity of the testers involved. Quite often it all comes down to marketing, and as such is typically out of the hands of the original testers, or they were instructed to, or perhaps even subconsciously, set the boundaries of their testing to arrive at a set of results which were favorable for a marketing bent.

The results may be perfectly valid, and quite often they usually are. Very easy for independent review to catch people out who are blatantly lying, as was the case of Professor Hwang Woo-suk. His research failed to stand up in the light of independent scrutiny, and so was caught out in a lie.

No, the clever marketer knows the strengths of their products, the scenarios or tests to choose which most favorably present their chosen produt in the best light, and the sets of test results to omit, both for their own products and/or their competitors, whether in part or in whole of completely ignoring a competing product, for purposes of presenting the strongest possible marketing case. Let's face it, industry testers who receive a pay packet derived from one of the products that they are testing are not going to publically present results unless they can find a set of results that is marketable and thereby facilitates receiving more pay. Anyone who says otherwise is either a liar, or they are very soon out of a job or out of business. Take your pick.

The art of "tactical deception" (aka marketing) is often achieved through careful omission of the truth. At best one can expect a playing down the product's weaknesses by attempting to dismiss the importance of the weak results, typically in fine print or in a side link, while presenting big pretty graphs of the strong points emblazoned with strong words about how such results are truly important and relevant.

More often than not, it is extremely important to analyse what is NOT being presented to you, far moreso than what IS being presented to you, when dealing with results derived from a source of conflicted interest. The results may very well be true, but the picture being presented can often be much like that of the publically viewable physical representation of "Escher's Triangle" on display in France. It looks perfect when viewed from a single very limited viewpoint, but when viewed from other angles that are not being controlled by the tester/marketer, all may well not be what it seems.

[BillA]

gmat I agree with,
Cathar digressed into the same old marketing stuff, tarring the 'tester' with the same brush used for the salesman
jeez

again, perhaps a split:
- Scientific Integrity and Ethics for marketing and sales
- Scientific Integrity and Ethics for engineers and scientists

some questions for Cathar:
Pray tell what data set was ever complete ?
What is the obligation for one mfgr to test another's product ?
Those that test their own should not be permitted to describe their product ?
Your aggressive attacks on mfgr's data will only serve to inhibit the release of such data, your quite apparent agenda in immediate condemnation w/o reflection or analysis WILL reduce the data available to this community.
Do you understand what I am saying Stew?..
If I have to battle you to establish the validity of test results based on recurring allegations of bias by you, Bruce, and all your sycophants - I will let you guys generate and post some numbers for a change.

[jaydee]

Generate some numbers? Why? So we can be told how wrong they are or why the method used is not satisfactory? After all the debate every set of results I ever seen posted is now all bull shit.

Also about the topic I have always hammered on 3rd party results. Hell remember all the review bashing we done here? Get some perspective.

[BillA]

jd
I am not the arbiter of technical correctness,
just try to be realistic about the difference between engn and DIY (review stuff)

you seem to have forgotten the roles here,
did I not propose a CPU based DIY platform ? (shot down with bs scaremongering)
but no, simple is not sufficient and a die sim is required for procooling (though pH uses a CPU)

all test results are not bs to me, I can sort them if the details are known
3ed party . . . . . review bashing, I totally agree; but if the 3ed party testing is crap how is such info valuable is assessing the (naturally lying) mfgr's assertions ?
-> if a reviewer presumes to correct a mfgr's description then their own testing capability should at least have a similar capability to that of the mfgr being corrected

remember the thread topic: Scientific Integrity and Ethics
3ed party better ?
no, far worse if one gets into the freebie thing plus ignorance

[Lothar5150]

Bill why even bother arguing with Cathar...clearly he is always right...which is precisely what I am talking about. HUBRIS is a motivation

[Lothar5150]

Originally Posted by gmat

Little background: I've been tempted with the scientific / research path, i had the opportunity to join national research centers. But the pay was a misery...

gmat, I'm in complete agreement with you. I'd even go further and say that deceptive practices with the use of data (.i.e. marketing) are often uncovered by competitors which are also a type of peer review.

[BillA]

I do not contest what Cathar describes, but pounding on marketing antics while presumably discussing 'Scientific Integrity and Ethics' is wide of the mark

BTW, not arguing with Cathar anymore; will state once only (trying to reduce the tit-for-tat crap)

[Cathar]

What? The?

Take one generic point about marketing and marketing abuse of testing, and some here personalise it into a self-obsessive rant.

The post was directed at no one. I was more thinking of car manufacturers and testers who live off the continuing good-will and receipt of cars, and computer CPU manufacturers with their marketing departments that provide best-scenario test results when I made that post, then some here come along and take it so personally?!

Tit for tat? The content of some of the last 5 posts is somewhat eye opening to say the least. Perhaps I have unintentionally struck a nerve on the real reason why this thread was created?

Wow! Didn't expect that sort of response. Guess I shouldn't be suprised.

[BillA]

car reviewers could be on a par with computer bits reviewers ?
are you equating car reviewers with say . . . . testing by Swiftech (only they disclose such) or by myself/CoolingWorks ? (Re: Scientific Integrity and Ethics)
your command of English is excellent, I assume you mean what you say

I agree, I think this thread is a troll (both of us are quite predictable, nothing to fret about)
while I work on getting over myself you might consider better analogies

[Cathar]

The post was not meant as an attack on any individual, company, or product. If anyone chooses to view it as applicable to a specific set of individual parties, that is wholly their choice. I will not be drawn into personalising such a clearly generic statement. Read it in the correct frame of mind and it obviously applies to many industries, not any one in particular. It was more of a generic "Caveat Emptor" post.

The opening post spoke in general terms. I responded in general terms. I don't see why it had to draw so much offense, unless of course the intent of the opening post was not meant to be general at all, and was really just a softening lead up to something else.

[guandi]

Going back to the original post...

I used to write reviews for two "up and comming" review sites that are now sadly defunct after a death and a birth gave the owners something else to think about, but while they were up i had 2 reviews in particular that gained alot of attention and i received plenty of angry emails about the validity of my results.

The problem was that my results did not tally up to the supposed results given by the manufacturer, one product was an automotive heatercore that had been cut down to fit a 120mm fan and had a new end plate and barb soldered on and was then *dipped* in acrylic paint. The people that made it had supplied it saying it would disipate heat better than the then new on the market Black Ice and let everyone know this on the packaging and website, and maybe it did before they cut it in half, and covered it in acrylic paint. my results showed this and i was then sent an email basically saying How dare i give a rad i got for free a bad review, i told them i was being as unbias as i could be and that it wasnt my fault that a standard car heatcore and the black ice had out done them in the same loop on the same rig with the same CPU running PI but alas after a flame war lasting several weeks the site and the review was pulled.

Second was supposedly a contender to the DD Maze 3, i had the same problem as above, results that didnt tally with the manufacturers advertised results and yet again i was the fall guy, this time however i was sent a letter from some lawyer that didnt even understand what a waterblock was and she threatened me with legal action for misrepresentation of a product unless i changed the results, i told them they could take me to court, lock me up and throw away the key as i would never falsify my findings but two days later the site was gone, and although my review was lost years ago i later found out i was right about the blocks poor performance from another reviewer.

It makes me wonder what would of happened if the site hadnt of been taken down, would I of been taken to court? would i of been able to prove my findings to a non technically savy jury? At the time i was still in school and hardly a scientist being last in my class, and as i'm slightly autistic, dispraxic and dysgraphic (i cant write or hold a conversation without getting lost but i know my computers, engines, guns and turbines) i would never be able to hold my own in court and it makes me wary of doing reviews of performance critical products in case it happens again.

But still, To anyone else the danger is still there, will accurate testing procedures with results that vary from those that are expected immediatly put you in disrepute with the testing community? Is changing results in order to fall in line with everyone else acceptable? for the sake of technological advancement I hope not.

[BillA]

DIY on the cheap
I do suspect that had you spent some money to cal your temp system you would have told them to pound salt (as you did) w/o a worry
- so the site owners had no commitment, not your fault

some measurements will vary (the temp differential depending on measurement method), others should be very close (flow, pressure, air/coolant temps) depending on the calibration

a tester using uncalibrated equipment is in a poor position to assert much

[jaydee]

senseless rample....

[guandi]

It wasnt cheap but it was accurate. i managed to get some very accurate (.001'c) probes from a lab supply company in france, i had a cut down temp probe glued against the side of the core with thermal adhesive as well as two air flow and temp sensors stuck in front and behind the rad, i had some water temp sensors salvaged out of a laser cutter from my dads work that handily went straight into a seriel port, they were siliconed into the line between the block, rad and pump and a flow rate sensor fed into the T-line so that i could work out any pressure change and thus take pretty much everything into account. Most of the things i tested were just tested against each other, i would use the same loop just switching the product being tested and noting the difference across the sensors, i ended up getting quite anal about it even waiting several days to get the right room temperature for mesuring the air differential in my dank little cellar.

Ah, the days when i did things with both halves of my ass

But of course, who takes the teenage hobbyist seriously when compared to a large company even if its only dipping its oar into a pond it knows nothing about.

This has lead me to other thoughts - what if this accuracy was what caused the difference in results, maybe it was the manufacturer who was incorrect in the first place and my findings merely made them look worse than even they thought they were.

Still, the site owners lack of commitment is probably what saved my ass from a courtroom thrashing all the same.

[gmat]

are we discussing scientific integrity, or internet review sites ? Aren't we mixing up issues ? Apart from very few exceptions (pHaestus comes to mind) no one in the 'computer hobbyist world' has been using scientific method or tools. BillA has done some work for watercooling big guns, but it's not really 'product reviews'. We're mixing up consumer oriented write-ups and scientific work.
If we look at the small watercooling world, there has been very few scientific research. Most of it has been done already by automotive mfgers (who are the first interested) and by high end computing companies (Cray..). In this industry the only scientists i suspect are working hard are working for companies like Papst, Laing, and other industrial oriented companies. Not for computer hobbyist market oriented ones (were are not really the main customers for Papst or Laing, for example). I know scientists are working on watercooling for high power lasers or high power electronics in research centers. Amazingly (?) they end up with solutions similar to us (microchannels, high pressure pumps, etc). The difference is they don't have to market a product.
So don't mix up the two worlds, we are in a small hobbyist world, apart from the big industrial world. In *our* small world the *few* scientists that can be heard of have very few peers to criticize their work (and it seems, they're all on ProCooling !).

[BillA]

scientific integrity is the topic
just had some graffiti "about marketing and marketing abuse of testing", an expression of anti-company concern I believe

I 'classify' my testing as the generation of engn data, which as the resolution increases becomes rather more scientific (abusing some terms)
integrity for me is little more than objective recording; competent and honest work is assumed, and verified based on its criticality

[bigben2k]

Originally Posted by BillA

...again, perhaps a split:
- Scientific Integrity and Ethics for marketing and sales
- Scientific Integrity and Ethics for engineers and scientists
...

Agreed on the split, but where does the DIY'er fall?

"marketing and sales" if reviewing for a site
or
"engineers and scientists" if an independant individual doing testing for the exercise?

DIY testers are rare. Imagine if ProCooling had two.

[BillA]

while grouping permits sorting, the characteristic is not group specific - the hypothesis is not the conclusion
"Scientific Integrity and Ethics" is a mindset, not the product of an activity
there used to be more testers until the upward technical creep began some years ago
happens in every field of endeavor, called progress generally

[Lothar5150]

Actually jaydee my point is that there are other interest which can lead to the manipulation of data. Commercial interest doesn’t necessarily lead to manipulation and so called third parties are not free of potential bias.

FYI my company’s commercial interest lay largely in OEM work.

[Incoherent]

I am not sure if Scientific Integrity and Ethics have much to do with anything in this case.

Simply a matter of better the devil you know than the devil you don't. Human nature.

Sadly.

[jaydee]

Originally Posted by Lothar5150

Actually jaydee my point is that there are other interest which can lead to the manipulation of data. Commercial interest doesn’t necessarily lead to manipulation and so called third parties are not free of potential bias.

FYI my company’s commercial interest lay largely in OEM work.

Sorry Lothar, I meant my last post to be extremely sarcastic. I don't think is came across as so. Will edit.