11 replies to this topic

[Staf_in_Wah]

Apparently there has been a lot of cheating going on in Android Benchmarks. Sadly, your News Mistress is a bit of a newbie and this has me going...

 

Please Login or Register to see this Hidden Content

 

Please Login or Register to see this Hidden Content

  59.24KB   97 downloads

So, check out this expose on AnandTech. It sure makes you wonder if the specs on that new device you are eyeing are correct. At least I think that's what this means.

Please Login or Register to see this Hidden Content

 

 

 

[portscan]

Seems to me all that happens is the phone utilizes the complete GPU and CPU power of the device when it detects benchmark. In my eyes that's what I want it to do. I want to see how bad rear end my phone really is. A true test would be at maximum power. Unless I completely do not understand the article, I disagree.

[nakedtime]

I agree with portscan.

 

This isn't cheating. It is allowing the phone to be used to its potential.

 

Look up "thermal throttling" if you'd like an explanation. 

Eg. The nexus 4 thermally throttles the CPU at 60C which led to some abysmal benchmarking. Most phones do not throttle until 80C.

 

Is a phone that normally throttles at 60C really worse than a phone that throttles at 80C? IMO throttling at 60C is better, all sammy has done is up that threshold to 80C for benchmarks. (I dont know the exact numbers for Sammy phones, I just know the 60C for Nexus 4 and 80C for the Bionic)

[Int_Rnd_Pooka]

If, during normal user operation, the phone throttles at 60C, but at 80C for benchmarking, that's cheating. It means that there's no way in hell you're going to achieve the bench results because you have to bypass the built in throttle. Now, if you can go into the N4 settings and disable the throttle limit, that's fine. BUT, the benchmarks should always be run using the same settings as the system itself, not in some optimal sandbox type environment that cannot be reached without serious tweaks to the system that can't be reproduced through the phone's normal settings menus.

 

Yeah, test at maximum power. But maximum power should be determined by the actual system settings as is, not by raising the bar just for benchmarks.

[portscan]

I see the point, but I can run 15 different tests using Quadrant Standard on my S4 and get different results each time. Hovers between the 12,400 to 12,600 range. I didn't see them mention Quadrant Standard in the article. None of it really means much to me. All I know is the S4 is a bad rear end phone and does bad rear end stuff, bad rear end fast. In theory if Samsung doesn't see Quadrant as one of the big Benchmark boys, then the cheat isn't triggered when Quadrant is launched. That should tell me that the readings from Quadrant for my device are accurate.

[BytecodeMe]

If you'll never see that kind of performance in real world use and only when you're getting some inane number from a benchmark that can never be truly reliable because of its own unreliable algorithm, thermal throtting/unthrottling, CPU and GPU frequency + voltage boosting, background processes, the status of the systems caches, Androids own memory killer and other strategies to raise benchmarking than what really us the point???

To make the phone get a good number so the kiddies can sleep soundly knowing they supposedly have the leetest fastest phone out there. Will you ever see that kind of performance in real world use? No. I don't see how you could twist around how it isn't cheating?



Sent from my Nexus 4 using Tapatalk 4

[Nagaro]

This really goes to show just how big of an impact benchmarking has.  Honestly, benchmarking is more or less a way to brag about your stuff.  It's like, children and them doing stuff.

 

Sure, benchmarking is fun, and I do it all the time, but as already mentioned it's far from what you get real world performance.  Do you really need a quadrant score of  over 10000 to open the play store smoothly? 

[nakedtime]

you DO see that performance in the real world.

 

Also, IDK about the thermal throttling table for the note 3, but you can edit the table on the N4 to whatever you want.

 

Think of Goku performing the Kaioken technique, Thermal Throttling allows you to have short bursts at very high voltage.

 

**edit

Its fine if you want to call it cheating. I'm just saying its not fine if you are upset/care about it.

[Int_Rnd_Pooka]

BytecodeMe - Nail on the head.

 

Nagaro - The only thing I use benchmarks for is to compare my phone to itself. If I make changes to something in the kernel and alter my clock speeds and whatnot, it's nice to be able to take a before and after snapshot. I don't bother comparing to other phones because .. Bionic. I could compare with the GNex, I suppose. Everything else is out of my league

 

nakedtime - Goku references aside, we're not upset about it (not anyone here that I see, anyway). Mostly it's just a case of wanting people to admit that they're manipulating their numbers to look better than they are. But you don't see that performance real-world if they're enabling what are effectively secret settings to acheive the test results. Also, can't we go with Kaioken times ten for the analogy? Kaioken 1x isn't really that impressive

[nakedtime]

BytecodeMe - Nail on the head.

 

Nagaro - The only thing I use benchmarks for is to compare my phone to itself. If I make changes to something in the kernel and alter my clock speeds and whatnot, it's nice to be able to take a before and after snapshot. I don't bother comparing to other phones because .. Bionic. I could compare with the GNex, I suppose. Everything else is out of my league

 

nakedtime - Goku references aside, we're not upset about it (not anyone here that I see, anyway). Mostly it's just a case of wanting people to admit that they're manipulating their numbers to look better than they are. But you don't see that performance real-world if they're enabling what are effectively secret settings to acheive the test results. Also, can't we go with Kaioken times ten for the analogy? Kaioken 1x isn't really that impressive

 

Kaioken x10 would up your power consumption to something on the order of kilowatts. I dont think your motherboard is capable of supplying that kind of power.

[NullPointerException]

I agree.  I dont think its cheating. I think its using the benchmark to its full capacity as well.

[Int_Rnd_Pooka]

Is it providing results that the average user can achieve without obscure tweaks?