Lampooning Benchmarks

Wed, 26 Sep 2007 0:43 UTC

When I first came across Jonathan Street’s “7 tips for lightning fast PHP sites” blog post via PHPDeveloper.org, my first reaction was something like: “Egads! These benchmarks are stupid and misleading! These functions are simply aliases of each other. There should be no discernible difference, and any buffoon should realize this fallacy!” This was before I clicked through from PHPDeveloper.org to read his post.

Then, I clicked through to the post, and I was still dumbfounded that Jonathan found marginal microsecond differences between how the functions performed and was advocating the use of one function over the other based on this “data.” I questioned the testing, and I was prepared to dig through the PHP source to write up a post refuting his claims.

And then I did something smart…

I followed a link to his follow-up post entitled “Better Benchmarks” in which he explains that his previous post was a lampoon that “was supposed to be a spoof celebrating the worst aspects of these types of posts.” He went on to explain:

I had thought that with comparing aliases of functions seven times over people would realise what I was doing but apparently my post was just too close to the sad reality and lacking in sufficient humour for people to catch on.

When reading it in this light, it actually is really humorous. He goes on to explain another round of testing he did that was intended to show a more accurate measurement of the results. His findings are not unexpected, and he admits that:

Overall I conclude that there is no statistical difference between the aliases of a function in the tests that have been run. Although this was the expected result I hope that the analysis presented is sufficiently rigorous to discount the possibility of personal bias in the benchmark.

However, for the naysayers out there, I’ll post some lines from ext/standard/basic_functions.c in the PHP source. This should prove that the only difference in these functions is the name itself. All other code is shared. (Note that is_int() and is_integer() are actually both aliases of is_long(), even though the manual prefers is_int()).

PHP_FALIAS(chop,            rtrim,      arginfo_rtrim)
PHP_FALIAS(ini_alter,     ini_set,      arginfo_ini_set)
PHP_FALIAS(doubleval,    floatval,      arginfo_floatval)
PHP_FALIAS(is_int,        is_long,      arginfo_is_long)
PHP_FALIAS(is_integer,    is_long,      arginfo_is_long)
PHP_FALIAS(join,          implode,      arginfo_implode)
PHP_FALIAS(fputs,          fwrite,      arginfo_fwrite)
PHP_FALIAS(sizeof,          count,      arginfo_count)

So, what can we learn from this? For starters, always read follow-up posts, and never take part in function benchmarking wars.