Linux lacks testing methodologies
How can potential buyers judge the differences in performance among applications running on various Linux distributions? Linux kernel stability and reliability testing is quite sophisticated, thanks mainly to efforts such as the Linux Test Project, but measuring application performance on Linux is more difficult. The Open Source Development Labs is calling for application vendors to put their products to the test for scalability, security and clustering. In keeping with the open source approach, the lab is also calling on vendors to share their testing and results.
OSDL lab manager and open source test-giver Tim Witham is on a mission to push Linux performance testing to higher-level, real-world applications, to produce reliable, retestable, comparable data that will let users compare the operating systems or open source applications in a transparent fashion.
Witham said everybody seems to have a different idea of what performance metrics means. For developers, it may be how fast they can compile a turnaround; for database administrators, it is a question of speed and support for threads in I/O and consistency in response time. One of OSDL's first significant findings, according to Witham, was the identification and fixing of a few I/O scheduling issues that did not show up on "normal" I/O tests.
