Qualcomm Releases Vellamo Benchmark Aimed at Mobile Browser Performance [Hands-On]

We all love benchmarks right? We get a shiny new smartphone or tablet and one of the first things we do is run a few benchmarks for comparison, I do at least. I’m always trying new things and tweaking CPU speeds trying for a high score. Qualcomm has developed a new benchmark system aiming for something entirely different. Its goal is testing mobile web browser performance across the Android platform.

We all know our 4G isn’t fast enough and we will always want better speeds. Mobile browsers have came a long ways in a short amount of time but how well do our browsers actually perform. And are certain manufacturers and carriers better than others? Qualcomm has just released Vellamo, a suite of 11 separate benchmarks dedicated to evaluating and testing all the various hardware and software subsystems a mobile browser depends on. Things like HTML, javascript, pixel blending, screen scrolling, sun-spider, V8 and more all into 1 massive test.

Vellamo benchmarking tool enables mobile users to evaluate which Android devices deliver the best mobile Web experiences. Vellamo provides a holistic view into browser performance by measuring each component systematically, providing results for CPU and memory, scrolling, JavaScript, HTML5, canvas rendering speed and network access.

Here is a quick video overview to show you more about this browser benchmark and how it actually works

https://www.youtube.com/watch?v=3eNWjE1qoqM

By creating this benchmark it allows users and manufacturers both the availability to test not only the performance, but also the stability of their smartphone or tablet. You just got a new kernel for your Nexus S and want to test it for stability and performance while overclocking this can help even though its mainly testing the browser it fully loads the system and should find any faults in stability.

Sy Choudhury, Qualcomm’s Director of Product Management, told us that Vellamo was originally created for internal testing to help engineers test every aspect in browsing, then optimize code and performance for all Qualcomm chipsets. In doing so the test is not only efficient and accurate, but it also is not refined or optimized for any one chipset and will give true results across all platforms. This makes it useful and available for end users, as well as phone and tablet OEMs for optimization and testing.

Vellamo is currently available for Android 2.2 and up as well as Honeycomb tablets. I went ahead and ran a few tests of my own on not only my Sensation 4G as shown above, but also on my Motorola Xoom (Tegra 2) to see how it compared with the competition. My Motorola Xoom scored 930 compared to the Galaxy Tab 10.1 in the graphs scoring 957. Don’t worry I quickly passed that with a little overclocking.

After overclocking my Motorola Xoom to 1.6 Ghz I was able to score a wonderful high of 1181 and blow the Galalaxy Tab 10.1 out of the water. So far it looks to be treating hardware pretty fair and not being biased to Qualcomm chipsets but then again we don’t have a Honeycomb Tablet with Qualcomm to test so that will have to wait for another day.

OK, so do you want to know what Vellamo actually tests? Here are the descriptions that Qualcomm provides inside the benchmark:

Rendering, See The Sun Canvas: Tests the performance of HTML 5 Canvas-based graphics.
Rendering, Deep Sea Canvas: Tests the performance of HTML 5 Canvas-based graphics.
Rendering, Aquarium Canvas: With many simultaneous objects, this stresses the performance of both the HTML 5 Canvas-based objects and the JavaScript engine.
Rendering, Pixel Blender: Measures the web graphics-related bandwidth available for blitting and blending operations.
JavaScript, Surf Wax Binder: Tests the array performance of the browser’s bindings between the Browser and JavaScript engine
JavaScript, Sun Spider (Online): Measures JavaScript performance using Rev 0.9.1 of the industry standard Sun Spider suite.
JavaScript, V8 Benchmark (online): Measures JavaScript performance using rev 6 of the industry standard V8 benchmarking suite.
User Experience, Ocean Flinger: This is a test that tests scrolling smoothness and performance using a demo news site.
User Experience, Image flinger: Stress tests the browser scroll performance when using a page made up of dense images. May take a minute to load.
User Experience, Text Flinger: Stress tests the browser scroll performance using a text-heavy web page. May take a minute to load.
Networking, Networking Loader: Repeatedly loads and reloads a sample web page to determine HTTP networking stack performance. This happens quickly, and hence you may only see the base color background.

FACT: Vellamo is named after the Finnish goddess of the sea. The benchmark tests web-surfing (get it? surfing in the sea? Look at the Vellamo logo.)

The Vellamo Mobile Web Benchmark is available right now in the Android Market so feel free to give it a try on a few different devices. We will be messing with it over the next few days and will continue to update if we find anything interesting.

So what do you guys all think about benchmarks, specifically ones designed in-house like Qualcomm did here. Are they a fair way to test multiple devices across different OEM’s? Or will this chase better numbers on Qualcomm and Snapdragon chipsets? Sy said no, and that it will be an even result across all platforms. You can even hit the “last numbers” button in menu to get a full breakdown for a full explanation of the results. Fee free to share your thoughts and results in the comment section below.