The web is evolving and so should the JavaScript benchmarks that measure its performance. Today, we are releasing Octane, a JavaScript benchmark suite that aims to measure a browser’s performance when running the complex and demanding web applications that users interact with daily.

Most of the existing JavaScript benchmarks run artificial tests that were created on an ad-hoc basis to stress a specific JavaScript feature. Octane breaks with this tradition and extends the former V8 Benchmark Suite with 5 new benchmarks created from full, unaltered [1], well-known web applications and libraries. A high score in the new benchmarks directly translates to better and smoother performance in similar web applications.

Here is an overview of the new tests:

• Box2DWeb runs a JavaScript port of a popular 2D physics engine that is behind many well-known simulations and web games. 
• Mandreel puts a JavaScript port of the 3D Bullet Engine to the test with a twist: The original C++ source code for the engine is translated to JavaScript by Onan Games’ Mandreel compiler, which is also used in countless web-based games. 
• Pdf.js is based on Mozilla’s PDF reader and shows how Javascript applications can replace complex native browser plug-ins. It measures how fast the browser decodes a sample PDF document. 
• GB Emulator is derived from an open source emulator of a famous game console running a 3D demo. 
• CodeLoad measures how quickly a JavaScript engine can bootstrap commonly used JavaScript libraries and start executing code in them. The source for this test is derived from open source libraries (Closure, jQuery). 

Besides an expanded set of benchmarks, Octane also has an interface that makes it easier to read and that adapts automatically to tablet and mobile screens.



You can try out Octane yourself, browse the source code, or read more about each benchmark at the Octane site. Still have some questions? Have a look at the FAQ page.

[1] Beside glue logic and emulation of canvas / DOM interaction where necessary. 

Posted by Stefano Cazzulani, Product Manager

As of current dev and beta channel releases, V8 uses a new algorithm based on counters to decide which functions to optimize. This greatly increases performance for small JavaScript programs. For example, on the SunSpider benchmark, which focuses on extremely short-running tests, V8's speed improved by about 25%.

When executing JavaScript, V8 at first compiles it to machine code with a very fast compiler that doesn't optimize the code it produces. V8 has a second, optimizing compiler that generates much faster machine code, but takes much more time to do so, so it has to be used selectively. That's why V8 must try to predict which functions will benefit most from optimization, and carefully decide when to optimize them.

In the past, V8 stopped once every millisecond to look at currently running functions, and eventually optimized them. For long-running programs, this worked great, but short-running programs often finished before they could benefit much from the optimizing compiler -- a single millisecond can be a long time to wait before optimizing! In addition, V8 often made different optimization decisions each time a JavaScript program ran, sometimes overlooking small but performance-critical functions.

The new version of V8 makes earlier and more repeatable optimization decisions by analyzing the running program in more detail. It uses counters to keep track of how often JavaScript functions are called and loops are executed in a program, approximating the time spent inside each function. That way V8 is able to quickly gather fine-grained information about performance bottlenecks in a JavaScript program, and to make sure that the optimizing compiler's efforts are spent on those functions that deserve it most.

The new algorithm is contained in the current beta channel releases -- go give it a spin now!

Posted by Jakob Kummerow, Software Engineer

Today we are releasing version 7 of the V8 Benchmark Suite. This new version adds Oliver Hunt’s 2D Navier-Stokes fluid dynamic simulation, which stresses intense double array computations. These complex double array computations are today common in games, graphic and scientific applications.

The new test shows the recent improvements V8 has made in handling advanced physics computations: the current Chrome 18 (today in beta) delivers a 5% score improvement compared to the current Chrome 17. Chrome 19 (today in canary), where the full set of improvements is being released, delivers a whopping 25% score improvement compared to Chrome 17.



With these additions, the V8 Benchmark Suite is now a more comprehensive collection of eight tests, including OS kernel simulation, crypto and string operations, memory management stress-tests, and as of today, double array computations.

We plan to keep updating the suite by adding more tests. These updates are a reflection of Chrome’s commitment to keep pushing the boundaries of speed, optimizing the engine for today’s more demanding web apps.

Posted by Stefano Cazzulani, Product Manager

Today we are announcing the release of Chrome’s new incremental garbage collector (GC) which dramatically improves interactive performance of web apps and HTML5 games.

The V8 project has made huge progress improving peak performance of web apps. With the advent of technologies like WebGL we’re seeing the emergence of highly interactive and graphically intensive apps, such as the new version of Google Maps, new games and demos. But with these new uses comes a need for better interactive performance in JavaScript.

Avoiding pauses is vital to achieving good interactive performance. Previously, garbage collection pause times depended on the amount of memory used. Therefore, large interactive apps were impacted by pauses that caused hiccuping. V8’s new GC reduces pause times dramatically while maintaining great peak performance and memory use.

To evaluate the new GC, we took the most memory intensive peak performance test from the V8 Benchmark Suite and used it to make a stress test for interactive performance. In our testing the maximum time to render a frame including pause time is reduced from 272ms to 50ms.

The new GC in Chrome improves interactive performance and opens up new possibilities for the interactive web. If you are developing highly interactive web apps or games, please try it out and share your experiences. It is available now on the dev channel.

Posted by Vyacheslav Egorov and Erik Corry, Software Engineers

Benchmarks are incredibly important for influencing the direction of JavaScript engines. Over the past two years, JavaScript has gotten dramatically faster across all major browsers, particularly in areas that popular benchmarks measure. As it turns out, browser developers are a competitive bunch. However, as JS engines get faster, benchmarks must evolve in order to keep pushing them in the right direction.

We’ve been constantly updating our V8 benchmark suite to force us to get faster in areas that are important to web developers. We’ve fixed bugs and added new tests for regular expressions and memory management. We’re very focused on JavaScript performance, so we scrutinize our benchmark carefully to make sure that it’s as useful a measuring stick as possible.

The two other widely cited JS benchmarks are SunSpider from Apple, and Kraken, a new benchmark from Mozilla.

SunSpider was one of the first suites of tests, first appearing in 2007. It’s done a lot to help improve JS engines, but many of the tests in the suite are less relevant to the web in 2011. Even for the more relevant tests, JavaScript has gotten so fast that many finish in only a few milliseconds. This just isn’t long enough to figure out which engine is faster--a golf cart and a Tesla will finish a 10-foot race in nearly the same time.

To make the benchmark more meaningful, we’ve experimented by making the race longer by running each of the tests in SunSpider 50 times consecutively. While repeating a trivial test many times isn’t a great solution, it does provide an opportunity for some optimization. With this change, the results begin to reflect Chrome’s true performance. It’s more than 30% faster on the SunSpider suite overall and up to 4x faster on some of the individual tests.

Kraken, a more modern benchmark, is in better shape. Unfortunately, the canonical version of the benchmark has not been updated to reflect all the latest changes which address at least one important bug. As a result, the benchmark is less useful and has even (mis)led us to spend time making some irrelevant optimizations in Chrome. To help us focus on the right things, we’re now testing the latest version of Kraken built directly from Mozilla’s source code repository.

We’re posting a modified version of SunSpider and the latest version of Kraken to make it easy to run the benchmarks in your own browser and compare results.

Posted by Mike Belshe, Software Engineer