Multicore is unstoppable: the economics are driving it. Therefore we need software to exploit the huge opportunities that these new architectures offer

You have been there, a conference that was full of brilliant new technology that you will be able to use … possibly sometime in the future.

Multicore World is different.

We present you today what you can start using today – because multicore technology is with us already – and it’s not as hard to employ as you might think. Multicore World 2012 goes to great lengths to find the best in the field of hardware and software development and bring them all to Wellington to give you a cut through the current state of the multicore world.

Confirmed Keynote Speakers are James Reinders, Intel Chief Software Evangelist, Tim Mattson (Khronos OpenCL Group, Senior Research Scientist, Intel) and Martin McKendry (ex-SVP at Avaya, Siebel, Openwave)

Check the impressive line-up of international guest speakers here with six of them coming from the US exclusively for Multicore World 2012.  Join them: CFP is still open till 20 February 2012

Multicore World 2012 will be at Wellington Town Hall, 27-28 March 2012. More information in the conference website Multicore World

First Step
———-

We analyze a typical request of your site. To display your site the browser first fetches the page that is generated by your PHP application – served off Apache. This page includes many static resources: the browser fetches all of those to display the site. This step goes back to your Apache server that already served your PHP content and serves all static files the same way. This is a perfectly valid way of dealing with static files if your site has a low and continuous traffic pattern. If you deal with peak loads frequently or your average load is more than 50-70 concurrent users per server then Apache becomes a performance bottleneck.

To reduce the work Apache needs to do, we put nginx in for serving static files. Nginx only off loads the heavy lifting of the PHP work to Apache. This way we cut down the memory and CPU usage already by about 30-40% depending of your website.

Now we have to start modifying code as HipHop uses a stripped down version of PHP 5.2. The most important difference is the missing ‘eval’ function.

Second Step
———–

We look into your code base, at the structure of your site and port it to run on the HipHop engine. It is a pretty standard procedure and once done your development cycle will not change a lot. You gain a dramatic change in performance. All changes are compatible with standard PHP.

HipHop takes your PHP code and translates it into C++ which we compile down to a Linux binary. Facebook, probably the largest PHP site in the internet, reduced their computing needs to power the site by nearly 50% by moving to HipHop.

Third Step
———-

Here occurs the biggest customization work but with the biggest rewards. If you use a multi-processor architecture (which you do if you run on bare metal or high spec virtualised servers) then you can parallelise tasks in your programs to run on all the CPUs you have available in parallel.

We will present a custom HipHop version with TBB (Threading Building Blocks) functionality that enables us to run for and while loops in parallel and increase the performance that way.

Depending on the amount of for/while loops in your code and the nature of them and the work they do we can optimize them to run on many CPUs instead of sequentially on one. Your for loops will suddenly be extremely fast.

Our PHP language extensions give you certain data structures and a parallel_for function that is thread save and auto scales to the amount of CPUs in your system. This last step will give you a total of up to 50% resource drop and up to 70% speed-up compared to your Apache setup.

Contact Open Parallel for more information

This is interesting for a number of reasons: say you have an array of items, which you know are going to be native integers, you can store them with a whole lot less memory than the current situation:

Under Reini’s proposal, this could be reduced to:

Yes, that’s about two data structures for the entire (fixed size) array. This is the most dramatic of the memory reductions and would bring Perl quite close to C in terms of memory use for data without resorting to relatively slow pack/unpack systems like Judy. It would also be very fast.
There are also provisions for other typed scalar values (such as float, strings, etc) and also a solution for hashes called “perfect hashes”.

Already this is all very good stuff, but it was the treatment of :const that caught my eye;

my int $x : const;

This guarantees that once Perl has allocated and initialized the value, it may not change. This would allow one of the worst problems with the existing threads::tbb library to be avoided.

Currently, the data structures that you share through threads::tbb::concurrent and other containers must not be updated after they are saved to the container. However, it is very difficult to ensure this and there are some operations which change memory, which you wouldn’t think are writes – even innocuous things such as printing an integer, which may upgrade the data structure from an IV (integer only) to a PVIV (integer + string “dualvar”) to cache the conversion. The current API avoids this problem mostly by copying memory structures very quickly on the first access, and then subsequent accesses will work with the copied data. But it is not foolproof, and it’s possible that any access to data structures which have been shared may cause another thread to segfault the process due to an unsafe data structure access.

This proposal would provide the perfect API for ensuring that changes are thread-safe: requiring that only :const values may be put inside it. I would posit that the result would not only be easier to work with and safer by design, but use less memory and run faster, as the values do not need to be copied to share between threads in the first place.

But it gets even better.

With “perfect” hashes, and :const and typed arrays and scalars, the Moose-like idiom of __PACKAGE__->make_immutable, which forbids a range of run-time dynamic behavior, could go even further: it could mark the actual package stash (symbol table) as :perfect or :const. If the Perl_clone() API takes this into account, then thread creation in Perl could get a whole lot faster. Currently, creating a new Perl thread basically duplicates the entire heap of your program – this comes from its design as a replacement for the Unix copy-on-write fork(), which was never implemented for Windows. If modules can mark which parts of them have finished changing, and may no longer change, then those parts don’t need to be duplicated. Even if they are not :const – just being :perfect could allow the perfect hash keys to be shared, and so on.

One of the senior Perl figures I talked to about this was excited that this would be “real” :const, not the “humpty-dumpty const” behavior required by C++. In C++, the const keyword can be overridden by mutable. This was intended for things like reference counts, but in any case those are probably best handled by the relevant API calls specific to reference counts (which can use atomic types).

The summary is that Moose programs which use the make_immutable API would be able to start new threads, using far less CPU time and memory to do so; as well as the other benefits of this code, such as using less memory for object instances.

Of course, this is all very much a roadmap, and requires a lot of community acceptance of the idea, and working proposals in terms of patch sets and so on. However, Reini is hopeful for targeting Perl 5.16 for the first few changes, and 5.18 for more of them. At the current rate of Perl releases, that could mean this hitting a production Perl release as early as 2012.

Writing a LaTeX article-style white paper is an unusual way to communicate a new idea in the Perl community, which are normally focused on CPAN releases, conference presentations, blog posts etc.  However I chose this format for a wider audience – as the research itself applies to many of the dynamic language environments out there, and I try to characterize which interpreters it is applicable to – along with warnings about the difficulties in proceeding.

Actually, the first attempt at this effort was for PHP/Zend, to go alongside other OpenParallel successes with PHP/HipHop.  However it became obvious early on that the depth of previous research and support framework I had access to made targeting PHP first not a feasible option.  At least not for someone without the oral transmission of how the Zend internals work.

Here’s the abstract, with a little summary commentary:

Perl’s history has seen two threading models; one shared-state model now dis-
continued, and one heap-duplicating model with a very inefficient shared state ap-
proach. There is also an Erlang-style threading library on CPAN, threads::lite,
more efficient and scalable but specific in functionality.

This summarizes the “Threading History” section of the paper.  Threading has a bad rap to date, with many language communities considering it a flawed approach, inherently instable or unnecessary if you have fork().  I’m with the TBB authors on this – that it’s more of an API issue.  This section tries to give the section of the relevant history for Perl, as it explains why in the Perl community the situation is so.  As the person who compiled the Perl git history, my focus for this section was certainly on what is available in git and on CPAN.

A task-oriented parallelisation approach permits parallel operations on data sets
as well as pipeline-based programming. threads::tbb, the core invention of the
paper, uses Intel’s Threading Building Blocks (TBB) along with a system of lazy
cloning for state, and is shown to result in speed-ups for embarassingly parallel tasks
to 8 processor cores or more.

There is a section in the paper where I have effectively summarized the contents of the TBB approach for the lazy by virtue Perl programmer.  It references James Reinders’ book, still the best source of information on TBB.  It also summarizes the results section, where I show that:

• the approach works for “synthetic” tasks – ie, inner loops, with the best time increase being a 7-times speed up on an 8-core system.
• described effects of memory allocation mutex contention are observed, and can be worked around using the TBB scalable memory allocator proxy.
• The API is useful enough to be useful to retrofit a legacy program.

The results are applicable in principle to other languages which are built around
boxed variables and state machine interpreters, such as PHP/Zend or standard
Python.

As I mentioned before the jump, most of the difficulties are around the nature of having a state machine and are applicable to a selection of other Virtual Machines.  This assertion would need to be tested by those other communities of course.

The cloning logic duplicates core Perl 5 code in intent, the API for which could be
cleaned up to avoid some minor API intrusions. Green fields interpreter approaches
would benefit from a const concept to avoid duplication in the first place, and for
safer operation.

The “lazy clone” approach I mentioned has some very large caveats, and I attempt to explain why they are necessary in terms of achieving practical speed-ups, mitigating factors to the decision, and design approaches for avoiding the problems in the first place.

Transformations for task-orientation are similar to those required for event-
oriented programming, with potential to parallelize event frameworks, or for APIs
which span the two styles.

This is a possibility I want to expand upon in the future – I believe that the parallel_while API in TBB would allow you to make event-style frameworks which allow multiple events to be processed in parallel.  This would be interesting and perhaps a new style of programming.

Do you have a slow PHP site?

Here is how we help you deal with it: optimizing PHP with HipHop!

We are specialists in HipHop and help companies to transition to HipHop

Read about our work with Intel’s TBB in WordPress (White Paper) and WordPress on HipHop (White Paper).

Open Parallel is a consulting company that can help you grow your PHP based application and transition to a HipHop based application stack. We also offer customized HipHop engines that exactly serve your needs, as HipHop can be more than just the standard “off-the-shelf” version that is available online. Our extensive experience in both, the HipHop engine and porting PHP projects to HipHop will help you scale without the need of extensive re-training of your developers.

On top of being a Consultancy Company doing HipHop work, we also do Research into the HipHop engine and enhance it with our deep and direct knowledge of Intel’s TBB (Threading Building Blocks).

Contact us. Or hear directly about our work at Linux Conference Australia 2011. We will present “Integration of Intel’s TBB into Facebook’s HipHop” on Friday 28 January 2011 in LCA2011 in Brisbane, Australia.

Introduction
When we at Open Parallel started to look at a way to implement TBB into PHP and HipHop, we initially planned to tackle the HipHop optimizer and try to come up with some sort of automatic conversion that uses TBB where possible. This turned out to be a not workable task for the moment and we went down the track of mapping TBB commands directly to new PHP functions.

The idea was to implement TBB commands into HipHop and write PHP equivalents for the case where either HipHop is not compiled with our patches or where the project runs on a more traditional stack.

First command: parallel_for
We started out with parallel_for as it looked like one of the commands that suit a web project best. There are many loops in WordPress and the ability to effect free parallelization on them looks like a very promising option. As not all loops can be run in parallel, we opted for writing a patch set for WordPress that used the new syntax and commands.

On the HipHop side, TBB was already used for memory management and adding the parallel_for command was pretty straightforward. We extended the known language constructs of HipHop by the command parallel_for and created a patch set for the HipHop code base. With these patches in place, we were able to compile PHP constructs containing the keyword parallel_for on the HipHop engine.

After initial tests with the new command and demo scripts on how to use it, we took the WordPress code apart and tried to find a foreach loop that was used frequently. We found that the plugin code had one of the more frequently used foreach loops and rewrote it using our new parallel_for – it worked after some debugging.

Verdict so far: extending HipHop is not entirely straightforward but can be done.

Adding TBB to the PHP language is possible, but the developer using it needs to judge if a loop can run in parallel or not. The language construct we currently use might not be straightforward enough for the junior PHP developer, but then again, they will most probably not be our target audience anyway, as they have to learn how to judge if something can run in parallel first.

Bottom line: An intermediate PHP developer can use the TBB extension after some initial training on what to look for.

An example invocation of the parallel_for function looks like this:
function test_pfor_callback($indices) {
foreach ($indices as $idx) {
echo “{$idx}\n”;
flush();
}
}
parallel_for(0, 1000, ‘test_pfor_callback’);

WordPress on HipHop
The next step was to run the hacked WordPress version on HipHop and compare it to the original version. This bit was the impressive bit. Just to recap, we added one command out of the TBB functionality to PHP and then modified one foreach to run with our parallel_for implementation.

In the initial tests, we saw a significant drop in memory usage and a sizable reduction of the time our test suite needed to run. We subsequently moved more code into parallel_for constructs and refined our testing, and even though our gains in terms of memory dropped, we managed to gain more on the execution time.

Adding TBB
The lessons learned from this experiment are interesting. HipHop is a better way to run PHP – we kind of expected this – but adding TBB to the game for more than the memory management looks like the way forward. TBBs parallel_for gave us a considerable speed-up and a significant decrease in memory consumption

This experiment also showed that there is a vast field of optimization in script languages for optimizing loops. We will look into this in more depth and try to come up with a more general TBB implementation in PHP and HipHop in the future. The reduced memory footprint and the decrease of execution time due to running tasks in parallel are addressing very real life problems, and fixing this in a more general way looks like a very much needed language extension.

Details of the Tests
Going into the details of the tests, we explain our test set-up and show how these tests can be replicated on your hardware to see what speedups you can gain – the first tests ran on real networks on different physical machines whereas the later ones ran with the one described below.

We tested on virtual machines running VirtualBox with HipHop and WordPress (including MySQL) running in the VM and Tsung running on the host machine. The host was running MacOS 10.6.4 and the nodes were running Ubuntu 10.04. For each test the VM was rebooted, HipHop started with our Open Parallel startup script and then Tsung started from the host.

Collectd was used to gather the performance data from the node systems and no other services as the required ones were running at the time of the tests, in order to reduce noise. We even stopped using Nginx as a frontend loadbalancer to make sure we were really only measuring the pure time HipHop needs to render the page.

The Open Parallel WordPress repository can be found on github and there are some repositories that hold additional resources that might be needed to run the tests or come in handy for testing. We also published our Tsung config to illustrate how our typical user session looked like that we ran our tests with.

Memory Usage Reduction
Testing revealed that we were able to reduce the memory usage by about 100MB in our current version. This value moves quite a bit depending on the functions that are moved into the parallel_for construct and it is quite hard to come up with a MB/invocation number. We did some in-depth testing in that area and all we can say is that the amount of memory used was always below the amount the unpatched version used – from time to time not by much – but always below.

The parameter that was constantly below the unpatched version – and by a lot – was the connection time and the number of sessions handled in parallel. We were able to optimize in that area to the extend of more than one minute less time needed to complete our test run, and managed to get the amounts of pages served per second up to 7.2 compared to 6.3 in the unpatched version.

The attached graphs show the memory and CPU usage we recorded during the test runs and show the responses/sec for the different setups.

We are happy to supply more information if needed to replicate the tests on your systems and are happy to discuss the outcomes on the HipHop mailing list

HipHop - Standard - CPU

HipHop - standard - Memory

HipHop - Optimized - CPU

HipHop - Optimized - Memory

Pages - Seconds - Standard

Pages - Seconds - Optimized

© Open Parallel White Paper
November 2010

When we started out thinking about TBB (Threading Building Blocks) and its various implementations in script languages, PHP was a strong contender from day one.

PHP is very widely used, and some projects that started out as small pet projects on the side grew to massive sites that have to deal with all the usual scalability problems. Some of them decided to rewrite the whole platform in a new language that scales better -Amazon is one of them; others stick to the language, but try to optimize the performance.

To optimize the performance of PHP there are a number of screws that can be adjusted and the real execution performance of PHP is often only a minor concern. Database performance was a major problem for Yahoo! for a while. The improved art of caching is becoming more and more relevant, with Memcached, Varnish and Redis three very strong products complementing the older version of running Squid in front of the web server farm.

PHP execution speed has for a long time been an area of tuning as well, and the Zend optimizer was for a long time pretty much the only way to speed things up. The Zend optimizer is commercial and does not have a deep market penetration as far as we can tell. The other version of running PHP faster was phc, a PHP compiler that generated an optimized binary. Phc does not have a deep market penetration either and the project is not the most active.

Facebook is one or the largest shops running on a PHP stack in the moment and they were trying to speed up the execution time of PHP with a project called HipHop.

HipHop is Facebook’s answer to the lack of high performance optimizers for PHP and comes in two versions. The HipHop engine can either statically compile the entire PHP project to a binary that can be run with a HipHop wrapper, or run in a just-in-time compiler mode. The JIT mode is especially handy for development and debugging and is the mode we used for most of our development and testing.

HipHop takes a PHP source tree and compiles it first to optimized c++ and then compiles c++ to a binary that runs on Linux and some UNIX versions. It uses the GNU tool chain for compilation and makes use of TBB for memory management in the moment only.

As our focus was to improve the execution speed of a well known PHP project, we picked WordPress and ported it over to HipHop.

Our initial tests looked very promising and after further testing the community started patching WordPress as well and we saw patches in the WordPress root that made it possible to compile it on HipHop. As the WordPress team was moving forward new incompatibilities were introduced, but the support for up to date patch sets is improving on a daily basis.

We started measuring performance on a traditional Apache based LAMP stack and compared it to a HipHop based stack. Our development machines were standard Core 2 Duo machines running virtual box with the Ubuntu 10.04 LTS release and either Apache or HipHop and a second VM with our test code. We ran all the tests from the same box between VMs to reduce network latency and then on different physical machines to measure the impact of network overhead and reduce the impact of our monitoring. The network was standard wireless with 54MBit and no other significant traffic than the two VMs communicating with each other.

The test code was using Tsung and generated a Digg style attack with a max of 250 concurrent connections. The emitting side was timed to a total of 70 seconds starting with an interval of 1 second for new users then dropping down to an interval of a quarter of a second and then rising to a second again. Tsung is written in Erlang and can handle lots of concurrent connections in a very efficient way. In fact the impact of running the test code on the same physical machine was not as significant as we had expected, and the results via network were not too different to the ones run locally CPU and memory wise.

The problem with our test set-up was the lack of big enough machines to complete the Apache test entirely to the end. The development VMs ran on 1GB of memory and ran out of memory and started swapping pretty much instantly after starting the test. Even running Apache on a dedicated machine with 2GB of ram the attack maxed out the box pretty fast and the tests were cancelled after about 6 minutes as the box running Apache was not responding anymore.

The results we saw from running Apache was that initially the connection setup is really fast, but then the amount of resources Apache needs to process PHP are stacking up too fast and in the end the server swaps to death.

The next test was to run WordPress on HipHop and we had success with our tests. The time to complete the entire test was 4 minutes 20 seconds and we had at all times memory left on the server. The limiting factor here was the CPU, as far as we can tell.

The results are pretty obvious. Apache cannot be as optimized as a compiled binary version of the PHP project and we expected the HipHop version to run faster than the Apache version. The interesting bit was how fast Apache ran out of resources (which was after about the 100th connection) and how badly it ground to halt. The VM had to be killed after each test run and the testing code was not able to run through an entire test.

HipHop is a considerable improvement for running PHP projects and we can confirm a significantly reduced memory footprint. The problem seems to be CPU-bound now and we will address this in the next test.

© Open Parallel White Paper
October 2010

We tried to performance test the HipHop engine against Apache and then try to see how our changes to the HipHop engine influence the performance of WordPress.

We started out to performance test WordPress on Apache and HipHop. To get stats of the pure PHP performance we used a tool called tsung that is built for stress testing various protocols including HTTP.

Tsung can be configured to not fetch the entire site with all the static files like stylesheets and images but also to fetch only the PHP interpreted files. As we were not interested in performance numbers about static file serving we decided to only fetch PHP files with tsung.

Tsung can simulate a typical “viral attack” where a small number of concurrent connections start to build up load which then builds up to lots of concurrent connections and drops off to a normal base load. We configured tsung to run a maximum of 250 concurrent connections. The connections would be initiated in the simulator in three timed phases the times on the simulator are preset, the time that the server needs to answer all the queries will vary quite a bit as we can see later.

The performance numbers for the initial session connection times were slightly in favor of Apache but with growing numbers of simulated users the performance degraded quite dramatic whereas HipHop even in the just-in-time mode was very stable in terms of request times and resource consumption. The main problem with Apache was the resource consumption which simply maxed out our development servers.

The next step was to prove that our parallel_for implementation worked and actually increased the performance. In order to find a good place to hack WordPress and add our parallel_for to it we found wp_includes/plugin.php. In there is a function called apply_filters which had a for each loop that looked like it could benefit from parallel execution. apply_filters is used quite often in the WordPress code base so it seemed like a good point to get started.

We started off with the WordPress version of Hui Chen who has a very current patch set for running WordPress on HipHop and added our patches for the parallel_for. Our HipHop version is a fork off Hui Chen’s HipHop fork as well. We have a patch set for the parallel_for functionality that we have also pushed to our github account.

Running the benchmarks we can see that our parallel_for version uses less memory and generates less load on the server. It also had lower response times across all measured characteristics in tsung. The graphs also show that the overall time the test needed to complete our simulation was considerably longer for the version without parallel_for.

Lets read the numbers. The parallel_for version needed nearly 100MB less memory to run the test and was close to twice as fast. We moved a lot of work from PHP to the HipHop engine and measured that PHP would greatly benefit from a parallel_for function that runs native in C++. TBB looks like a perfect fit for those problems and our results show this quite nicely.

If we look at the impact this one change had onto the overall performance of WordPress on HipHop we can easily see how a wider implementation of parallel_for would benefit the performance on this stack.

the memory graph:

the CPU graph:

Here is the abstract with links

Integration of Intel’s Threading Building Blocks into Facebook’s HipHop for PHP

Hardware companies take the lead by shipping only Multicore processors as the standard for the industry. It is their solution to the power wall: reducing the operating voltage while making transistors smaller and faster. By using Multicore chips they will avoid that your latest laptop will burn your lap. Chip designers are concentrated now on gathering together lots of cores in a single chip: 4 cores are the norm today and chips with dozens and even hundreds of cores are already available or to be launched in the following few years.
But chipmakers forgot to check if the software that will exploit this massive processing power was available. Programmers now need to invent ways to write applications that take advantage of this increasing number of cores without spending too much time developing them or lowering its quality and performance. Not to mention how decades of legacy code will perform in the new hardware. (Read The Trouble with Multicore)

Is this context we established Open Parallel, a group that is working to create an ecosystem of developers, academia and industry to further multicore processing into the mainstream. The presentation will show the evolution and milestones that happened since 2005, but will focus on a current project: TBB into HipHop.

“Intel® Threading Building Blocks (TBB) is an open source library that helps to take advantage of multicore processor performance. It offers an approach to expressing parallelism in a C++ program.”

“HipHop for PHP transforms PHP source code into highly optimized C++. It was developed by Facebook and was released as open source in early 2010.”

At the beginning of 2010 we teamed up with Intel to get the word out about their TBB. Around that time, Facebook released their HipHop engine. We agreed to implement TBB into the HipHop engine as a first project to get our team working on TBB. We are now at a stage where we contributed back code to the community with some TBB features exposed and ready to implement in PHP projects via the HipHop engine.

In July 2010 we announced our HipHop branch/extension that aims to push the integration of TBB’s library into HipHop beyond the already utilized data structures (such as the
concurrent_hash_map). In particular we are looking to get some parallel functionality exposed to PHP itself through this extension. We have already had some initial success with a parallel_for implementation that can be used from PHP scripts using a familiar API (Aligned with the C++ API but slightly modified to better suit PHP style).

Our repository can be found at http://github.com/OpenParallel/hiphop-php, the extension is simply named tbb, and can be found in the usual paths in the source tree.

This experiment gave us some deeper insights into how parallel programming with TBB works and how we can start helping others use these technologies. At the time of writing the abstract, we are porting WordPress to HipHop with the new TBB-HipHop commands. We aim to show our code and full results at LCA2011

Today July 26, 2010 we announce our HipHop branch/extension that aims to push the integration of Intel’s Threading Building Blocks library into HipHop beyond the already utilized data structures (such as the concurrent_hash_map).
In particular we are looking to get some parallel functionality exposed to PHP itself through this extension.

We have already had some initial success with a parallel_for implementation that can be used from PHP scripts using a familiar API (Aligned with the C++ API but slightly modified to better suit PHP style).

Our repository can be found at http://github.com/OpenParallel/hiphop-php, the extension is simply named tbb, and can be found in the usual paths in the source tree.

The thread TBB in HipHop shows the discussion with Facebook engineers about our work