MCE reviews

RSS | Module Info

MCE (1.834) *****

MCE is brilliant. It makes it easy to speed up a wide variety of applications just by wrapping them in a simple of lines of code. The interfaces are sometimes a little non-obvious, but it's infinitely simpler to use MCE than handling parallel processing by hand. Now that I know about it, I find myself using MCE in all sorts of applications.

MCE (1.515) *****

I recently had a task that involved a 500G file. This fixed length file needed to be flipped or pivoted based on the content in certain fields. My first attempt at this was taking from 6 to 7 days, so I decided to give this module a try and got it down to a little over 2 days - amazing results! My only problem was understanding some of the documentation, which was probably because my programming orientation is more procedural than object oriented. But once I got some help from someone more knowledgeable, everything clicked into place. This is a great module!!

MCE (1.002) *****

We are using MCE to monitor and manage over 20,000 devices on our network. We do a get of 20 metrics and a walk of two trees of over 20,000 devices in about 100 seconds using MCE! MCE is truly invaluable - we would not be able to scale our service without it!

MCE (1.002)

The below is a test running on a Sun T5120 with eight cores each at 1.165 GHz. This chip has 32 logical cores.

$ ls -l o.ap
-rw-r--r-- 1 red other 929645146 Nov 9 19:46 o.ap

## binary wc utilizes one core

$ time wc -l o.ap

4625247 o.ap

real 0m36.031s
user 0m34.167s
sys 0m1.864s

## example in MCE

$ time ./ --max_workers 4 -l o.ap

4625247 o.ap

real 0m6.676s
user 0m24.002s
sys 0m1.741s

$ time ./ --max_workers 8 -l o.ap

4625247 o.ap

real 0m3.820s
user 0m25.939s
sys 0m1.911s

$ time ./ --max_workers 16 -l o.ap

4625247 o.ap

real 0m2.911s
user 0m34.679s
sys 0m2.333s

## example in MCE

$ ./ --max_workers 9 o.ap
41164 2006/09/29/Dynamic-IDE
9252 2006/07/28/Open-Data
6016 2003/07/25/NotGaming
3703 2006/01/31/Data-Protection
3700 2003/09/18/NXML
3700 2003/10/16/Debbie
3238 2003/06/23/SamsPie
2778 2005/11/03/Cars-and-Office-Suites
2778 2005/07/27/Atomic-RSS
2778 2006/01/08/No-New-XML-Languages

## Compute time: 1.818

The Sun OS box tested here runs at 1.165 GHz.

The Sun OS box at
was running at 1.4 GHz.

The fastest time submitted was a perl script utilizing Sys::Mmap and 32 workers
taking 1.51 seconds.

So, in comparison and accounting for the processor speed difference, MCE is
able to perform very well utilizing 9 workers. 1.51 / 1.818 * 1.4 GHz = 1.163 GHz

MCE (1.002) *****

This module has been a HUGE time saver. Programs processing very large volumes of data that were running in hours are now running in 10 minutes or less. Just turn the parts of your script that take a long time to run when done serially into subroutines and this module will give you the ability to process parts of it in parallel and aggregate it back up.

Nice work, Mario! Don't let anyone tell you otherwise.