| Module Info
| Add a review of MCE
I recently had a task that involved a 500G file. This fixed length file needed to be flipped or pivoted based on the content in certain fields. My first attempt at this was taking from 6 to 7 days, so I decided to give this module a try and got it down to a little over 2 days - amazing results! My only problem was understanding some of the documentation, which was probably because my programming orientation is more procedural than object oriented. But once I got some help from someone more knowledgeable, everything clicked into place. This is a great module!!
We are using MCE to monitor and manage over 20,000 devices on our network. We do a get of 20 metrics and a walk of two trees of over 20,000 devices in about 100 seconds using MCE! MCE is truly invaluable - we would not be able to scale our service without it!
The below is a test running on a Sun T5120 with eight cores each at 1.165 GHz. This chip has 32 logical cores.
$ ls -l o.ap
-rw-r--r-- 1 red other 929645146 Nov 9 19:46 o.ap
## binary wc utilizes one core
$ time wc -l o.ap
## wc.pl example in MCE
$ time ./wc.pl --max_workers 4 -l o.ap
$ time ./wc.pl --max_workers 8 -l o.ap
$ time ./wc.pl --max_workers 16 -l o.ap
## widefinder.pl example in MCE
$ ./wide_finder.pl --max_workers 9 o.ap
## Compute time: 1.818
The Sun OS box tested here runs at 1.165 GHz.
The Sun OS box at www.tbray.org/ongoing/When/200x/2007/...
was running at 1.4 GHz.
The fastest time submitted was a perl script utilizing Sys::Mmap and 32 workers
taking 1.51 seconds.
So, in comparison and accounting for the processor speed difference, MCE is
able to perform very well utilizing 9 workers. 1.51 / 1.818 * 1.4 GHz = 1.163 GHz
This module has been a HUGE time saver. Programs processing very large volumes of data that were running in hours are now running in 10 minutes or less. Just turn the parts of your script that take a long time to run when done serially into subroutines and this module will give you the ability to process parts of it in parallel and aggregate it back up.
Nice work, Mario! Don't let anyone tell you otherwise.