stevecreedon stellt das Thema vorRuby Storm - Distributed work on a self-scaling system using RabbitMQ
Some time ago we reduced a 27 hour process to less than 20 minutes by distributing the workload using RabbitMQ. The system has evolved into something akin to Java Storm but in Ruby. It self-scales using our own scaling logic that analyses queue sizes, deploys additional workers where needed then adds servers there aren't enough for the extra workers - and scales down too of course.
The system is agnostic of the task at hand. At the moment we use it update pricing, property and availability at Wimdu but really it could be used for any large data crunching task and writing ruby workers is very easy.
NOTE: I won't be back in Berlin until 10th Jan so will have to wait until Feb talks at earliest