Scaling MapReduce Vertically and Horizontally
Scaling MapReduce Vertically and Horizontally.Glass wing is a MapReduce framework that uses OpenCL to exploit multi-core CPUs and accelerators. However, compute device capabilities may vary significantly and require targeted optimization. Similarly, the availability of resources such as memory, storage and interconnects can severely impact overall job performance. In this paper, we present and analyze how MapReduce applications can improve their horizontal and vertical scalability by using a well controlled mixture of coarse- and fine-grained parallelism.
Specifically, we discuss the Glass wing pipeline and its ability to overlap computation, communication, memory transfers and disk access.Additionally, we show how Glass wing can adapt to the distinct capabilities of a variety of compute devices by employing fine-grained parallelism. We experimentally evaluated the performance of five MapReduce applications and show that Glass wing outperforms Hadoop on a 64-node multi-core CPU cluster by factors between 1.2 and 4, and factors from 20 to 30 on a 23-node GPU cluster. Similarly, we show that Glass wing is at least 1.5 times faster than GPMR on the GPU cluster
Similar IEEE Project Titles
- MRSMRS: Mining repetitive sequences in a MapReduce setting
- Scheduling MapReduce tasks on virtual MapReduce clusters from a tenant’s perspective.
- Incremental FP-Growth mining strategy for dynamic threshold value and database based on MapReduce
- MapReduce-based warehouse systems: A survey
- Improving MapReduce Performance Using Smart Speculative Execution Strategy