Dynamic data rebalancing in Hadoop.
Dynamic data rebalancing in Hadoop.Current implementation of Hadoop is based on an assumption that all the nodes in a Hadoop cluster are homogenous. Data in a Hadoop cluster is split into blocks and are replicated based on the replication factor. Service time for jobs that accesses data stored in Hadoop considerably increases when the number of jobs is greater than the number of copies of data and when the nodes in Hadoop cluster differ much in their processing capabilities.
This paper addresses dynamic data rebalancing in a heterogeneous Hadoop cluster. Data rebalancing is done by replicating data dynamically with minimum data movement cost based on the number of incoming parallel mapreduce jobs. Our experiments indicate that as a result of dynamic data rebalancing service time of mapreduce jobs were reduced by over 30% and resource utilization is increased by over 50% when compared against Hadoop.
Similar IEEE Project Titles
- Performance evaluation of HDD and SSD on 10GigE, IPoIB & RDMA-IB with Hadoop Cluster Performance Benchmarking System .
- Workload Analysis, Implications, and Optimization on a Production Hadoop Cluster: A Case Study on Taobao.
- Job scheduling in Hadoop with Shared Input Policy and RAMDISK .
- Investigating the inclinations of research and practices in Hadoop: A systematic review
- Performance Evaluation of Read and Write Operations in Hadoop Distributed File System