In-memory I/O and replication for HDFS with Memcached: Early experiences
In-memory I/O and replication for HDFS with Memcached: Early experiences.Hadoop is the de-facto standard platform for large-scale data analytic applications. In spite of high availability and reliability guarantees, Hadoop Distributed File System (HDFS) suffers from huge I/Obottlenecks for storing the tri-replicated data blocks. The I/O overheads intrinsic to the HDFSarchitecture degrade the application performance. In this paper, we present a novel design (MEM-HDFS) to perform intelligent caching and replication of HDFS data blocks in Memcached that can significantly improve the I/O performance.
In this design, we consider different deployment strategies for the Memcached servers (local and remote) and guarantee persistence of the Memcached data toHDFS on cache replacements. Performance evaluations show that MEM-HDFS can increase the read and write throughput of HDFS by up to 3.9x and 3.3x, respectively. Our design can also significantly speed up the data loading (to HDFS) phase. It reduces the execution times of data generation benchmarks like, TeraGen, RandomTextWriter, and RandomWriter by up to 50%, 39%, and 48%, respectively. The performances of other benchmarks like TeraSort and Grep are also improved by the proposed design.
Similar IEEE Project Titles
- Situation aware computing for big data.
- Big Data Infrastructure for analyzing data generated by Wireless Sensor Networks.
- Mobile Internet big data platform in China Unicom.
- Big data for better health planning.
- Making Big Data, Privacy, and Anonymization Work Together in the Enterprise: Experiences and Issues.