In unity there is strength: Showcasing a unified big data platform with MapReduce Over both object and file storage
In unity there is strength: Showcasing a unified big data platform with MapReduce Over both object and file storage. Big data platforms often need to support emerging data sources and applications while accommodating existing ones. Since different data and applications have varying requirements, multiple types of data stores (e.g. file-based and object-based) frequently co-exist in the same solution today without proper integration. Hence cross-store data access, key to effective data analytics, can not be achieved without laborious application re-programming, prohibitively expensive data migration, and/or costly maintenance of multiple data copies.
We address this vital issue by introducing a first unified big data platform over heterogeneous storage.In particular, we present a prototype joining Apache Hadoop MapReduce with OpenStack’s open-source object store Swift and IBM’s cluster file system GPFSTM. A sentiment analysis application using 3 months of real Twitter data is employed to test and showcase our prototype. We have found that our prototype achieves 50% data capacity savings, eliminates data migration overhead, offers stronger reliability and enterprise support. Through our case study, we have learned important theoretical lessons concerning performance and reliability, as well as practical ones related to platform configuration. We have also identified several potentially high-impact research directions.
Similar IEEE Project Titles
- Energy-aware Scheduling of MapReduce Jobs for Big Data Applications
- End-to-end Optimization for Geo-Distributed MapReduce
- Hashdoop: A MapReduce framework for network anomaly detection
- FedLoop: Looping on Federated MapReduce
- Impact of MapReduce Task Re-execution Policy on Job Completion Reliability and Job Completion Time