In unity there is strength: Showcasing a unified big data platform Over both object and file storage with mapreduce ideas.
In unity there is strength: Showcasing a unified big data platform Over both object and file storage with mapreduce ideas.Big Data platforms often need to support emerging data sources and applications while accommodating existing ones. Since different data and applications have varying requirements, multiple types of data stores (e.g. file-based and object-based) frequently co-exist in the same solution today without proper integration. Hence cross-store data access, key to effective data analytics, can not be achieved without laborious application re-programming, prohibitively expensive data migration, and/or costly maintenance of multiple data copies.
We address this vital issue by introducing a first unified big data platform over heterogeneous storage. In particular, we present a prototype joining Apache Hadoop MapReduce with OpenStack’s open-source object store Swift and IBM’s cluster file system GPFSTM. A sentiment analysis application using 3 months of real Twitter data is employed to test and showcase our prototype. We have found that our prototype achieves 50% data capacity savings, eliminates data migration overhead, offers stronger reliability and enterprise support. Through our case study, we have learned important theoretical lessons concerning performance and reliability, as well as practical ones related to platform configuration. We have also identified several potentially high-impact research directions.
Similar IEEE Project Titles
- Design of handover self-optimization using big data analytics.
- A scalable machine learning online service for big data real-time analysis.
- Scalable big data computing for the personalization of machine learned models and its application to automatic speech recognition service.
- Addressing human bottlenecks in big data.
- Accelerating Spark with RDMA for Big Data Processing: Early Experiences.