In-Map/In-Reduce: Concurrent Job Execution in MapReduce
In-Map/In-Reduce: Concurrent Job Execution in MapReduce.Hadoop based Map Reduce (MR) has emerged as big data processing mechanism in terms of its data intensive applications. In data intensive systems, analysis and visualizations as a result of various algorithms can lead to differentiable and comparable results. Current implementations of MR facilitates to reuse the results of MR jobs in other MR jobs and to distribute the cloud resources among jobs.
However, very little work is done in terms of using same data for multiple algorithms at the same time in a single job using either shared resources or dynamic resource allocation based on the data and scheduling of Map Reduce jobs. In this paper we propose a method to execute multiple algorithms on same data in HDFS concurrently and to use the same available resources by dynamically managing the task assignment and results aggregation. Our proposed approach reduces the execution time and supports multiple algorithms execution in parallel. In-Map/In-Reduce shows 200% decrease in execution time.
Similar IEEE Project Titles
- Aeromancer: A Workflow Manager for Large-Scale MapReduce-Based Scientific Workflows
- In unity there is strength: Showcasing a unified big data platform with MapReduce Over both object and file storage
- Energy-aware Scheduling of MapReduce Jobs for Big Data Applications
- End-to-end Optimization for Geo-Distributed MapReduce
- Hashdoop: A MapReduce framework for network anomaly detection