TideWatch: Fingerprinting the cyclicality of big data workloads.
TideWatch: Fingerprinting the cyclicality of big data workloads.Today, batch processing frameworks like Hadoop MapReduce are difficult to scale to multiple clouds due to latencies involved in inter-cloud data transfer and synchronization overheads during shuffle-phase. This inhibits the MapReduce framework from guaranteeing performance at variable load surges without over-provisioning in the internal cloud (IC). We propose BStream, a cloud bursting framework for MapReduce that couples stream-processing in the external cloud (EC) with Hadoop in the internal cloud (IC). Stream processing in EC enables pipelined uploading, processing and downloading of data to minimize network latencies.
We use this framework to meet job deadlines. BStream uses an analytical model to minimize the usage of EC. We propose different checkpointing strategies that overlap output transfer with input transfer/processing and simultaneously reduce the computation involved in merging the results from EC and IC. Checkpointing further reduces job completion time. We experimentally compare BStream with other related works and illustrate performance benefits due to stream processing and checkpointing strategies in EC. Lastly, we characterize the operational regime of BStream.
Similar IEEE Project Titles
- Spatial big data and wireless networks: experiences, applications, and research challenges.
- Pythia: Faster Big Data in Motion through Predictive Software-Defined Network Optimization at Runtime.
- Big data technologies in support of real time capturing and understanding of electric vehicle customers dynamics.
- Simulating Big Data Clusters for System Planning, Evaluation, and Optimization.
- MRPrePostA parallel algorithm adapted for mining big data.