A Discrete Data Dividing Approach for Erasure-Code-Based Storage Applications
A Discrete Data Dividing Approach for Erasure-Code-Based Storage Applications.Erasure codes are promising for improving the reliability of the storage system due to its space efficiency compared to the replication methods. Traditional erasure codes split data into equalsized datablocks and encode strips in different data blocks. This brings heavy repairing traffic when clients read parts of the data, since most strips read for repairing are not in the expected blocks.
This paper proposes a novel discrete data dividing method to completely avoid this problem. The key idea is to encode strips from the same data block. We could see that for repairing failed blocks, the strips to be read are either in the same data block with corrupted strips or from the encoded strips. Therefore, nodata is wasted. We design and implement this data layout into a HDFS-like storage system. Experiments over a small-scale testbed shows that the proposed discrete data divided method avoids downloading data blocks that are not needed for clients during the repairing operations.
Similar IEEE Project Titles
- Processing performance on Apache Pig, Apache Hive and MySQL cluster
- Method and system for educational networking1
- Efficient traffic speed forecasting based on massive heterogenous historical data
- SETiNS: Storage efficiency techniques in No-SQL database for Cloud based design
- Distributed in-memory cluster computing approach in scala for solving graph data applications