Language based web crawling on big data.
Language based web crawling on big data. Online textual and visual data that are created and used by web users have been increasing dramatically and continually. This increase has caused the need for easy and fast access to online data and facilitated the development of alternative means of access to this data. Nowadays, web crawlers are the most efficient and popular tools used for accessing big volumes of data available on the web.
In this paper, a web crawler which works on a distributed Hadoop cluster for crawling web pages with content of a predefined language is described. A language identification tool is developed for enabling the system to focus only on a specific language. In this study, the accuracy of the language identification tool is evaluated on a small data set (consisting of 4729 web pages). The performance of the focused web crawling system is reported on a big data set of 86 million web pages containing Turkish content
Similar IEEE Project Titles
- A Big Data Architecture for Large Scale Security Monitoring.
- Experiments with computing similarity coefficient over big data.
- A Google approach for computational intelligence in big data .
- Toward personalized and scalable voice-enabled services powered by big data.
- A standard for benchmarking big data systems .