EXTRACTING DATA FROM WEBSITES ON THE EXAMPLE OF DEVELOPING A PLUGIN FOR WEB SCRAPING

Abstract

Internet search involves not only getting acquainted with this or that information in order to gain new knowledge, it also involves comparing, analyzing and summarizing data found on different websites and pages. This problem can be solved by monotonous manual copying of the required information into your own files and formatting them according to your needs. Such a process is inefficient because it takes a lot of time.

Web scraping is an automated process of extracting huge amounts of data from websites and converting it into structured data. Programs that carry out such processes are called web scrapers and are able to retrieve required HTML content, work with JavaScript, filter received data and output it as a ready-to-use database, Excel spreadsheet, CSV file or separate API etc.

Presented is an implementation option of web scraping plugin for Chromium-based browsers. The plugin has client and server sides. Implementing client part involves designing and creating interface of plugin pop-up window, web page elements highlight on hover. Server side of the plugin analyzes requests, generates links and performs element selection.