This project leverages big data technologies to build a comprehensive platform for agricultural market analysis. It combines web scraping, distributed data processing, and visualization technologies to empower agricultural decision-making.
The system consists of four modules: data collection, storage, processing, and visualization. Data is sourced via web crawlers to capture both historical and real-time data. These are stored in HDFS and processed with SQL pipelines. Processed data is organized into a Hive data warehouse and stored in MySQL for efficient querying.