关键词: 水质监测, 分布式处理框架, 水质指数, 云南省九大高原湖泊

Abstract: The rapid accumulation and growth of water quality monitoring data brings new opportunities and challenges to water quality research. In view of the problems of poor timeliness, unintuitive data display, and inefficient data processing faced by water quality monitoring and analysis at present, this paper constructs WaterSpark, a distributed processing framework for real-time water environment monitoring and big data analysis, based on big data technology and data visualization technology, using the improved Canadian Council of Ministers of the Environment water quality index (CCME-WQI) and the Spark machine learning library (MLlib), Applying the water quality monitoring data of nine plateau lakes in Yunnan province, the results show that WaterSpark has excellent performance in real-time water quality transmission, cleaning and archiving, and efficient computing and analysis. It can enable large-scale water quality data to be captured and analyzed timely and accurately, and the distributed data sets and clusters can cope with the growing water quality data to ensure performance scalability, and to support more water quality indicators and water quality monitoring on a larger scale.

Key words: water quality monitoring, distributed processing framework, water quality index, nine plateau lakes in Yunnan province