基于Hilbert曲线的轨迹大数据分布式存储与索引方法

doi:10.13474/j.cnki.11-2246.2024.0619

测绘通报 ›› 2024, Vol. 0 ›› Issue (6): 109-114,138.doi: 10.13474/j.cnki.11-2246.2024.0619

• 学术研究 • 上一篇

基于Hilbert曲线的轨迹大数据分布式存储与索引方法

陈凯¹, 宋炜炜¹, 金宝轩², 李永宁¹, 普红勋¹

1. 昆明理工大学, 云南昆明 650093;
2. 云南省自然资源厅信息中心, 云南昆明 650224

收稿日期:2023-11-01 发布日期:2024-06-27
通讯作者: 宋炜炜。E-mail:weiweisong@kust.edu.cn
作者简介:陈凯(1999—),男,硕士生,主要研究方向为时空大数据、数据库存储和索引。E-mail:chenkai@stu.kust.edu.cn
基金资助:
云南省重大科技专项计划(202202AD080010)

A distributed storage and index method of trajectory big data based on the Hilbert curve

CHEN Kai¹, SONG Weiwei¹, JIN Baoxuan², LI Yongning¹, PU Hongxun¹

1. School of Land and Resource Engineering, Kunming University of Science and Technology, Kunming 650093, China;
2. Information Center Department of Natural Resources of Yunnan Province, Kunming 650224, China

Received:2023-11-01 Published:2024-06-27

摘要/Abstract

摘要： 针对具有时空特征的轨迹大数据快速增长的趋势及对其快速查询的需求,传统的关系型数据库对海量轨迹数据的存储和特定的查询需求具有一定局限性,而非关系型数据库又难以满足海量数据的高效索引需求,轨迹数据的存储和索引效率仍亟待提高。本文设计并实现了一种基于HBase数据库的存储和检索的框架,以应对时空轨迹数据的高效管理。首先,构建了一种新颖的Rowkey结构,结合时空索引工具构建了GeoMesa-HBase底层存储模型;然后,融合了基于Hilbert曲线的编码技术构建空间索引,提高了轨迹数据的存储和检索效率;最后,为了评估所提方法的有效性,将其存储和查询性能与传统存储数据库(HBase和MySQL)和Geohash索引进行比较。试验结果表明,本文方案能够有效提升轨迹数据的存储和检索效率,这对于应对轨迹大数据管理领域面临的多项挑战具有重要现实意义。

关键词: 轨迹数据, 时空索引, 时空存储, HBase, Hilbert曲线

Abstract: In response to the rapid growth of trajectory big data with spatio-temporal characteristics and the urgent need for its fast query, traditional relational databases have certain limitations on the storage of massive trajectory data and specific query requirements, while non-relational databases are difficult to meet the efficient indexing requirements of massive data, and the efficiency of the storage and indexing of trajectory data is still in urgent need of improvement. In this paper, a framework for storage and retrieval based on HBase database is designed and implemented to cope with the efficient management of spatio-temporal trajectory data. Firstly, a novel Rowkey structure is designed, and the GeoMesa-HBase underlying storage model is constructed by combining spatio-temporal indexing tools. Secondly, a Hilbert curve-based coding technique is integrated to construct the spatial index, which improves the storage and retrieval efficiency of trajectory data. In order to evaluate the effectiveness of the proposed method, this paper compares its storage and query performance with traditional storage databases (HBase and MySQL) and Geohash index. The experimental results show that the scheme is able to achieve effective storage of trajectory data and improve the retrieval efficiency, which is of great practical significance in addressing the challenges associated with trajectory big data management.

Key words: trajectory data, spatio-temporal index, spatio-temporal storage, HBase, Hilbert curve

中图分类号:

P208

陈凯, 宋炜炜, 金宝轩, 李永宁, 普红勋. 基于Hilbert曲线的轨迹大数据分布式存储与索引方法[J]. 测绘通报, 2024, 0(6): 109-114,138.

CHEN Kai, SONG Weiwei, JIN Baoxuan, LI Yongning, PU Hongxun. A distributed storage and index method of trajectory big data based on the Hilbert curve[J]. Bulletin of Surveying and Mapping, 2024, 0(6): 109-114,138.

参考文献

[1] YUAN Jing,ZHENG Yu,ZHANG Chengyang,et al. T-drive: driving directions based on taxi trajectories[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems.San Jose,Californi:ACM Press,2010:99-108.
[2] JIANG Weiwei,LUO Jiayun. Big data for traffic estimation and prediction: a survey of data and tools[J]. Applied System Innovation,2022,5(1): 23.
[3] ZHENG Yu. Trajectory data mining: an overview[J].ACM Transactions on Intelligent Systems and Technology,6(3):29.
[4] STONEBRAKER M. SQL databases v. NoSQL databases[J]. Communications of the ACM,2010,53(4): 10-11.
[5] DING X,CHEN L,GAO Y,et al. UlTraMan: a unified platform for big trajectory data management and analytics[J]. Proceedings of the VLDB Endowment,2018,11(7): 787-799.
[6] HE Huajun,LI Ruiyuan,BAO Jie,et al. JUST-traj: a distributed and holistic trajectory data management system[C]//Proceedings of the 29th International Conference on Advances in Geographic Information Systems.Beijing:ACM Press,2021: 403-406.
[7] TONG Xiaochong,CHENG Chengqi,WANG Rong,et al. An efficient integer coding index algorithm for multi-scale time information management[J]. Data & Knowledge Engineering,2019,119(C): 123-138.
[8] KWON D,LEE Sangjun,LEE S. Indexing the current positions of moving objects using the lazy update R-tree[C]//Proceedings of the 3rd International Conference on Mobile Data Management MDM. Singapore:IEEE,2002: 113-120.
[9] GUAN Xuefeng,BO Cheng,LI Zhenqiang,et al. ST-hash: an efficient spatiotemporal index for massive trajectory data in a NoSQL database[C]//Proceedings of 2017 International Conference on Geoinformatics.Buffalo,NY:IEEE,2017: 1-7.
[10] HUGHES J N,ANNEX A,EICHELBERGER C N,et al. GeoMesa: a distributed architecture for spatio-temporal fusion[C]//Proceedings of 2015 Geospatial Informatics,Fusion,and Motion Video Analytics V. Baltimore,MD:[s.n.],2015:128-140.
[11] LI Ruiyuan,HE Huajun,WANG Rubin,et al. TrajMesa: a distributed NoSQL storage engine for big trajectory data[C]//Proceedings of 2020 IEEE 36th International Conference on Data Engineering (ICDE). Dallas,TX:IEEE,2020: 2002-2005.
[12] LAWDER J K,KING P J H. Using space-filling curves for multi-dimensional indexing[C]//Proceedings of 2000 British National Conference on Databases. Berlin,Heidelberg: Springer Berlin Heidelberg,2000: 20-35.
[13] MOKBEL M F,AREF W G,KAMEL I. Analysis of multi-dimensional space-filling curves[J]. GeoInformatica,2003,7(3): 179-209.
[14] WU Y H,CAO X,AN Z. A spatiotemporal trajectory data index based on the Hilbert curve code[C]//Proceedings of 2020 IOP Conference Series: Earth and Environmental Science. [S.l.]:IOP Publishing,2020,502(1): 012005.
[15] 曹布阳,冯华森,梁峻浩,等. 利用Hilbert曲线与Cassandra技术实现时空大数据存储与索引[J]. 武汉大学学报(信息科学版),2021,46(5): 620-629.
[16] STONEBRAKER M,ABADI D J,BATKIN A,et al. C-store:a column-oriented DBMS[M]//Making Databases Work:the Pragmatic Wisdom of Michael Stonebraker.[S.l.]:Association for Computing Machinery,2018:491-518.
[17] GAO Fan,YUE Peng,WU Zhaoyan,et al. Geospatial data storage based on HBase and MapReduce[C]//Proceedings of the 6th International Conference on Agro-Geoinformatics. Fairfax,VA:IEEE,2017: 1-4.
[18] XIONG Shuyi,OUYANG Xue,XIONG Wei. Distributed or centralized: an experimental study on spatial database systems for processing big trajectory data[C]//Proceedings of 2023 IEEE International Conference on Big Data Analytics (ICBDA). Harbin:IEEE,2023:8-13.

基于Hilbert曲线的轨迹大数据分布式存储与索引方法

A distributed storage and index method of trajectory big data based on the Hilbert curve

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 14

编辑推荐

Metrics

本文评价

[1]	梁丽芳, 王燕云, 田时雨, 张恒才. 高精度CORS站像控点自动提取方法[J]. 测绘通报, 2022, 0(11): 123-127.
[2]	邬群勇, 王祥健. 出租车轨迹数据的频繁轨迹识别[J]. 测绘通报, 2021, 0(11): 70-75.
[3]	张昊, 张健钦, 郭小刚, 卢剑, 陆浩. 轨迹大数据云存储与热力图生成方法[J]. 测绘通报, 2021, 0(10): 146-149.
[4]	黄亚锋, 向隆刚, 高萌. 轨迹数据的非关系管理及相似性分析[J]. 测绘通报, 2020, 0(6): 81-86.
[5]	武英豪, 李成名, 吴政, 武鹏达. 基于地理格网的复杂路线车辆通行时间估算方法[J]. 测绘通报, 2019, 0(6): 61-65.
[6]	梁明, 陈文静, 段平, 李佳. 轨迹压缩的典型方法评价[J]. 测绘通报, 2019, 0(4): 60-64,70.
[7]	李金磊, 慈谕瑶, 郑坤, 郭绍龙, 燕继红, 陈宇萍, 吴艳民, 李仕漪. 面向地理信息大数据的时空事件关系可视化分析框架[J]. 测绘通报, 2019, 0(12): 101-104.
[8]	王善辉, 张和生. 一种基于车辆轨迹数据的城市建成区边界识别方法[J]. 测绘通报, 2019, 0(1): 56-59,64.
[9]	胡瀚, 向隆刚, 王德浩. 出租车轨迹数据的道路提取[J]. 测绘通报, 2018, 0(7): 53-57.
[10]	刘旭, 陈云波, 施昆, 黄强. 结合Canopy-K-means算法和出租车轨迹数据的公交车站预测方法[J]. 测绘通报, 2018, 0(11): 63-68.
[11]	赵庶旭, 屈睿涛, 刘昌荣. 一种基于轨迹特征划分的交通轨迹数据分析方法[J]. 测绘通报, 2018, 0(11): 73-77,82.
[12]	牟乃夏, 徐玉静, 张恒才, 陈洁, 张灵先, 刘希亮. 移动轨迹聚类方法研究综述[J]. 测绘通报, 2018, 0(1): 1-7.
[13]	段颖超, 张健钦, 李明轩, 杜明义. 一种公交到站时间预测方法[J]. 测绘通报, 2016, 0(5): 50-53.
[14]	范建永龙明熊伟. 基于Hadoop的云GIS体系结构研究[J]. 测绘通报, 2013, 0(11): 93-97.