Semantic segmentation of 3D real scene based on segment anything model

doi:10.13474/j.cnki.11-2246.2024.1216

Abstract

Abstract: Scene semantic segmentation based on deep learning and computer vision technology is currently a hot research topic. This paper proposes a 3D real scene semantic segmentation framework that includes “scene input-preprocessing-model inference-semantic segmentation”. By transforming the 3D real scene as input into multi-view 2D images through orthogonal projection, segmentation inference is carried out, and segmentation masks are generated and further processed,achieving the object selection, singulation, and semantic processing of 3D real scene.The experiments show that the method has good semantic segmentation performance and efficiency.

Key words: segment anything model, 3D real scene, semantic segmentation

CLC Number:

P237

LI Feng, XUE Mei, ZHAN Yong, YANG Yuan. Semantic segmentation of 3D real scene based on segment anything model[J]. Bulletin of Surveying and Mapping, 2024, 0(12): 101-105.

References

[1] 张荣庭,张广运,尹继豪.复杂城市动态图卷积网络三维场景语义分割法[J].测绘学报,2023,52(10):1703-1713.
[2] KIRILLOV A,MINTUN E,RAVI N,et al. Segment anything[EB/OL].[2023-04-23].https://arxiv.org/abs/2304.02643v1.
[3] ZHANG Chunhui,LIU Li,CUI Yawen,et al.A comprehensive survey on segment anything model for vision and beyond[EB/OL].[2023-05-23].https://arxiv.org/abs/2305.08196v2.
[4] 王淼,黄智忠,何晖光,等.分割一切模型SAM的潜力与展望:综述[J].中国图象图形学报,2024,29(6):1479-1509.
[5] GLENN Jocher,AYUSH Chaurasia,JING Qiu.Ultralytics YOLOv8[EB/OL].[2024-06-23].https://docs.ultralytics.com/models/yolov8/.
[6] LIANG Weicong,YUAN Yuhui,DING Henghui,et al.Expediting large-scale vision transformer for dense prediction without fine-tuning[EB/OL].[2022-10-22].https://arxiv.org/abs/2210.01035v1.
[7] XIONG Yunyang,VARADARAJAN B,WU Lemeng,et al.EfficientSAM: leveraged masked image pretraining for efficient segment anything[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2024:16111-16121.
[8] ZHAO Xu,DING Wenchao,AN Yongqi,et al.Fast segment anything[EB/OL].[2023-06-23].https://arxiv.org/abs/2306.12156v1.
[9] LI Bo,XIAO Haoke,TANG Lv.ASAM: boosting segment anything model with adversarial tuning[EB/OL].[2024-06-10].https://arxiv.org/abs/2405.00256v1.
[10] XU Xiuwei,CHEN Huangxing,ZHAO Linqing,et al.EmbodiedSAM: online segment any 3D thing in real time[EB/OL].[2024-06-10].https://arxiv.org/abs/2408.11811
[11] 李帅,初启凤,袁如金,等.SAM在遥感影像通用语义分割中的应用[J].测绘与空间地理信息,2024,47(S1):1-4.
[12] 杨博,谢杨柳,张天航,等.基于SAM的无人艇多源图像分割研究[J].无人系统技术,2024,7(2):73-81.
[13] 孙兴,蔡肖红,李明,等.视觉大模型SAM在医学图像分割中的应用综述[J].计算机工程与应用,2024,60(17):1-16.
[14] 吴曈,胡浩基,冯洋,等.分割一切模型(SAM)在医学图像分割中的应用[J/OL].中国激光,1-30[2024-10-30].http://kns.cnki.net/kcms/detail/31.1339.TN.20240520.1426.004.html.
[15] ZHANG Chaoning,HAN Dongshen,QIAO Yu,et al.Faster segment anything: towards lightweight SAM for mobile applications[EB/OL].[2024-06-10].https://arxiv.org/abs/2306.14289v2.