基于NeRF的城市实景高精度三维建模技术

doi:10.13474/j.cnki.11-2246.2024.0422

摘要/Abstract

摘要： 为了更好地将NeRF高精度三维建模应用于城市实景三维数字化重建,本文基于NeRF技术将大型场景划分为子NeRF,通过在场景内构造多个正八面体初始化多边形网格,并在训练过程中不断优化多边形面的顶点。训练完成后,得到编码器-解码器网络的权重,通过顶点优化对每个独立块进行不同层次的多边形网格细化。从视图范围捕获城市概览的卫星级图像到显示建筑复杂细节的地面级图像变化,构建面向城市细节层级(level-of-detail)和空间覆盖范围的多尺度数据,通过渐进式学习(progressive learning)神经网络体素渲染模型,使用多层感知器(MLP)实现体积密度和颜色的参数化,采用分层抽样方法实现预定义视角近平面和远平面之间光线的排序距离向量,实现大规模场景的实时交互式渲染。然后将GIS与NeRF相融合,为多数据融合、查询、分析、测量、标注和共享等任务提供了全新的解决方案,实现即时、流畅地拖拽、缩放和360°无死角地浏览和观看场景,这种融合可以轻松地将各种数据源整合,应用于城市规划、土地管理和环境监测等三维场景中,以进行空间分析。

关键词: 三维建模, NeRF神经辐射场, 训练, 渲染

Abstract: In order to better apply NeRF high-precision 3D modeling in the 3D digital reconstruction of urban real scenes,this paper divides the large scene into sub-NeRF based on NeRF technology,and initializes the polygon mesh by constructing multiple octahedral bodies in the scene. And the vertices of the polygon faces are continuously optimized during the training process. After the training is completed,the weights of the encoder-decoder network are obtained,and different levels of polygon mesh refinement are performed on each independent block through vertex optimization. From satellite-level images that capture city overviews to ground-level images that show complex details of buildings,multi-scale data for urban detail and spatial coverage are constructed through progressive learning. The neural network voxel rendering model uses a multilayer perceptron (MLP) to realize the parameterization of volume density and color,and uses a hierarchical sampling method to realize the sorting distance vector of rays between the near plane and the far plane of a predefined viewing angle,so as to realize real-time interactive rendering of large-scale scenes. Then,GIS and NeRF are fused to provide a new solution for tasks such as multi-data fusion,query,analysis,measurement,annotation and sharing,so as to achieve instant and smooth dragging,zooming and 360° browsing and viewing of scenes without dead ends. This fusion makes it easy to integrate various data sources for spatial analysis in 3D scenarios such as urban planning,land management and environmental monitoring.

Key words: 3D modeling, NeRF neural radiance field, training, rendering

中图分类号:

P208

孙建华, 李巍, 袁伟哲, 王峰, 谷佳铭. 基于NeRF的城市实景高精度三维建模技术[J]. 测绘通报, 2024, 0(4): 129-134.

SUN Jianhua, LI Wei, YUAN Weizhe, WANG Feng, GU Jiaming. High-precision 3D modeling technology of urban real scene based on NeRF[J]. Bulletin of Surveying and Mapping, 2024, 0(4): 129-134.

参考文献

[1] FRIDOVICH K,YUA M,et al. Plenoxels:radiance fields without neural networks[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans:IEEE,2022:5501-5510.
[2] PARK K,SINHA U,BARRONJ T,et al. Nerfies:deformable neural radiance fields[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV).Montreal:IEEE,2021:5865-5874.
[3] BARRONJ T,MILDENHALL B,VERBIN D,et al. Mip-NeRF 360:unbounded anti-aliased neural radiance fields[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).New Orleans:IEEE,2022:5470-5479.
[4] CHEN Zhiqin, FUNKHOUSER T,HEDMAN P,et al. Mobilenerf:Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures[C]//Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver:IEEE,2023.
[5] DENG Kangle,LIU A,ZHU Junyan,et al. Depth-supervised NeRF:fewer views and faster training for free[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).New Orleans:IEEE,2022:12882-12891.
[6] XIANG LIYuanbo,XU Linning,PAN Xingang,et al. BungeeNeRF:progressive neural radiance field for extreme multi-scale scene rendering[C]//Proceedings of EuropeanConference on Computer Vision. Cham:Springer,2022:106-122.
[7] TURKI H,RAMANAN D,SATYANARAYANAN M. Mega-NeRF:scalable construction of large-scaleNeRFs for virtual fly-throughs[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans:IEEE,2022:12922-12931.
[8] TANCIK M,CASSER V,YAN Xinchen,et al. Block-NeRF:scalable large scene neural view synthesis[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans:IEEE,2022:8248-8258.
[9] MILDENHALL B,SRINIVASAN P P,TANCIK M,et al. NeRF[J]. Communications of the ACM,2022,65(1):99-106.
[10] BOTSCH M,KOBBELT L, PAULY M,et al. Polygon mesh processing[M].Watic,MA,US:CRC press,2010.
[11] GUO M,FATHI A,WU Jiajun,et al. Object-centric neural scene rendering[J].Computer Vision and Pattern Recognition, 2020:08503.
[12] MARTIN B,RADWAN N,SAJJADIMS M,et al. NeRF in the wild:neural radiance fields for unconstrained photo collections[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville:IEEE,2021:7210-7219.
[13] MVLLER T,EVANS A,SCHIED C,et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics,2022,41(4):1-15.
[14] NEFF T,STADLBAUER P,PARGER M,et al. DONeRF:towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks[J]. Computer Graphics Forum,2021,40(4):45-59.
[15] NIEMEYER M,GEIGER A. GIRAFFE:representing scenes as compositional generative neural feature fields[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville:IEEE,2021:11453-11464.
[16] 赵宇滨,倪欢,牛晓楠. 基于改进CycleMLP的高分遥感图像采石场识别[J]. 测绘通报,2023(2):84-90.