(2021-10-09)3D-Reconstruction Review(3)

Advances in Geometry and Reflectance Acquisition (Course Notes)

内容:

• accurate acquisition methods for geometry
  and reflectance
• efficient acquisition pipeline to fulfill the demands of industry with respect to mass digitization of 3D contents
• different types of reflectance behavior ranging from diffuse over opaque to specular surfaces or even translucent and transparent surfaces

局限性:

• only well-suited for a limited range of surface materials
• not adequate if no prior information with respect to the surface reflectance behavior is available
• also discuss recent advances towards an efficient, fully automatic acquisition in the scope of the concluding remarks

1.Introduction

对于人眼来说:

• 观察到的颜色和材质 not appear arbitrarily,而是由每个材料不同的反射行为导致的,这些反射行为基于材料的特征以及外表面的形状,材质特点,光照特征的共同作用.
• 通过对已经观察到的物体和材料的影响,可以推处further insights ,关于物理或者功能的材质
• Indeed, based on visual perception, we not only get impressions about a characteristic look but also an accompanying “feel” for materials.

但是会碰到很多的digital objects or materials,存在这样的问题:
there is often no alternative to an accurate digitization of physical objects including both geometry and reflectance characteristics where even the finest details of surface geometry and surface materials should be accurately captured.

1.1 Material Acquisition in Industry and Object Digitization in Cultural Heritage

• 电影/游戏/广告等应用中的digital materials应该transport look and feel的特征,使得其表现的像真的一样,来增加其所在场景的真实性.
• In order to obtain accurate digitized surrogates of physical materials,both the surface profile and the surface reflectance behavior have to be acquired appropriately.
• 除了上述的应用之外,还有很多应用需要3D的形状和反射行为,比如 digital preservation of objects for cultural heritage,比如通过 highly optimized devices as discussed in:[].
• 虽然geometry常以点云等形式存储,但是反射率以n terms of different reflectance functions that describe the material appearance depending on a multitude of involved parameters进行存储

  1.2 The Acquisition Ecosystem

  
• Hardware: required for the acquisition.包括各种sensor,照明设备,用来build a gantry或者搭建其他的设备的部件.
• Calibration:techniques for the radiometric and geometric calibration of the involved components.通过Calibration techniques可以在多个view以及多个光照条件下观察物体.这需要考虑组件的arrangement以及sensor和照明设备的特征.
• geometry acquisition techniques:为了可信的重建,该技术需要被用来得到geometry足够的细节.各种文献中已证明,不存在通用的处理可能由所有material组成的物体的技术,任意技术都局限在a limited fraction of the materials.
• reflectance acquisition techniques:一样是a limited fraction of the materials.由于其复杂性导致了a number of parameters are resolved and measured.各种发射光照模型需要different acquisition devices进行配合.由于measure的参数越多,需要的时间就越长,所以人们选择合适的,只得到需要参数的模型.
• visualization:visualization techniques to depict the digitized models.
• assistance systems:increase the efficiency,such as e.g. a prior material recognition step

1.3 Course Objectives

• accurately capturing the 3D shape of objects and optical properties of materials is especially challenging for objects made of materials with complex reflectance behavior
• 没有通用的方法,user typically selects appropriate ones based on his experience
  
• 对创建大量数字内容的需求也将重点转向automatic and highly efficient solutions.由于这里可能没有先验知识,the presence of the individual, occurring surface materials should guide the acquisition process.
• instead of naively processing(上面图片的过程中),只有那种物体表面部分的反射行为可方法对应时,才选择他们,这样更加高效.而且,based on the material properties, there is also the possibility to automatically detect cases where none of the available reconstruction techniques is appropriate.从而在actual acquisition之前,这里需要有个可信的对present surface materials.的识别,然后利用该信息取指导acquision的过程.
  
• 这篇survey主要将三个方面的问题
  1.advances in geometry acquisition(section 4),
  2.advances in reflectance acquisition(section 5),
  \3. concepts towards a more efficient automatic acquisition process(section 6)

2.Preliminaries of Material Appearance

• 分析mmaterial apperance,对material的性质以及他们如何能够从image content推导出来的研究是很重要的
• key observation:the visual complexity of surface appearance is characterized by the complex interplay of surface material, surface geometry and illumination.
• a chicken-and-egg problem:standard acquisition devices值能够捕获the coupling of the respective modalities,而又需要将这些模式分开进行分析,而这又需要每个模式相关的priori information.
• Section 2.1: the characteristics of material appearance and discuss the dependency of material appearance with respect to scale, illumination and surface geometry.
• Section 2.2:从与反射性质导出的光线传输特征进行的a taxonomy of surface classes(relevant for 3D geometry acquisition)
• Section 2.3: an overview of commonly used reflectance models

  2.1 Basics of Material Appearance

  We first may have a closer look at the underlying physical effects that characterize material appearance.Section 2.1.1中回顾radiometry 然后咋2.1.2中讨论 the physical background with respect to light exchange at material surfaces

  2.1.1 Radiometric Quantities

• ray optics:由于光线直线传播的特点,光通常被表示成straight lines,并称之为 ray representation
  A ray $\mathbf{r}$ can be parameterized as a mapping $\mathbb{R}^{+} \rightarrow \mathbb{R}^{3}$ using the ray origin $\mathbf{o} \in \mathbb{R}^{3}$ and a ray direction $\mathbf{d}=\left(d_{1}, d_{2}, d_{3}\right)^{T}$
  $$
  \begin{aligned}
  \mathbf{r}: \mathbb{R}^{+} & \rightarrow \mathbb{R}^{3} \
  s & \mapsto \mathbf{o}+s \mathbf{d}
  \end{aligned}
  $$
• 对于polychromatic光(多色光,单色光称为monochromatic),常被表示为a spectral power distribution $\lambda\in R^+\to L(\lambda)\in R^+$,其中L是 electromagnetic radiative power.这些波长的框分为 ultraviolet (UV) spectrum,infrared (IR) spectrum以及可见光wavelengths λ between 380nm (violet) and 780 nm (red).本课程中,光一般用tristimulus values R/G/B 表示,他们的值可以通过derived from a spectral power distribution based on inner products with suitable color matching functions such as e.g. CIE RGB [Smith and Guild 1931].得到
• radiant energy:Q[J],

the radiant flux or radiant power:$\Phi[J s^{-1}]=\frac{dQ}{dt}$
(flux中没有表示方向,但是光的反向是存在的,比如glossy or
specular materials)
irradiance:$E=\frac{d\Phi}{dA}[Wm^{-2}]$( arriving at the surface)
exitance:$E=\frac{d\Phi}{dA}[Wm^{-2}]$( leaving the surface)
对于surfaces with inhomogeneous flux,the irradiance or exitance depends on the local surface point x,$E=E(x)$
对于surfaces with homogeneous flux,the irradiance can be represented by the total flux per surface area
intensity:$I=\frac{d\Phi}{d\omega}[W~sr^{-1}]$(The unit of the solid angle is Steradian [sr])
It defines the flux per differential solid angle.
radiance:$L=\frac{d\Phi}{dAd\omega}[Wm^{-2}sr^{-1}]$(这里的A是projected differential area dA,沿着light flow的direction),所以又有公式
$L(\omega)=\frac{d^2\Phi}{dA_{\perp}d\omega}=\frac{d^2\Phi}{dAcos\theta d\omega}$,其中inclination angle θ is defined between the local surface normal n and the direction of the light flow
重要公式:到达某个面元的irradiance可以通过在该表面的upper hemisphere Ω上进行积分得到
$$
E=\frac{\mathrm{d} \Phi}{\mathrm{d} A}=\int_{\Omega} L \cos (\theta) \mathrm{d} \omega
$$
可以发现,这里的$\theta$和$\omega$是相关的.

• 在真空的假设下,, the radiance remains constant along a ray [Glassner 1995].而对于medium的情况,光线传播可以建模为interaction events that change the power of the light ray.
• Polarization:describes the orientation of the electromagnetic wave perpendicular to the propagation direction in space.在识别技术中,偏振现象优势被用来 employing polarization filters to separate the direct and the global components

2.1.2 Light Interaction at Surfaces

• 这里总共可能有12维(因为$x_i,x_r$在曲面上,所以他们分别是二维的)
• 一般来说可以假设:“ Typical assumptions made in the great majority of publications are that the light transport at the surface happens in an infinitesimal period (i.e. $t_i = t_r$), that there is no time dependency of the reflectance behavior (i.e.$t_0 = t_i = t_r$), that the wavelength remains unchanged (i.e. $λ_i = λ_r$) and that the incoming flux is completely reflected at the surface (i.e. $x_i = x_r$).“
• 考虑时需要分scale进行考虑”take into account that material appearance is a scale-dependent phenomenon”,分别是microscopic scale,a slightly coarser scale(相对于前面那个),mesoscopic scale,macroscopic scale.
• microscopic scale:domain of quantum optics,这些结构不会被眼睛观测,但是对材料外观有极大的贡献,determine the appearance of all materials.
• a slightly coarser scale:domain of wave optics(考虑光线和与波长近似的小结构interaction的现象,比如diffraction与polarization)
• mesoscopic scale:光线与表面上的fine details的相互作用,比如scratches,engraving,weave-patterns od textiles和embossing of leathers.这些表面的结构可能会导致 self-shadowing, self-occlusions,or interreflections
• macroscopic scale:3D geometry对于一些材质仍然影响其apperance.woven cloth, brushed metal等可能在照片中会distorted,因为其对物体geometry的依赖.
• 但是上述几种scale只对近距离观察起作用.对于增加的距离,在fine surface detail上的the effect of light exchange会变得不可见.所以他们会被当成irregularities in a different kind of microscopic scale.例子:”shininess of specular objects or translucency might also depend on the distance between object and observer.”
• 当考虑一个a highly specular surface with a rough surface profile from a close range时,因为人眼的分辨率足够将其分为不同的patch并得到其法向,所以这个材料显得specular.但是当距离增加时,只能够观察到superposition of these pathces.这样便会导致从specular到diffuse的过渡.然而,当考虑一个highly specular surfaces时,这个表面随着距离的增加仍然是specular的.
• 这些现象表明:The definitinon of scale is of dynamic nature.

  2.2 A Taxonomy of Surface Classes

  前面已经提到,没有一种acquisition technique可以处理任意的材料,需要根据the acquisition principles applicable to the individual material groups 来分类材料.这些分组是与complexity of their visual appearance有关的.
  这里只考虑固体的分类,不考虑volumetric phenomena(fog or fire).这里的分类是根据下面的survey进行的:
  IHRKE, I., KUTULAKOS, K. N., LENSCH, H. P. A., MAGNOR, M., AND HEIDRICH, W. 2010. Transparent and specular object reconstruction. Computer Graphics Forum 29, 8, 2400–2426.
  

  2.2.1 Rough Surfaces with Diffuse or Near Diffuse Reflectance

  需要注意的是,入射光和反射光这里有两个和法向相关的角,在diffuse的情形中,是如入射角有关,与反射角无关的.
  $$
  L_{diffuse}=L_ik_{diffuse}cos\theta,\
  cos θ = n d_i
  $$
  这里k_{diffuse}是衡量表面diffusivity的常量.

  2.2.2 Glossy Surfaces with Mixed Diffuse and Specular Reflectance

  $$
  L_{glossy}=L_{diffuse}+L_{specular}
  $$
  more light is reflected into preferred directions and the observed material appearance is view-dependent

  2.2.3 Smooth Surfaces with Ideal or Near Ideal Specular Reflectance

  (almost) completely reflected into the direction:
  $$
  d_{o,ideal~reflection}=2n(nd_i)-d_i
  $$
  由于这种情况下,镜面反射的特征没有自己的characteristic appearance 而是反映的是surrounding environment in a view-dependent manner.所以the geometry reconstruction for objects with such a reflectance behavior is rather challenging

  2.2.4 Surfaces Where Light is Scattered Multiple Times Underneath the Surface

  这种现象在 translucent object中较多,由于光纤transport within the object导致的.一些光线进入物质进行scatter,然后意味着一部分的光线离开物质的位置和入射点不同.这将会导致a blurring of the observed pattern.这种现象 make a triangulation-based reconstruction from the decoded correspondences rather unreliable.

  2.2.5 Smooth Surfaces with Ideal or Near Ideal Specular Refraction

  Snell’s law:$\eta_{1} \sin \theta_{1}=\eta_{2} \sin \theta_{2}$
  这里存在的问题是,objects might also exhibit inhomogeneous reflectance characteristics(树立在水中的木杆在交界处有一个kink),by a spatially varying refractive index or by inclusions of Lambertian or opaque material components as given in many minerals.
  

2.3 常用的反射模型

摘要:论文中经常出现的SVBRDF,还有6D,4D模型怎么来的

• 基于前面的分类,可以知道:diffuse and specular components以及可能发生的材料下的scattering以及refraction需要考虑进反射模型中,同时各种reflectance acquisition需要在相应的反射模型假设下进行设计.各种不同的模型针对于部分可能的材料,而且希望在尽可能少的参数下对材料在acceptable的时间中进行一个可信描绘.
• 对于diffuse的物质,更多的考虑different material characteristics而不是modeling surface reflectance of mirrors,类似的,考虑同时有diffuse and specular或者translucent 和transparent的物质,considering the respectively relevant characteristics of the individual materials更加重要.即reflectance acquisition strongly depends on the representation used to model the reflectance of a particular material
• 很通用的表示:$\rho\left(\mathbf{x}{i}, \theta{i}, \varphi_{i}, t_{i}, \lambda_{i}, \mathbf{x}{r}, \theta{r}, \varphi_{r}, t_{r}, \lambda_{r}\right)$,这里$t_i=t_r,\lambda_i=\lambda_r$一般成立.
  

详细称呼:

• BSSRDF:bidirectional scattering-surface reflectance distribution function
• SVBRDF:spatially-varying bidirectional reflectance distribution function
• BTF:bidirectional texture function
• BSSDF:bidirectional subsurface scattering distribution function
• BRDF: the bidirectional reflectance distribution function
• SLF:surface light field
• SRF:surface reflectance field

一行一行的来理解:

• 8D模型要求的是反射场定义在一个凸的表面删,然后viewpoint以及light都来自于bounding volume.这样可以通过base incideng light field的线性表达式来定义新的光照条件,
• 8D到SVBRDF是假设没有内部的scatter,从而入射和反射点是一致的.而到BTF是假设各处入射光照都是一致的(light source infinitely far away),即$\rho_{L F, i}\left(\mathbf{x}{i}, \theta{i}, \varphi_{i}\right)=\rho_{L F, i}\left(\theta_{i}, \varphi_{i}\right) .$,所以这里SVBRDF和BTF看起来形式差不多,但是意义上有差别.n comparison to SVBRDFs, BTFs allow to capture local subsurface scattering characteristics as well as mesoscopic effects such as interreflections, self-masking or self-occlusions.当考虑各向同性的(homogeneous reflectance)是,8D模型由编程了BSSDF模型$\rho_{\mathrm{BSSDF}}\left(\theta_{i}, \varphi_{i}, \mathbf{x}{r}-\mathbf{x}{i}, \theta_{r}, \varphi_{r}\right)$
• 在BSSRDF的基础上,再要求non-subsurface scattering reflectanc,得到BRDF.在BTF的基础上,固定lighting的方向得到SLF,固定view的方向得到SRF.
• 对于2D模型来说,它是建立在diffuse reflectance的基础上的,得到texture maps of bump maps.

上面的层次图来源于
M ¨ULLER, G., MESETH, J., SATTLER, M., SARLETTE, R., AND KLEIN, R. 2004. Acquisition, synthesis and rendering of bidirectional
texture functions. In Eurographics 2004 State of the Art Reports, 69–94.
后面在section 5还会继续讨论 individual reflectance models and their acquisition的细节.
关于上面每个模型详细的论文见:p13

3.Calibration

• 为什么要讲Calibration?:If several measurements of the geometry in a scene or the corresponding reflectance behavior have been performed,these measurements need to be brought into some kind of reference system.
• 在这里仅仅为了establish an awareness regarding why individual calibration techniques are needed and some rather general ideas on how a calibration of individual setup components can be performed,对于细节来说,由于参考具体文献.
• 由于individual components相对于物体表面的位置/方向以及device-specific 的特征对于geometry和reflectance的an accurate acquisition是很重要的.从而geometric calibration是很重要的,它可以得到 the involved illuminants, imaging sensors and the object surface during the acquisition之间的联系.这里还包括在投影中的相机的focal length以及principle point.radiometric calibration也是很重要的,它考虑了sensor以及illuminants的radiometric性质.这在将测得的反射性质用于 some kind of reference system for radiance中很重要.

  3.1 Geometric Calibration

• The geometric calibration aims at the specification of relative positions and orientations of the individual components involved in the setup by either using certain known calibration objects or possibly also objects with unknown shape
• 对于这些物体的不同的观测需要进行register,同时需要考虑光源的位置.同时通常没有足够的sensor来提供足够稠密的view-light configurations.所以需要通过turntable来模拟很多sensor的效果.

  3.1.1 Camera Calibration

  详细介绍见:HARTLEY, R. I., AND ZISSERMAN, A. 2004. Multiple view geometry in computer vision (2nd edition). Cambridge University Press,
  Cambridge, UK
  由于这一部分较为基础,不做记录,对应页号为P15

3.1.2 Light Source Calibration

• 有些时候,虽然light source是放在一个机器臂上的,他们的位置可以直接得到,但是”positioning systems often do not offer the possibility for a precise specification of a light source position and orientation”,所以Calibration是必须的.
• 下面的文章中用到的方法是基于一致位置以及半径的mirroring calibration sphere上的highlight observations.
  (1) CHEN, T., GOESELE, M., AND SEIDEL, H.-P. 2006. Mesostructure from specularity. In Proceedings of the IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR), vol. 2, 1825–1832.
  (2)WEINMANN, M., RUITERS, R., OSEP, A., SCHWARTZ, C., AND KLEIN, R. 2012. Fusing structured light consistency and Helmholtz
  normals for 3d reconstruction. In Proceedings of the British Machine Vision Conference (BMVC), 1–12.
  (3)SCHWARTZ, C., SARLETTE, R., WEINMANN, M., AND KLEIN, R. 2013. DOME II: A parallelized BTF acquisition system. In Proceedings
  of the Eurographics Workshop on Material Appearance Modeling: Issues and Acquisition, 25–31.
• 其关键想法是追踪从calibrated camera的perspective center出发的ray,穿过图片上highlight的位置,到达scene中,这里他们被一致的shpere geometry 反射.”Reflected rays obtained from highlights observed from multiple spheres allow the reconstruction of the light source position which is given by their intersection”
  
• 而在这篇论文中:SCHWARTZ, C., SARLETTE, R., WEINMANN, M., RUMP, M., AND KLEIN, R. 2014. Design and implementation of practical bidirectional
  texture function measurement devices focusing on the developments at the University of Bonn. Sensors 14, 5, 7753–7819.
  使用来一种同时对光源位置以及calibration sphere的位置的非线性优化算法,来减少观察到的反射中的 re-projection error.
• 在ACKERMANN, J., FUHRMANN, S., AND GOESELE, M. 2013. Geometric point light source calibration. In Proceedings of Vision, Modeling
  and Visualization (VMV), 161–168.的论文中,与上面的方法相似,只是在优化的过程中没有refine calibration sphere.
• 而在这两篇论文中,M ¨ULLER, G., MESETH, J., SATTLER, M., SARLETTE, R., AND KLEIN, R. 2004. Acquisition, synthesis and rendering of bidirectional
  texture functions. In Eurographics 2004 State of the Art Reports, 69–94.
  SCHWARTZ, C., WEINMANN, M., RUITERS, R., AND KLEIN, R. 2011. Integrated high-quality acquisition of geometry and appearance
  for cultural heritage. In Proceedings of the International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage
  (VAST), 25–32.
  如果camera 的闪光也作为光源,那么光源的位置可以基于闪光相对于相机透视中心的偏移量进行估计

  3.1.3 Turntable Calibration

• Turntable在大多数的setup中用来模拟更远处light source或者image sensor的存在.
• 相当精准的对于turntable center和turntable axis的估计可以通过rotating calibration targets达到.这些target被相机观察到(相机也可以通过这些target进行标定),然后对应的观察可以用来得到turntable的参数.一些高质量的turntable可以直接提供足够精确度的rotation angle.

  3.2 Radiometric Calibration

• 除了几何calibration之外,在使用像素值进行geometry以及reflectance的重建时,光源发出的光线的性质以及传感器将接收到的光转换为特定的像素值的特征也需要进行考虑.
• …

  4. Advances in Geometry Acquisition

  尽管有很多几何重建的方法,但是经证明,许多传统的技巧不允许重建objects with arbitrary surface reflectance的geometry,而只能重建objects with a rather simple diffuse surface reflectance behavior.这也就导致来大量的diverse material-specific acquisition techniques,每个技巧只能针对于特定范围的材料.很多方法的材料分类根据前面Section 2的分类方法.
  在介绍trend之前,首先我们在4.1节介绍an overview of standard principles (Section 4.1) that are used in many of the geometry acquisition techniques.

  4.1 Basic Geometry Acquisition Principles

  The acquisition of the 3D surface geometry of objects can be approached based on different principles. The key differences of the individual methods include whether methods are active or passive, contact-based or non-contact-based and optical or non-optical
  
  

Active methods就是可以向物体打a certain type of energy,并且可以被the components of the respective acquisition setup or in terms of contact-based feelers that produce a characteristic signal when touching the surface观测到,而passive的方法是只有fixed and distant illumination,并且关注对这种照明条件下optical object appearance的特征.

4.1.1 Principles for Passive Geometry Acquisition

observation and analysis of the optical appearance of objects under fixed illumination without systematic manipulations.

4.1.1.1 Shape-from-Shading Techniques

• image上面得到的一个view的2D的信息是不足以重建3D的物体形状的,这是由于该问题不适定性的本质导致的(ill-posed),为了能够重建表面的geometry,需要一些prior information或者assumption来得到额外的约束
  (经常看到提远近场光的:https://baike.baidu.com/item/%E8%BF%91%E5%9C%BA%E5%85%89%E5%AD%A6/11045876?fr=aladdin)
• 传统的shape-from-shading techniques基于分析单一的view观察到的,在可控的环境下可控的点光源光照下,观察到的intensity information,而illumination characteristics,比如波长以及它的origin(一般是远场光,the light source position is typically significantly farer away than the extents of the considered object surfaces)和方向由prior calibration procedure已知.但是现在的问题仍然是ill-posed.
• 而对于diffuse surface的Lambert’s law,可以添加更多的约束:
  $$
  I(x,y=\alpha(x,y) r l \cdot n
  $$
  这里$I(x,y)$是2D图像上观察的的intensity,l是已知的incoming illumination,$\alpha(x,y)$是一致的表面的albedo.r是known intensity r of the illumination
• 额外的广泛使用的假设包括 an orthographic image projection and the absence of shadows and interreflections(正射,没有阴影以及多次反射).然后现在的方程组对于一个像素只有一个约束,所以仍然是under-constrained的,这里对于每一个表面上的反向有两个未知的参数$p=\frac{dz}{dx}$以及$p=\frac{dz}{dy}$.因为这个原因,需要更近一步的假设,这个假设,假设$\alpha(x,y)$为常数,并且对法向$n$中的z分量进行normalization得到$n=[-p,-q,1]^T$,并且限制梯度(比如enforcing smoothness),或者使用the self-shadow boundary进行限制.
• 针对于shape-from-shading更加详细的讨论见DUROU, J.-D., FALCONE, M., AND SAGONA, M. 2008. Numerical methods for shape-from-shading: A new survey with benchmarks.
  Computer Vision and Image Understanding 109, 1, 22–43.
• 在surface normal的基础上,表面的geometry可以通过法向场积分的技巧得到(normal field integration techniques.).大多数的shape-from-shading techniques都聚焦于单张图片的几何重建,因此only a 2.5D height map can be derived

4.1.1.2 Shape-from-Texture Techniques

• 一些物质表现出了相当regular的表面样式可能可以用来surface reconstruction,比如fabrics(regular structure是由编织的样式决定的).Shape-from-Texture Techniques关注假定有regular statistical ot geometric surface pattern的物品的表面几何重建