Multi-View Stereo based on Deep Learning

Multi-View Stereo (MVS) has become a key player in fields like 3D mapping, video games, and autonomous driving, allowing us to create detailed 3D reconstructions from multiple images. Recent advancements, especially those leveraging deep learning, have made MVS techniques more efficient and accurate. In this blog post, we'll dive into the latest methods shaping the future of MVS, with a particular focus on MVSNet and its successors. I'll break down how MVSNet works, covering essential steps like feature extraction, cost volume regularization, and reconstruction. At the end, there is a comparison how various models perform, using popular benchmarks like the DTU and Tanks and Temples datasets.

1. Introduction

In this blog, we examine the principles of multi-view stereo, a technique central to the generation of 3D models from multiple 2D images—a complex problem that has been the focus of research for many years.¹ Tools like COLMAP ² are one way to achieve precise three-dimensional representations from diverse image datasets. Traditional 3D reconstruction algorithms are relatively straightforward to implement, however, they are not without limitations, particularly when dealing with non-ideal conditions such as non-Lambertian surfaces, lacking abrupt color boundaries and uncalibrated camera sensors. However, the rise of Deep Learning (DL) and the exponential growth in computing power have dramatically transformed the landscape of scene reconstruction. These technological advancements have enabled the handling of vast datasets containing numerous scenes, facilitating easier comparison and evaluation of different reconstruction methodologies. This shift has been instrumental in circumventing the earlier constraints associated with non-Lambertian surfaces, opening up new possibilities for capturing and representing scenes under diverse and non-ideal conditions. The applications resulting from these advancements are diverse. They range from enriching immersive experiences in virtual reality environments to supporting the reconstruction of historical sites in archaeology and facilitating more accurate architectural designs.³

2. MVSNet

In the following, MVSNet by Yao et al. is introduced as the baseline model for this blog post.⁴ MVSNet is considered a successful early adopter of DL and is frequently cited in subsequent research. The architectural framework of MVSNet is comprehensively shown in figure 1.

First, MVSNet initiates its depth inference process by extracting deep visual image features of $N$ images through a simple multi-layer 2D CNN. At the third and sixth layers, the strides are set to two, effectively reducing the height and width of the feature space by a total factor of 4, respectively. These extracted features are then organized into a set, denoted by $\{ \mathbf{F}_{i}\}_{i=1}^{N}$ for each image $i$. The feature extraction network is shared for all input images providing an efficient training mechanism for the model. Subsequently, the extracted features must be transformed into a cost volume. The cost volume is a data structure that incorporates geometric constraints and prior knowledge. The process is divided into three steps: (1) the computation of differentiable homography, (2) the formulation of a cost metric, and (3) the application of cost volume regularization.

To obtain such a cost volume, the authors relate the feature maps according to their homography. In particular, the feature maps undergo a transformation into frontoparallel planes with respect to the frustum of the reference view. The homography matrix $\mathbf{H}_i(d)$ considers for that camera intrinsics, rotations, and translations, denoted by $\{\mathbf{K}_i, \mathbf{R}_i, \mathbf{t}_i\}_{i=1}^N$ respectively, where $i$ denotes the image (see eq. 1). The planar transformation $\mathbf{{x}'} \sim \mathbf{H}_i(d) \cdot \mathbf{x}$ relates then the coordinates of points at depth $d$ in one feature map to the coordinates of the reference feature map. Ultimately, the feature maps are converted into feature volumes of shape $\frac{H}{4} \times \frac{H}{4} \times D \times F$, where $D$ denots the depth, and $F$ the number of channels of the feature map.

The warping operation is formulated in a differentiable way, allowing end-to-end trainability.

Following step (1), in step (2) the feature volumes are aggregated along the feature channel dimension to obtain a single cost volume. The feature volumes could be aggregated using the mean (as shown in ⁵). Yao et al. instead choose to use the variance as it is able to capture feature differences (see eq. 2).

In step (3), the raw cost volume may contain some noise which leads to a performance drop. To address this issue the cost volume is being smoothed (regularized). To achieve this the authors apply a multi-scale 3D CNN, designed in a U-Net-like structure. The encoder-decoder construction allows to perceive and reinforce neighboring information. From that the normalized probability for each depth hypotheses is retrieved through the softmax function resulting in a probability volume. This probability volume has two purposes. First, it can be used for computing the depth of a single image as well as offering a confidence measure of the inference.

To derive an initial depth map $D$ there are two proposed operations. One could either apply the argmax function to the probability volume $P$, or use the soft argmin (as suggested in ⁶). Since argmax is indifferentiable, end-to-end training cannot be employed. Hence, MVSNet uses the soft argmin. The soft argmin operation is the expected value along in depth direction and is being computed as follows:

$P(d)$ returns the probability for all pixels being at depth $d$.

Although the depth map can be considered final, it still may lack quality in different areas, i.e. oversmoothing at reconstruction boundaries. To overcome this issue, an additional network, a so-called depth residual learning network, is being deployed. The network takes the concatenation of the initial depth map and the resized reference image as an input. The objective of this model is to learn the depth residuals. The depth residuals are then added to the original depth map.

As seen in figure 1 two losses are being computed (see eq. 4). Yao et al. consider the mean absolute difference between the actual depth map $d(p)$ and the initial depth map $\widehat{d}_i(p)$ and between the actual depth map and the refined depth map $\widehat{d}_r(p)$. Pixels without any associated ground truth are excluded. Hence, only $\textbf{p}_{valid}$ is being considered.

3. Modular optimization based on MVSNet

MVSNet has shown state-of-the-art performance on benchmarks due to many factors and effective utilization of DL and CNN advancements. The model's end-to-end training capability with minimal adjustments garnered community interest. While MVSNet demonstrated a novel architecture, it has not fully realized its maximum potential (see table 1). The modular design offers the possibility to replace certain parts of the architecture. Hence, numerous networks were published afterwards that excel in different categories. To systematically analyze these advancements, Zhu et al. propose to categorize MVS-algorithms into feature extraction, cost volume construction, and cost volume regularization.⁷ Seitz et al. break the properties of MVS-algorithms into scene representation, photo-consistency measure, visibility model, shape prior, reconstruction algorithm, and initialization requirements down.⁸ Especially the reconstruction algorithm is of interest since most models only infer a depth map, yet some benchmarks require a point-cloud instead. Therefore, reconstruction algorithms that perform such transformations, should be highlighted. In addition, the loss function should also be of interest. To sum up, the next section focuses on feature extraction, cost volume regularization, loss, and reconstruction.

3.1. Feature extraction network

MVSNet employs a simple 2D CNN (in a Feature Pyramid Network (FPN) style) for feature extraction. However, more complex architectures like U-Net or ResNet are better suited for feature extraction (as exemplified in ^{9 10}).

GeoMVSNet integrates both approaches by iterating through images in multiple stages, progressing from low-resolution to full resolution.¹⁰ At each stage, geometric information from prior stages are incorporated at two distinct steps.

In the context of feature extraction, only the first step, namely leveraging prior geometric information in the feature extraction step, is of interest and will be highlighted. At the earliest stage, no prior information can be utilized. Therefore, the authors initially implement a basic FPN to extract features from the most downscaled version of the input images. These extracted features undergo a process resulting in a low-resolution depth map. This coarse depth map is then integrated into the feature extraction step for the next finer stage. Technically, this is done by processing the depth map as a 2D discrete signal, allowing to transform it into the frequency domain via the Discrete Fourier Transformation. A low-pass filter removes then high-frequencies before the frequency domain is inverted back. Consequently, the depth map considers a full-scene geometry perception, filtering out high-frequency signals and drastic misinformation. The "improved depth map" is then employed in the subsequent feature extraction. A branching mechanism combines the newly extracted features at higher resolution with the cleaned coarse depth from the previous stage, serving as a geometric prior. This iterative process enhances the model's ability to generate accurate depth maps throughout all later stages.

With the growing success of transformers in NLP, emerging networks for MVS, more novel models incorporate transformers into their feature extraction pipeline (such as those in ^{11 12 13 14}). Transformers excel in capturing long-range dependencies and global contextual information.¹⁵ In contrast to CNNs, which process images in a grid-like fashion, Vision Transformers (ViT) treat the entire image as a sequence of tokens, allowing them to learn relationships between distant pixels.¹⁶ This global perspective proves beneficial in MVS scenarios where understanding spatial context across multiple views is crucial. A notable example illustrating this concept is MVSTR.¹⁴ Here the authors start by extracting features using a simple CNN network. These extracted features, along with their positional embeddings, are then passed into a global-context transformer as input, capturing long-range dependencies within each view. The output is subsequently fed into another transformer, the 3D-geometry transformer, which captures dependencies across multiple views. These transformers alternate for $Z$ times to enhance the understanding of global and local information within and across multiple views. While ViTs are more scalable than CNNs, training them is challenging due to the substantial amount of data they require. Networks like MVSFormer highlight the importance of leveraging pre-trained ViTs.¹¹

3.2. Cost volume regularization

A main distinction between MVS models can be found in the methodology that employs cost volume regularization. The strategies for designing cost volume regularization can be divided into three distinct approaches, namely, 3D CNN ⁴, Recurrent Neural Network (RNN) ^{17 18}, and coarse-to-fine-approaches ^{19 20 21 22 23 24}. Figure 2 provides visual representations.

3.2.1 3D CNN

MVSNet is an early adopter of a U-Net-like 3D CNN for cost volume regularization.⁴ The advantage of using 3D CNNs lies in its capability to combine both local and global features across all dimensions in $D$.

3.2.2 RNN

However, 3D CNN are computationally expensive which limits its ability to use finer depths. For addressing the challenge of excessive memory usage, some models opt for a different approach by substituting 3D CNNs with sequential 2D CNNs along the depth dimension $D$. RNN-approaches consider previous depth hypotheses by subsequently passing them as context information for later depths. This modification ensures that only one slice of the cost volume is processed in the GPU memory at a time reducing the memory issue, but also providing finer flexibility in scaling the dimension $D$. Earlier techniques incorporate spatial information within a specific distance.²⁵ R-MVSNet proposes cost volume regularization using Gated Recurrent Unit (GRU).¹⁸ GRUs are a RNN-architecture initially proposed for learning sequential text. It gathers spatial and uni-directional context information in the depth direction. To enhance regularization, R-MVSNet stacks three GRU layers. The cost volume $\left \{ C(i) \right \}_{i=1}^D$ is seen as $D$ cost maps in depth direction. After applying the recurrent regularization the result is $\left \{ C_r(i) \right \}_{i=1}^D$, where $C_r(t)$ depends on the current and previous cost maps $\left \{ C(i) \right \}_{i=1}^{t}$. Before applying the GRUs, a convolutional layer is applied to retain context information. D$^2$HC-RMVSNet represents an additional approach.¹⁷ R-MVSNet, in comparison, experiences some loss regarding the context information. To tackle this issue D$^2$HC-RMVSNet introduces a 2D U-Net-like architecture, where each layer is a Long Short-Term Memory (LSTM) cell. This architecture enables the aggregation of multi-scale context information, reducing memory consumption down to only 19.4% of R-MVSNet, while efficiently processing dense original-size cost volumes.

3.2.3 Coarse-to-fine

One limitation of these RNN-based methods, however, is that they are less efficient with respect to parallelizability than traditional CNNs which can negatively impact the run time. Hence, models such as MVSNet, or R-MVSNet downscale the images to obtain a cost volume with lower resolution. From there the models upsample or implement post-refinement to generate a final result in high resolution. Coarse-to-fine approaches solve this by progressively refining the reconstruction by initially generating a rough approximation and iteratively improving the details through successive stages of increasing refinement.^{19 20 21 22 23 24} CasMVSNet proposes to use a cascade cost volume.²¹ The cost volume at the beginning is based on large scale semantic features selected from a sparse depth hypotheses. As a result, the resolution of the cost volume is lower. Based on this coarse cost volume, an initial depth map is determined. The resolution is then gradually refined by combining the preceding depth map and finer semantic features used as a basis for subsequent stages. This process can be repeated multiple times. With every iteration the range of hypothesis depths is improved and a new cost volume with finer semantic features is derived.

3.3. Loss

The loss function is a measure of the dissimilarity between the predicted depth map and the ground truth depth or disparity map. With regard to the loss function, there are two different related challenges, namely, regression and classification, as well as unsupervised methods.

3.3.1 Regression vs. Classification

In machine learning, the choice of a loss function plays an important role in determining the final prediction. The optimization problem can be framed in various ways, either as a regression problem (exemplified in ^{21 4}), or as a classification issue (represented in ^{17 18 24}). In classification problems, the argmax function is commonly employed, accompanied by the use of cross entropy as a loss function. However, the inherently less smooth nature of classification can result in unsteadiness in depth values. Conversely, regression tackles the optimization problem differently, utilizing loss functions such as $L1$, as seen in the equation 4. Since a model, that uses regression, needs to learn a complex combination of weights under indirect constraints performed on the weighted depth but not on the weight combination, it tends to overfit and increase the difficulty of the model convergence.²⁶ Each approach reflects a distinct methodology in shaping the model's predictions.

3.3.2 Unsupervised Learning

Up to this point, the focus of this post has been on models trained through supervised learning. Nevertheless, numerous datasets are synthetically generated, presenting challenges in their creation. Consequently, unsupervised learning offers a valuable alternative, allowing models (such as ^{27 28 29}) to be trained in theory on simpler datasets without the availability of provided ground truths.

This blog post will not dive into unsupervised learning in depth, but I can highly recommend the following articles:

• Medium: Unsupervised Multi-View Stereo — An Emerging Trend
• Medium: 3D Reconstruction News — AAAI 2021

4. Results

In the following, I will first introduce popular datasets/benchmarks and afterwards I will set some models side by side.

4.1. Datasets/Benchmarks and Metrics

There are several benchmarks one could use (e.g. ^{30 31 32 33 34 35 36}). In the following, however, two of the most common datasets will be highlighted. The Tanks and Temples dataset comprises a collection of high-resolution images and corresponding depth maps captured from various indoor and outdoor environments.³² The dataset includes scenes with complex geometry ranging from an intermediate level to advanced difficulty, making it a valuable resource for evaluating stereo vision algorithms and depth estimation techniques. In addition, this dataset offers an online leaderboard. To compare the models with each other, the metric rank is introduced, which indicates the average rank across all scenes. In contrast, DTU is a multi-view stereo dataset containing 128 scans with 49 views under different lighting conditions, which doesn't offer any leaderboards.³⁰

Similar to Tanks and Temples, this dataset provides the ground truth in a point-cloud format.

As many models are only capable of generating depth maps, the provided data must be converted to depth maps to even allow training. This is done by first converting the data into mesh models and then into depth maps. Usually, the screened Poisson surface reconstruction algorithm is used for this transformation.³⁷ The fundamental objective of a model is to improve not only effectiveness but also efficiency. To measure the current performance level, metrics capture accuracy, completeness, GPU memory consumption, and run time. Accuracy, or precision, quantifies either the percentage of predicted points that align with the ground truth point-cloud or the mean/median distance, which applies to this report. Recall/Completeness measures either the proportion of ground truth points that are captured by the predicted point-cloud or the mean error distance between the closest points in the reconstruction and the ground truth. As before, the latter applies to this report. However, relying solely on either metric may not provide a comprehensive comparison due to different assumptions made by the methods. While stronger assumptions often lead to higher accuracy but lower completeness, both measures are essential for a fair comparison. The F-score, as a harmonic mean of precision and recall, fixes this.

4.2. Model Comparison

One notable limitation in our discussion is the absence of detailed information regarding GPU memory consumption and run-time across some of the models.

In evaluating the performance of MVS methods, the comparison with COLMAP², a non-learning baseline, serves as a fundamental baseline. Analyzing the results in table 1, it becomes evident that DL-based MVS methods perform superior compared to traditional methods, particularly in terms of completeness. This applies, however, not for all DL-based models. Earlier models, such as SurfaceNet ³⁸, were not able to outperform COLMAP. This changed with MVSNet. MVSNet, upon its release, demonstrated remarkable performance and briefly secured the top position in the Tanks and Temples benchmark.⁴ Taking a closer look at the difference between methods employing either a RNN or a coarse-to-fine approach to cost volume regularization, one can observe an asymmetrical performance. On the DTU dataset, coarse-to-fine methods (such as ^{19 20 21 22 23 24}) perform better than RNN-based models (such as ^{17 18}), but achieve worse results on the Tanks and Temple benchmark. A plausible explanation for this disparity lies in the nature of the datasets themselves. DTU, with its emphasis on complex details, provides an environment where coarse-to-fine methods excel by recognizing subtle depth differences. Conversely, looking at Tanks and Temples, characterized by a larger scale, RNN methods prove superior, leveraging their ability to use a larger depth scale. In the context of inference time and memory consumption, coarse-to-fine methods perform better than 3D CNN. This progress is achieved without compromising performance metrics, thereby reflecting an improvement in efficiency and effectiveness.

As of November 2023, MVSFormer¹¹ currently holds the top position on the Tanks and Temples leaderboard, closely accompanied by similar approaches like ET-MVSNet³⁹. From this observation, it can be speculated that methods employing transformers, such as ViTs, for feature extraction combined with coarse-to-fine regression exhibit superior performance compared to alternative approaches.

5. Conclusion

In summary, this exploration of MVS based on DL focused on the foundational MVSNet model, dissecting its components and identifying the following key properties: feature extraction, cost volume regularization, loss, and reconstruction. This understanding made it possible to identify and examine various approaches that aimed to enhance specific aspects, such as feature extraction, regularization, and reconstruction. The landscape of MVS has witnessed progress within theses categories. The comparison of these models on public benchmarks provided a quantitative measure of their performance, highlighting the strengths and weaknesses of each approach. In essence, this blog post underscores the remarkable progress made in DL-based MVS.