Learning Geometry

3.1 Geometric Datasets

Dataset	Focus
WordNet	Semantic categories
ImageNet	Images organized by semantic category
ShapeNet	Richly annotated 3D shapes

Other 3D Object and Scene Datasets:

Driving Datasets:

4D Datasets:

Virtual Environments:

AI + Geometry: Let \(S\) denote a shape and \(c\) a label or condition.

Shape analysis — \(P(c \mid S)\):
- Learning shape descriptors
- Shape classification, segmentation, view estimation, etc.
Shape synthesis — \(P(S)\) or \(P(S \mid c)\):
- Learning (conditional) shape priors
- Shape generation, completion, and reconstruction

3.2 Data-Driven Shape Analysis

Multi-View Representations:

Projection is required.
\(\mathrm{CNN}_1\): a ConvNet extracting image features
\(\mathrm{CNN}_2\): a second ConvNet producing shape descriptors

3D CNN on Volumetric Data:

Voxelization: Information loss.
Learn to project: Exploit the sparsity of 3D shapes and store sparse surface signals.
Sparse convolution: Output feature \(\mathrel{+}=\) input feature \(\times\) kernel.

Octree: Recursively partition space. Each internal node has exactly eight children; hash tables store the octree structure.

PointNet: The first learning tool for point clouds.

End-to-end learning for irregular point data
Unified framework for various tasks
Unordered input: construct symmetric functions using neural networks

Limitations of PointNet:

No local context for each point
Global features depend on absolute coordinates, making it hard to generalize to unseen scene configurations

PointNet++: Recursively apply PointNet to local regions.

Hierarchical feature learning
Local translation invariance

Monte Carlo Convolution: A method for performing convolution on point clouds using Monte Carlo sampling.

3.3 Data-Driven Shape Synthesis

Single-View Conditioned 3D Modeling: A canonical problem.

Model-Based Fitting (the first vision thesis):

Fit predefined 3D models to image features.
Limited by the expressiveness of simple geometric models.

Data-Driven Modeling (a modern view):

Learn image-to-3D mappings from large datasets using neural networks.
Reduce explicit geometric reasoning, though it remains helpful.

2D to 2.5D — Depth Prediction:

Input image \(\to\) convolutional network \(\to\) predicted depth.

Compare predicted depth \(y\) with ground-truth depth \(y^*\):

Squared error:

\[ L(y, y^*) = \sum_i \|y_i - y_i^*\|^2. \]

Log-space loss: With \(\alpha = \alpha(y, y^*)\),

\[ L(y, y^*) = \sum_i \left\|\log y_i - \log y_i^* + \alpha\right\|^2, \]

where

\[ \alpha = \frac{1}{n} \sum_i \left(\log y_i^* - \log y_i\right). \]

Image-Based 3D Object Modeling: Object-centric, supervised 3D learning.

Input image \(\to\) CNN \(\to\) predicted 3D; compare with ground-truth 3D.

Learning Workflow:

Decide on the model and objectives.
Collect training data: lots of [image, 3D] pairs — not easy to obtain.
Learn a predictor.
Evaluate the predictor.
Use the predictor.

Learning to Predict Volumetric 3D:

2D encoder: Image \(\to\) latent code.
3D decoder: Latent code \(\to\) \(D^3\) volume.
Loss: Compare the predicted and ground-truth 3D volumes.
Training data: Renderings of CAD models on random backgrounds, helping transfer to real images.

Implicit Volumetric Representation:

Instead of explicitly predicting a 3D voxel grid, learn a continuous occupancy function.
Given a latent code \(c\) and an arbitrary 3D query point \(p = (x,y,z)\), predict the probability that the point lies inside the object.

Learning to Predict Point Clouds:

Latent vector \(\to\) fixed-size point cloud. Compare two point sets with the Chamfer distance:

\[ \begin{aligned} d_{\mathrm{CD}}(S_1, S_2) &= \sum_{x \in S_1} \min_{y \in S_2} \|x-y\|_2^2 \\ &\quad + \sum_{y \in S_2} \min_{x \in S_1} \|x-y\|_2^2. \end{aligned} \]

Learning to Predict Parametric Point Clouds:

Latent vector + planar point \(\to\) generated 3D point.

Learning to Predict and Deform Meshes:

Advantages of using meshes:

Smoothness objectives respect connectivity: Laplacian smoothness and normal consistency.
Can sample a varying number of points.