<p align="center"> <h1 align="center">Relighting Neural Radiance Fields with Shadow and Highlight Hints</h1> <p align="center"> <a href="https://www.chong-zeng.com/"><strong>Chong Zeng</strong></a> · <a href="https://www.microsoft.com/en-us/research/people/guoch/"><strong>Guojun Chen</strong></a> · <a href="https://yuedong.shading.me/"><strong>Yue Dong</strong></a> · <a href="https://www.cs.wm.edu/~ppeers/"><strong>Pieter Peers</strong></a> · <a href="https://svbrdf.github.io/"><strong>Hongzhi Wu</strong></a> · <a href="https://www.microsoft.com/en-us/research/people/xtong/"><strong>Xin Tong</strong></a> </p> <h2 align="center">SIGGRAPH 2023 Conference Proceedings</h2> <div align="center"> <img src="nrhints-teaser.png"> </div> <p align="center"> <br> <a href="https://nrhints.github.io/"><strong>Project Page</strong></a> | <a href="https://nrhints.github.io/pdfs/nrhints-sig23.pdf"><strong>Paper</strong></a> | <a href="https://arxiv.org/abs/2308.13404"><strong>arXiv</strong></a> | <a href="#data-and-models"><strong>Data</strong></a> </p> </p>

Setup

Environment

The code is developed and tested on Linux servers with NVIDIA GPU(s). We support Python 3.8+ and PyTorch 1.11+. After getting a required Python environment, you can setup the rest of requirements by running:

bash
git clone https://github.com/iamNCJ/NRHints.git
cd NRHints
pip install -r requirements.txt

Data

Our data is compatible with NeRF Blender Dataset, except that we have extra fields in each frame for point light position.

You can download our data here.

Usage

Configuration System

We use tyro for configuration management. Description to all configurations can be found by running python main.py -h.

Training

bash
python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX

Refer to train_synthetic.sh and train_real.sh for training on synthetic and real data, respectively.

Note:

Our code automatically detects the number of GPUs and uses all of them for training. If you want to use a subset of GPUs, you can set the CUDA_VISIBLE_DEVICES environment variable.

For training on real captured scenes, we recommend turning on camera optimization by using config:nr-hints-cam-opt, which can significantly reduce the blurry effects. Since this is an improvement after the paper submission, details are described in the author's version.

Testing

bash
python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX --config.evaluation-only True

Refer to eval_synthetic.sh and eval_real.sh for testing on synthetic and real data, respectively.

Our pretrained models can be downloaded here.

Data and Models

Real Captured Scenes

Object	Data	Pre-trained model
Cat	Link	Link
Cluttered Scene	Link	Link
Pixiu Statuette	Link	Link
Ornamental Fish	Link	Link
Cat on Decor	Link	Link
Cup and Fabric	Link	Link
Pikachu Statuette	Link	Link

Synthetic Rendered Scenes

Note:
Our synthetic data rendering scripts are released at here.

Object	Data	Pre-trained model
Diffuse	Link	Link
Metallic	Link	Link
Glossy-Metal	Link	Link
Rough-Metal	Link	Link
Anisotropic-Metal	Link	Link
Plastic	Link	Link
Glossy-Plastic	Link	Link
Rough-Plastic	Link	Link
Short-Fur	Link	Link
Long-Fur	Link	Link
Translucent	Link	Link
Fur-Ball	Link	Link
Basket	Link	Link
Layered Woven Ball	Link	Link
Drums	Link	Link
Hotdog	Link	Link
Lego	Link	Link

You can use the script download_data.sh to download all data.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings {zeng2023nrhints,
    title      = {Relighting Neural Radiance Fields with Shadow and Highlight Hints},
    author     = {Chong Zeng and Guojun Chen and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
    booktitle  = {ACM SIGGRAPH 2023 Conference Proceedings},
    year       = {2023}
}