Commit ca2a8de1 authored by hpeng's avatar hpeng
Browse files

2022-04-12 snapshot

parent 53dcf4ad
build/
data/
out/
*/__pycache__/
*/*/__pycache__/
*/*/*/__pycache__/
*/*/*/*/__pycache__/
*.so
Copyright 2019 Lars Mescheder, Michael Oechsle, Michael Niemeyer, Andreas Geiger, Sebastian Nowozin
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
\ No newline at end of file
# occupancy_networks_ddp
# Occupancy Networks
![Example 1](img/00.gif)
![Example 2](img/01.gif)
![Example 3](img/02.gif)
This repository contains the code to reproduce the results from the paper
[Occupancy Networks - Learning 3D Reconstruction in Function Space](https://avg.is.tuebingen.mpg.de/publications/occupancy-networks).
You can find detailed usage instructions for training your own models and using pretrained models below.
## Getting started
If you find our code or paper useful, please consider citing
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
@inproceedings{Occupancy Networks,
title = {Occupancy Networks: Learning 3D Reconstruction in Function Space},
author = {Mescheder, Lars and Oechsle, Michael and Niemeyer, Michael and Nowozin, Sebastian and Geiger, Andreas},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
year = {2019}
}
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
## Installation
First you have to make sure that you have all dependencies in place.
The simplest way to do so, is to use [anaconda](https://www.anaconda.com/).
You can create an anaconda environment called `mesh_funcspace` using
```
cd existing_repo
git remote add origin https://git.ist.ac.at/hpeng/occupancy_networks_ddp.git
git branch -M main
git push -uf origin main
conda env create -f environment.yaml
conda activate mesh_funcspace
```
## Integrate with your tools
- [ ] [Set up project integrations](https://git.ist.ac.at/hpeng/occupancy_networks_ddp/-/settings/integrations)
Next, compile the extension modules.
You can do this via
```
python setup.py build_ext --inplace
```
## Collaborate with your team
To compile the dmc extension, you have to have a cuda enabled device set up.
If you experience any errors, you can simply comment out the `dmc_*` dependencies in `setup.py`.
You should then also comment out the `dmc` imports in `im2mesh/config.py`.
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## Demo
![Example Input](img/example_input.png)
![Example Output](img/example_output.gif)
## Test and Deploy
You can now test our code on the provided input images in the `demo` folder.
To this end, simply run
```
python generate.py configs/demo.yaml
```
This script should create a folder `demo/generation` where the output meshes are stored.
The script will copy the inputs into the `demo/generation/inputs` folder and creates the meshes in the `demo/generation/meshes` folder.
Moreover, the script creates a `demo/generation/vis` folder where both inputs and outputs are copied together.
Use the built-in continuous integration in GitLab.
## Dataset
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
To evaluate a pretrained model or train a new model from scratch, you have to obtain the dataset.
To this end, there are two options:
***
1. you can download our preprocessed data
2. you can download the ShapeNet dataset and run the preprocessing pipeline yourself
# Editing this README
Take in mind that running the preprocessing pipeline yourself requires a substantial amount time and space on your hard drive.
Unless you want to apply our method to a new dataset, we therefore recommmend to use the first option.
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
### Preprocessed data
You can download our preprocessed data (73.4 GB) using
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
```
bash scripts/download_data.sh
```
## Name
Choose a self-explaining name for your project.
This script should download and unpack the data automatically into the `data/ShapeNet` folder.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
### Building the dataset
Alternatively, you can also preprocess the dataset yourself.
To this end, you have to follow the following steps:
* download the [ShapeNet dataset v1](https://www.shapenet.org/) and put into `data/external/ShapeNet`.
* download the [renderings and voxelizations](http://3d-r2n2.stanford.edu/) from Choy et al. 2016 and unpack them in `data/external/Choy2016`
* build our modified version of [mesh-fusion](https://github.com/davidstutz/mesh-fusion) by following the instructions in the `external/mesh-fusion` folder
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
You are now ready to build the dataset:
```
cd scripts
bash dataset_shapenet/build.sh
```
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
This command will build the dataset in `data/ShapeNet.build`.
To install the dataset, run
```
bash dataset_shapenet/install.sh
```
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
If everything worked out, this will copy the dataset into `data/ShapeNet`.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
When you have installed all binary dependencies and obtained the preprocessed data, you are ready to run our pretrained models and train new models from scratch.
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
### Generation
To generate meshes using a trained model, use
```
python generate.py CONFIG.yaml
```
where you replace `CONFIG.yaml` with the correct config file.
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
The easiest way is to use a pretrained model.
You can do this by using one of the config files
```
configs/img/onet_pretrained.yaml
configs/pointcloud/onet_pretrained.yaml
configs/voxels/onet_pretrained.yaml
configs/unconditional/onet_cars_pretrained.yaml
configs/unconditional/onet_airplanes_pretrained.yaml
configs/unconditional/onet_sofas_pretrained.yaml
configs/unconditional/onet_chairs_pretrained.yaml
```
which correspond to the experiments presented in the paper.
Our script will automatically download the model checkpoints and run the generation.
You can find the outputs in the `out/*/*/pretrained` folders.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
Please note that the config files `*_pretrained.yaml` are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
### Evaluation
For evaluation of the models, we provide two scripts: `eval.py` and `eval_meshes.py`.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
The main evaluation script is `eval_meshes.py`.
You can run it using
```
python eval_meshes.py CONFIG.yaml
```
The script takes the meshes generated in the previous step and evaluates them using a standardized protocol.
The output will be written to `.pkl`/`.csv` files in the corresponding generation folder which can be processed using [pandas](https://pandas.pydata.org/).
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
For a quick evaluation, you can also run
```
python eval.py CONFIG.yaml
```
This script will run a fast method specific evaluation to obtain some basic quantities that can be easily computed without extracting the meshes.
This evaluation will also be conducted automatically on the validation set during training.
All results reported in the paper were obtained using the `eval_meshes.py` script.
## License
For open source projects, say how it is licensed.
### Training
Finally, to train a new network from scratch, run
```
python train.py CONFIG.yaml
```
where you replace `CONFIG.yaml` with the name of the configuration file you want to use.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
You can monitor on <http://localhost:6006> the training process using [tensorboard](https://www.tensorflow.org/guide/summaries_and_tensorboard):
```
cd OUTPUT_DIR
tensorboard --logdir ./logs --port 6006
```
where you replace `OUTPUT_DIR` with the respective output directory.
For available training options, please take a look at `configs/default.yaml`.
# Notes
* In our paper we used random crops and scaling to augment the input images.
However, we later found that this image augmentation decreases performance on the ShapeNet test set.
The pretrained model that is loaded in `configs/img/onet_pretrained.yaml` was hence trained without data augmentation and has slightly better performance than the model from the paper. The updated table looks a follows:
![Updated table for single view 3D reconstruction experiment](img/table_img2mesh.png)
For completeness, we also provide the trained weights for the model which was used in the paper in `configs/img/onet_legacy_pretrained.yaml`.
* Note that training and evaluation of both our model and the baselines is performed with respect to the *watertight models*, but that normalization into the unit cube is performed with respect to the *non-watertight meshes* (to be consistent with the voxelizations from Choy et al.). As a result, the bounding box of the sampled point cloud is usually slightly bigger than the unit cube and may differ a little bit from a point cloud that was sampled from the original ShapeNet mesh.
# Futher Information
Please also check out the following concurrent papers that have proposed similar ideas:
* [Park et al. - DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019)](https://arxiv.org/abs/1901.05103)
* [Chen et al. - Learning Implicit Fields for Generative Shape Modeling (2019)](https://arxiv.org/abs/1812.02822)
* [Michalkiewicz et al. - Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019)](https://arxiv.org/abs/1901.06802)
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 2048
model:
encoder_latent: null
decoder: batchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/ablation/architecture_nocbn
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: -1
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 2048
model:
encoder_latent: null
decoder: cbatchnorm_noresnet
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/ablation/architecture_noresnet
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: -1
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: [1024, 1024]
model:
encoder_latent: null
decoder: cbatchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/ablation/sampling_equal
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: -1
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 64
model:
encoder_latent: null
decoder: cbatchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/ablation/sampling_few
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: -1
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 2048
points_file: points_surface.npz
model:
encoder_latent: null
decoder: cbatchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/ablation/sampling_surface
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: -1
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
method: onet
data:
dataset: Shapes3D
path: data/ShapeNet
classes: null
input_type: img
train_split: train
val_split: val
test_split: test
dim: 3
points_file: points.npz
points_iou_file: points.npz
points_subsample: 1024
points_unpackbits: true
model_file: model.off
watertight_file: model_watertight.off
img_folder: img
img_size: 224
img_with_camera: false
img_augment: false
n_views: 24
pointcloud_file: pointcloud.npz
pointcloud_chamfer_file: pointcloud.npz
pointcloud_n: 256
pointcloud_target_n: 1024
pointcloud_noise: 0.05
voxels_file: 'model.binvox'
with_transforms: false
model:
decoder: simple
encoder: resnet18
encoder_latent: null
decoder_kwargs: {}
encoder_kwargs: {}
encoder_latent_kwargs: {}
multi_gpu: false
c_dim: 512
z_dim: 64
use_camera: false
dmc_weight_prior: 10.
training:
out_dir: out/default
batch_size: 64
print_every: 10
visualize_every: 2000
checkpoint_every: 1000
validate_every: 2000
backup_every: 100000
eval_sample: false
model_selection_metric: loss
model_selection_mode: minimize
test:
threshold: 0.5
eval_mesh: true
eval_pointcloud: true
model_file: model_best.pt
generation:
batch_size: 100000
refinement_step: 0
vis_n_outputs: 30
generate_mesh: true
generate_pointcloud: true
generation_dir: generation
use_sampling: false
resolution_0: 32
upsampling_steps: 2
simplify_nfaces: null
copy_groundtruth: false
copy_input: true
latent_number: 4
latent_H: 8
latent_W: 8
latent_ny: 2
latent_nx: 2
latent_repeat: true
preprocessor:
type: null
config: ""
model_file: null
inherit_from: configs/img/onet_pretrained.yaml
data:
dataset: images
path: demo
training:
out_dir: demo
generation:
generation_dir: generation
refinement_step: 30
simplify_nfaces: 5000
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 2048
model:
encoder_latent: null
decoder: cbatchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/img/onet
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: 20000
validate_every: 20000
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
resolution_0: 32
upsampling_steps: 2
inherit_from: configs/img/onet.yaml
data:
img_augment: true
model:
decoder_kwargs:
legacy: true
training:
out_dir: out/img/onet_legacy
test:
model_file: https://s3.eu-central-1.amazonaws.com/avg-projects/occupancy_networks/models/onet_img2mesh-0c7780d1.pt
generation:
generation_dir: pretrained
inherit_from: configs/img/onet.yaml
test:
model_file: https://s3.eu-central-1.amazonaws.com/avg-projects/occupancy_networks/models/onet_img2mesh_3-f786b04a.pt
generation:
generation_dir: pretrained
method: pix2mesh
data:
multiclass: True
path: data/ShapeNet
base_mesh: im2mesh/pix2mesh/ellipsoid/face3.obj
ellipsoid: im2mesh/pix2mesh/ellipsoid/info_ellipsoid.dat
img_folder: img_choy2016
img_size: 224
img_with_camera: true
with_transforms: true
pointcloud_target_n: 8000
model:
encoder_latent: simple
decoder: simple
encoder: pixel2mesh_cond
encoder_kwargs: {}
decoder_kwargs:
adjust_ellipsoid: True
hidden_dim: 192
feat_dim: 963
c_dim: 512
z_dim: 64
adjust_losses: True
training:
out_dir: out/img/pixel2mesh
batch_size: 12
print_every: 40
visualize_every: 10000
checkpoint_every: 1000
validate_every: 2000
model_selection_metric: chamfer
test:
threshold: 0.9
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
generate_mesh: true
generate_pointcloud: false
method: psgn
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
pointcloud_target_n: 1024
model:
decoder: simple
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/img/psgn
batch_size: 64
test:
eval_mesh: false
eval_pointcloud: true
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
generate_mesh: false
method: r2n2
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
model:
encoder_latent: null
decoder: simple
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/img/r2n2
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
test:
threshold: 0.4
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1