Neural Radiance Fields, or NeRFs, are a relatively new type of 3D model that can be generated from a set of 2D images for a scene. Follow these instructions to capture your own surroundings and suggest them for NeRF reconstructions.

 

📚Prerequisites

• Download the Mapillary app from the App Store or Google Play Store.
• On iOS: Make sure to activate wide-angle mode. Click on the Profile Icon in the app and then on the settings button on the top right and then switch on “Use wide-angle lens”.
• Familiarize yourself with the Mapillary capture flow.
  • Press the green capture button to start automatic capture while moving. If you see a “Low GPS accuracy” message, wait until it disappears.
  • The app captures automatically as you move.
  • Optional / For expert users: You can turn on manual capture for better control of what you capture. Click on the Profile Icon in the app and then on the settings button on the top right and then switch on “Allow manual capture”.

Note: In these capture instructions we focus on capturing with the Mapillary app itself, but you can also capture with an action camera, i.e. GoPro (though for NeRF capture it’s currently best not to use 360 camera modes), or capture images simply with the camera of your phone (i.e. to use the wide-angle camera on Android) and then upload these images via the Mapillary desktop uploader. The main thing is to make sure that your images are geo-tagged. Other than that, you can follow the instructions as outlined below.

🎡Find a good capture location and time

For your capture location and time make sure that:

• Your chosen location is interesting and ideally public, i.e. the main square of your city or an area near or in a point of interest.
• Your chosen location does not have too many big moving objects, like cars, trams, etc. A few or smaller moving objects, like a few people, are okay.
• Your scene has a walkable area that is roughly between 10m X 10m and 200m X 200m. If the location you want to capture is bigger, limit your capture area to a maximum of 200m X 200m.
• Your captured scene should not have too many reflective surfaces, like glass or shiny metal, and it is better to avoid water bodies.
• Time and weather: Ensure good and even lighting of the scene. You can get good results on a sunny day, but ideal is an overcast day with lots of diffuse light.
• Make sure that the camera lens is cleaned and that there is sufficient storage space on your device left.

📸Capture your scene

As a first step look at this example of a good NeRFable capture.
While you capture your scene make sure to follow these rules:

1. Stay within your chosen capture area, which should not exceed 200m x 200m (650ft x 650ft). Ideally your scene is (almost) square. Best results are obtained if your capture area is square, but up to a ratio of 2:1, i.e. 100m x 50m, you should still get good results.
2. Make sure there is only a limited amount of moving objects and ideally these moving objects are small, i.e. humans or a few cars, but no trucks, trains or trams.
3. Try to capture as many images as possible. As a rule of thumb you should capture 300+ images to get a good reconstruction. For excellent results try to capture 600+ images.
4. Never obscure the image yourself: Make sure that your finger or legs are never visible in the captured frames.
5. Try to move slowly to avoid motion blur as much as possible.
6. A good start is usually to start by circling around the scene: And try to gradually move between inside and outside views. Afterwards try to capture from as many walkable locations and different capture angles as possible.

7. Make sure that your scene including all objects of interest and the ground are captured from as many unique viewpoints as possible. Avoid capturing any part of the scene with just one or a few images especially do not capture with one of the following patterns.
   1. Don’t circle the scene or object of interest just once. Instead circle it multiple times and try to capture the scene from different heights (hip level, head level and above your head) and with different camera angles.

1. 2. Don’t capture a linear trajectory, i.e. do not follow just a street. Instead try to focus on a limited area which you capture with multiple images.

1. 3. Don’t approach the scene from a distance, only start capturing once you are inside the your chosen capture area.

8. Generally, it is best to capture in diffuse light conditions, i.e. an overcast day. Direct sunlight is not ideal, but if you cannot avoid it, try to avoid moving shadows as much as possible, especially your own shadow and shadows from big moving objects.

9. Try to avoid big texture-less surfaces in your scene (i.e. a big wall in the same color). However, if you can’t avoid these surfaces try to capture textureless surfaces always jointly with at least some areas that have texture.

10. Once you have reviewed these capture instructions start with the actual capture. Simply press the record button in the Mapillary app and start moving as indicated in these instructions. The app will capture images itself and there is no need to press the capture button.

Capturing a single scene should take no longer than 20 minutes after some practice.

 

📤Upload & suggest your scene

Once you have uploaded your capture it should be visible on mapillary.com within 2 days or less. Note: this does not mean that the imagery will be processed as a NeRF video.
To submit your imagery for NeRF consideration follow these steps:

1. Select a sequence on the Mapillary map and then click on the Image options at the bottom right (three dot symbol).

2. Click on Advanced, such that the drop down information shows as in the screenshot below. Then click on “Suggest for NeRF”.

3. After double checking that your selected sequence fulfills all requirements outlined in this capture guide, click on “Suggest for NeRF”. It is sufficient to tag only one sequence if your capture is split in multiple sequences.

This lets the Mapillary team know that you have followed these instructions and would like for us to process your capture and attempt to create a NeRF. Note that NeRF scenes will only be generated for very high quality captures and should be visible on the web within approximately a week. If you have any questions, please contact us or leave a comment in the forum.

 

Happy capturing!🚀