Mapillary's computer vision technology

Mapillary is a lot more than geotagged images that have been uploaded to a server. We use computer vision to extract a vast amount of data from the images.

Mapillary uses a technology called Structure from Motion (SfM) to create and reconstruct places in 3D. By matching points between different images, SfM is able to locate the point in a three-dimensional space and therefore determine its location on the map. The more images there are available for a specific point, the more accurately it can be reconstructed. In addition, SfM creates the kind of smooth transitions between images that you can see in the Mapillary viewer—again, provided that images have been taken within close proximity with enough overlap between them.

We also run semantic segmentation on the images. This means that the computer tries to understand what is in the image and assigns a category tag to each pixel. That enables us to detect different objects in the images (such as buildings, pedestrians, cars etc.). Semantic segmentation together with 3D reconstruction enables us to extract 3D positions of objects such as traffic signs, and display them on the map.

Read more on our blog about:

• 3D reconstruction (using OpenSfM)
• Extracting 3D point clouds (read more)
• Improving image positions (read more)
• Dense 3D point clouds with semantic information (read more)
• Semantic segmentation (read more)
• Traffic sign recognition (read more)