Verification projects

Verification Projects is a toolset that lets organizations improve the quality of map data in their area by manually verifying whether Mapillary’s algorithms have done a good job detecting objects in images. As a result, incorrect detections can be removed from the map data available on the platform. The verifications will also be used to further improve Mapillary’s detection algorithms, leading to higher data accuracy in the future.

Verification Projects consist of an admin dashboard for managing the project and a set of verification tasks, which are available through the Verifier tool.  For each type of object included in the project, there are two types of tasks:

1. Verifying detected objects;
2. Checking images for missing detections. 

Each project has a link that you share with people that do the verifications—you can choose if you keep it among your own team members or also distribute outside your organization. If you’d like our contributor network to help you do verifications, you can list your project on the Mapillary Marketplace.

This article covers:

1. Accessing Verification Projects
2. Setting up a project
3. Managing a project
   • Verifying detected objects
   • Spotting missed objects
   • Sharing the project with contributors
4. Editing and deleting projects
5. Impact on map data quality

For instructions about doing verification tasks in a project, see the article about the verifier tool.

 

Accessing Verification Projects

To start using verification projects, go to your organization dashboard on the Mapillary web app. As an admin, you’ll see the “Verification” item in the left-side menu (this is not visible to non-admin team members). Here you’ll see a list of all your projects. You can open each one to manage it or just check in on progress, as well as add new projects.

 

Setting up a project

To set up a new verification project, click on the “Add new project” button in the Data Verification tab of your organization dashboard. Each project is based on an existing shape, so you need to have at least one shape set up.

In the first step of setup, you’ll name your project (this will only be visible on the admin dashboard) and select the originating shape—in other words, choose the area where you want to verify data. You also choose whether you want your project to be listed on the Mapillary Marketplace, which can help you find more contributors to do verifications in your project.

Next, select if your project will be about verifying traffic sign detections (Mapillary identifies 1,500 traffic sign classes from 100 countries) or detections of other objects (42 object classes that Mapillary is able to position on the map as point features).

In the following step, you’ll pick the specific traffic sign classes or object classes that you want to include in your verification project. You can include as many as you’d like. Just think about your use case and workflow—it may be easier to get tasks done and manage projects when you break it down to smaller projects.

In the final step of the setup, you need to add a public name and description for your project. This will be seen by anyone that you share the project link with, and if you've decided to list your project on the Marketplace, it will be visible there too.

So it’s a good place to briefly explain why this project is being run (e.g. what will the data be used for) as well as what to pay attention to when verifying. For instance, objects that look similar but are still different (like manholes and catch basins), or what an object would typically look like in your chosen area (e.g. are benches expected to be made of wood or concrete or something else, etc.).

Last, to click on the Create project button at the bottom to complete the setup.

 

Managing a project

When you’ve set up a project, you will see it on the list in the Data verification tab of the organization dashboard. Click on a project to view its details, check progress, and retrieve the link to the project that you can share with people that will be doing the verifications.

When you click on the Details button for a task, you’ll see an overview of how many verifications there are in the task altogether and a breakdown of how many have already been done. To increase the quality of the verifications, at least two people need to similarly approve or reject each item so that it would get verified.

 

Verifying detected objects

In this task, people are shown object detections and they need to either approve or reject each one, depending on whether it’s the correct object.

• Detections not verified yet—nobody has approved or rejected these detections yet.
• Detections pending—someone has approved or rejected the detection but similar input from another person is needed before the detection becomes verified.
• Detections verified—at least two people have approved or rejected this detection and it is now verified.
• Total detections in project—the total number of detections of this object in your shape. This number can change as new imagery appears in the area (whether uploaded by you and your team or other Mapillary contributors).

 

Spotting missed objects

In this task, people are shown images which they need to approve or reject, depending on whether all cases of the object in question have been annotated in the image (e.g. approve if all stop signs visible in the image appear to be annotated).

Note that to make it easier to see, each image is served to the verifier as six subsequent crops. All stats are calculated based on whole images, not the individual crops. So if any of the crops get rejected because it’s not fully annotated, this means the whole image will be counted as missing annotations.

• Images not verified yet—nobody has approved or rejected these images yet.
• Images pending—someone has approved or rejected the image ( but similar input from another person is needed before the image becomes verified.
• Images verified—at least two people have approved or rejected this image and it is now verified.
• Total images in project—the total number of images in your shape. This number can change as new imagery appears in the area (whether uploaded by you and your team or other Mapillary contributors).

 

 

Sharing the project with contributors

The people that help you do verifications don’t need to be members of your organization; they only need to have a Mapillary account. The public link that you distribute to them will not give them access to the project dashboard, but just a project page where the public project name and description show up that you specified in the setup process, together with the list of tasks that the project includes.

 

 

Editing and deleting projects

Once a verification project is set up, you can’t edit the list of objects it includes, nor the shape it is based on. You should just create a new project if you need to verify more object classes.

You can’t delete a project but you can pause it if you want to take it offline. The public link to the project will then inform people about it. You can also edit the internal name that the project has on your dashboard. Both of these options are available in the project list view.

You can also edit the project’s public name and description at any time. For that, open the project and click on the little pen icon next to each of those fields in the project view.

 

Impact on map data quality

The tasks for validating object detections (i.e. segments of images that have been annotated as a certain object) have a direct impact on the map data available on the platform. Removing incorrect object detections improves the quality of map features since every map feature (an object that has been positioned on the map) is based on triangulating detections of an object in multiple images.

As humans, just like machines, may make errors, the verification process requires that at least two people reject an object detection before we consider it fully verified and remove it from the available data. When a detection is deemed to be false, we recalculate the positions of any map features that were using this detection for triangulation. This usually happens within a few hours after the detection has been verified.

The tasks for checking images for missing detections don’t currently have a short-term consequence. But it’s important input for improving the Mapillary algorithms so that the quality of machine-generated data would get better. All verifications from both types of tasks will be used in time as we develop our technology so that more and more object detections will be correct, and less and less objects will be missed in images.