Verification projects

Verification Projects is a toolset that lets organizations improve the quality of map data in their area by manually verifying whether Mapillary’s algorithms have detected objects correctly in images. As a result, incorrect detections can be removed from the map data available on the platform. The verifications will also be used to further improve Mapillary’s detection algorithms, leading to higher data accuracy in the future.

Verification Projects consist of an admin dashboard for managing the project and a set of verification tasks, available through the Verifier tool (one for each type of object that is included in the project). Each project has a link that you share with people that do the verifications—you can choose if you keep it among your own team members or also distribute outside your organization.

The verification project enables validating object detections, i.e. segments of images that have been identified as a certain object. But removing incorrect object detections also improves the quality and accuracy of map features, since every map feature (an object that has been positioned on the map) is based on triangulating multiple object detections.

This article covers:

1. Accessing Verification Projects
2. Setting up a project
3. Managing a project
4. Editing and deleting projects

For instructions about doing verification tasks in a project, see the article about the verifier tool.

 

Accessing Verification Projects

To start using verification projects, go to your organization dashboard on the Mapillary web app. As an admin, you’ll see the “Verification” item in the left-side menu (this is not visible to non-admin team members). Here you’ll see a list of all your projects. You can open each one to manage it or just check in on progress, as well as add new projects.

Setting up a project

To set up a new capture project, click on the “Add new project” button in the Capture tab of your organization dashboard. Each project is based on an existing shape, so you need to have at least one shape set up.

In the first step of setup, you’ll name your project (this will only be visible on the admin dashboard) and select the originating shape—in other words, choose the area where you want to verify map data.

Next, select if your project will be about verifying traffic sign detections (Mapillary identifies 1,500 traffic sign classes from 100 countries) or detections of other objects (42 object classes that Mapillary is able to position on the map as point features).

In the following step, you’ll pick the specific traffic sign classes or object classes that you want to include in your verification project. You can include as many as you’d like. Just think about your use case and workflow—it may be easier to get tasks done and manage projects when you break it down to smaller projects.

In the final step of the setup, you need to add a public name and description for your project. This will be seen by anyone that you share the project link with, so it’s a good place to briefly explain why this project is being run (e.g. what will the data be used for) as well as what to pay attention to when verifying. For instance, objects that look similar but are still different (like manholes and catch basins), or what an object would typically look like in your chosen area (are benches expected to be made of wood or concrete or something else…).

Last, to click on the “Create capture project” button at the bottom of the sidebar to complete the setup.

 

Managing a project

When you’ve set up a project, you will see it on the list in the Verification tab of the organization dashboard. Click on a project to view its details, check in on progress, and retrieve the link to the project that you can share with people that will be doing the verifications.

When you click on the “Details” button for a task, you’ll see an overview of how many detections of this object there are in your shape and a breakdown of how many of those have already been verified. To increase the quality of the verifications, at least two people need to similarly approve or reject each detection so that it would get verified.

• Not verified yet—nobody has approved or rejected these detections yet.
• Verifications pending—someone has approved or rejected the detection but similar input from another person is needed before the detection becomes verified.
• Verified detections—at least two people have confirmed or rejected this detection and it is now verified.
• Total detections in project—the total number of detections of this object in your shape. This number can change as new imagery coverage appears in the area (whether uploaded by you and your team or other Mapillary contributors).

The people that help you do verifications don’t need to be members of your organization; they only need to have a Mapillary account. The public link that you distribute to them will not give them access to the project dashboard, but just a project page where the public project name and description show up that you specified in the setup process, together with the list of tasks that the project includes.

Editing and deleting projects

Once a verification project is set up, you can’t edit the list of objects it includes, nor the shape it is based on. You should just create a new project if you need to verify more object classes.

You can’t delete a project but you can pause it if you want to take it offline. The public link to the project will then inform people about it. You can also edit the internal name that the project has on your dashboard. Both of these options are available in the project list view.

You can also edit the project’s public name and description at any time. For that, open the project and click on the little pen icon next to each of those fields in the project view.