The Verifier tool is a simple game-based tool for verifying Mapillary’s machine-generated object detections. Even the best algorithms always make some mistakes, so quality assurance by humans is a logical part of the workflow when using machine-generated data. The Verifier tool is a part of the Verification Projects toolset, which anyone can use for validating detections of chosen types of objects in their area of interest.
The Verifier tool serves verification tasks to people, where each task focuses on a specific object class, such as stop signs or fire hydrants. As people confirm or reject the detections, any faulty data will be removed shortly, so there is a quick improvement in the quality of the map data available on the Mapillary platform.
Furthermore, the verifications will feed back into Mapillary’s algorithms to help improve future detections (this is called the Human in the Loop—we’re introducing a human component into the learning loop of the machine).
To make the verification process more fun, the tool is built up as a game where you get points for the work you’ve done and compete with others on a leaderboard.
In this tutorial, you can get an overview of:
To start verifying, you need to be signed in with your Mapillary account. Unless you’re already signed into the Mapillary web, you’ll be asked to log in before you start. If you don’t have an account yet, click on the link on the screen to create one. Once you’ve signed in, you will be shown the object that you’re working with.
Your task is to verify if the object in the image has been correctly detected. You will be shown an image with a bounding box around the object that the algorithm has attempted to identify, together with a thumbnail or name of what this object has been identified as.
- If the object has been detected correctly, click on the green thumbs-up button to approve (or hit "E" on your keyboard).
- If it is not the correct object, click on the red thumbs-down button to reject(or hit "W" on your keyboard).
- If you are not sure (sometimes the image is not clear), click on the "Unsure" button (or hit "R" on your keyboard).
Note. Don’t hesitate to use the unsure-button if you have any doubts. Since the goal is data verification, it's important that you're confident about your decisions about each detection.
You will be shown one image at a time. When you approve, reject, or click "Unsure", you will be served the next image. However, if you make a mistake or change your mind, you can use the "Back" button (or hit "Q" on your keyboard) within 10 seconds to go back to the previous image and make a correction. Note that you can only go back one step! When you move to the next image once more, your decision about the previous one will be locked and submitted.
In addition to some low-quality images, it might not always be obvious whether to approve or reject the detection. Here are some examples of tricky cases you might encounter. If you see any other cases where you're not sure what to do, choose "Unsure". It might also be helpful to check our list of traffic signs and list of objects available as point features so you know which object classes our algorithms detect altogether.
- Traffic sign is part of a bigger sign—approve
- The rectangle is not drawn precisely around the object—approve
- Traffic sign looks slightly different but means the same—approve
- Traffic sign or object is tilted, rotated, or viewed from another angle—approve
- Traffic sign or object is a little bit hidden but it's clear which one it is—approve
- Traffic sign is actually just a picture of a sign—reject
- Traffic sign is self-made or someone has drawn on it but it looks "true enough" (so it's easy to see why the computer might not realize that it's fake)—reject
- Traffic sign looks somewhat similar but actually isn't the same—reject (this is where the lists of all traffic signs and detected objects might be handy, so you get an idea of what other objects something may be categorized as).
- Traffic sign has the same meaning but looks totally different—reject
- Traffic sign is blurred so you can't see what is on it—mark as unsure
To make this tool as fun as it is useful, it has been built as a game. You can see two numbers associated with your account as well as a leaderboard (on mobile, you might need to scroll down to see this).
The number that’s presented in the brackets is counting how many detections you’ve approved or rejected. You will see it increase it every time you click on the red or green button.
The other, more important number is given in bold font and that is your game score. You will get a point for every detection that someone else has voted on in the same way as you did—i.e. if you approve a detection that already has been approved by another person, or if someone else rejects a detection that you have rejected as well (the order plays no role, both of you get a point when the image is similarly voted on for the second time).
This point score is the basis of the leaderboard ranking. You will see the contributors and their scores so you can compare how you’re doing. If there haven’t been any matches between different people’s verifications yet, the leaderboard is empty. Keep on playing to get a good head start and points will start landing on your account as others submit more verifications.