The Verifier tool is a simple game-based tool for verifying Mapillary’s machine-generated object detections. Even the best algorithms always make some mistakes, so quality assurance by humans is a logical part of the workflow when using machine-generated data. The Verifier tool is a part of the Verification Projects toolset, which anyone can use for validating detections of chosen types of objects in their area of interest.
The Verifier tool serves verification tasks to people, where each task focuses on a specific object class, such as stop signs or fire hydrants. As people confirm or reject the detections, any faulty data will be removed shortly, so there is a quick improvement in the quality of the map data available on the Mapillary platform.
Furthermore, the verifications will feed back into Mapillary’s algorithms to help improve future detections (this is called the Human in the Loop—we’re introducing a human component into the learning loop of the machine).
To make the verification process more fun, the tool is built up as a game where you get points for the work you’ve done and compete with others on a leaderboard.
In this tutorial, you can get an overview of:
How it works
To start verifying, you need to be signed in with your Mapillary account. Unless you’re already signed into the Mapillary web, you’ll be asked to log in before you start. If you don’t have an account yet, click on the link on the screen to create one. Once you’ve signed in, you will be shown the object that you’re working with.
Your task is to verify if the object in the image has been correctly detected. You will be shown an image with a bounding box around the object that the algorithm has attempted to identify, together with a thumbnail or name of what this object has been identified as.
- If the object has been detected correctly, click on the green thumbs-up button to approve (or hit "E" on your keyboard).
- If it is not the correct object, click on the red thumbs-down button to reject(or hit "W" on your keyboard).
- If you are not sure (sometimes the image is not clear), click on the "Unsure" button (or hit "R" on your keyboard).
Note. Don’t hesitate to use the unsure-button if you have any doubts. Since the goal is data verification, it's important that you're confident about your decisions about each detection.
You will be shown one image at a time. When you approve, reject, or click "Unsure", you will be served the next image. However, if you make a mistake or change your mind, you can use the "Back" button (or hit "Q" on your keyboard) within 10 seconds to go back to the previous image and make a correction. Note that you can only go back one step! When you move to the next image once more, your decision about the previous one will be locked and submitted.
Examples of tricky cases
In addition to some low-quality images, it might not always be obvious whether to approve or reject the detection. Here are some examples of tricky cases you might encounter. If you see any other cases where you're not sure what to do, choose "Unsure". It might also be helpful to check our list of traffic signs and list of objects available as point features so you know which object classes our algorithms detect altogether.
- Traffic sign is part of a bigger sign—approve
- The rectangle is not drawn precisely around the object—approve
- Traffic sign looks slightly different but means the same—approve
- Traffic sign or object is tilted, rotated, or viewed from another angle—approve
- Traffic sign or object is a little bit hidden but it's clear which one it is—approve
- Traffic sign is actually just a picture of a sign—reject
- Traffic sign is self-made or someone has drawn on it but it looks "true enough" (so it's easy to see why the computer might not realize that it's fake)—reject
- Traffic sign looks somewhat similar but actually isn't the same—reject (this is where the lists of all traffic signs and detected objects might be handy, so you get an idea of what other objects something may be categorized as).
- Traffic sign has the same meaning but looks totally different—reject
- Traffic sign is blurred so you can't see what is on it—mark as unsure
The game
To make this tool as fun as it is useful, it has been built as a game. You can see two numbers associated with your account as well as a leaderboard (on mobile, you might need to scroll down to see this).
The number that’s presented in the brackets is counting how many detections you’ve approved or rejected. You will see it increase it every time you click on the red or green button.
The other, more important number is given in bold font and that is your game score. You will get a point for every detection that someone else has voted on in the same way as you did—i.e. if you approve a detection that already has been approved by another person, or if someone else rejects a detection that you have rejected as well (the order plays no role, both of you get a point when the image is similarly voted on for the second time).
This point score is the basis of the leaderboard ranking. You will see the contributors and their scores so you can compare how you’re doing. If there haven’t been any matches between different people’s verifications yet, the leaderboard is empty. Keep on playing to get a good head start and points will start landing on your account as others submit more verifications.
Comments
14 comments
A wrong sign for the task was displayed in the top right corner of the game, before I force refreshed the page with CTRL-F5. This happened in all the tasks, now its all fine.
Thanks for bringing this to our attention! I've sent this feedback to our developers. Keep the feedback coming here, or to support@mapillary.com /Madelen
It looks for me, that this are all individual games.. you start with every game with 0 .. so there is no total points you can collect.. would be, my opinion, better, having one account and getting all points to one account and not for every game extra points.. what do you think ?
Thanks for the suggestion, Thorsten! Definitely something to think about as we develop the game further.
/Katrin
I have a question about a tricky case - for "No Left Turn in the United States" - some signs are combination no left turn and no U-turn in a single sign so they mean both. Should those be approved or not?
Hi Nina, thanks for writing about this! We consider the combination sign as a sign on its own - so the answer is that this sign should not be approved as a "No left turn" sign. I hope this helps for this and future cases - let us know if you ever have any other thoughts on this! /Madelen
Perfect, that's what I've been doing. Thanks!
Good afternoon. I found a link to "Verifier tool" in user messages but did not find a normal transition, you can show where it is, for the future.
There is a suggestion to add a place for a comment under the answer buttons, there are not unique cases and the user will be able to leave a comment on which it will be possible to isolate such pictures and double-check separately, there are signs on the rear door of trucks and you can always understand that he is on the car or on the trailer operator on the road next to another sign.
or as an option buttons with ready answers:
1. the picture is not a sign
2. the picture is reversed
3. In the picture, a sign with other data (for example at a reference speed of 110 km / h, 10 km / h) and leave the field for the correct value.
I apologize for the translation, google translit.
Hi georgisys,
Thanks for your feedback! I'm not sure I understand what it is that you didn't find. Could you explain some more, please? I'm sure we'll manage with help of Google Translate. :)
Good suggestions and we'll consider them for sure. Right now, we just collect yes/no data about the signs. If you're not sure, please skip the image. If you want to ask us about a specific example, please take a screenshot of it and email us at support@mapillary.com.
Katrin
Добрый день, я не мог найти ссылку на сайте, чтобы перейти к Verifier-tool, я случайно нашел ее в комментариях другого пользователя, теперь я уже нашел ее очень глубоко на вкладках.
Hi there,
I see what you mean now! Yes, so you can access verification tasks through the Tasker https://www.mapillary.com/app/task?menu=true, and we also share new tasks on our Twitter and Facebook channels every week so make sure to follow. :)
Katrin
Hi there,
wonderful idea!
I addes the tool as webapp to my android homescreen and did a lot of tasks, but the score keeps resetting to zero. Which is kind of demotivating
what happened there?
Michele
Whereas this tool seems promising, I didn't understand where this data is stored. I understand that manual votes serve as algorithm's training, but I am wondering if you store those manual data and if (hopefully!) that geolocated data could be added back to OSM.
Please sign in to leave a comment.