The Verifier tool is a part of what we call Human in the Loop. It lets anyone contribute to the accuracy of Mapillary data by helping verify computer vision detections. These verifications will feed back into the system to help improve future detections. That’s where the name comes from—we’re introducing a human component into the learning loop of the machine.
In this tutorial, you can get an overview of how it works, how to get started, some tricky cases, and how you get points in the game.
Your task is to verify if the pattern in the image has been correctly detected. You will be shown an image with a red bounding box around the object that the algorithm has attempted to identify, together with a thumbnail of what this object has been identified as.
- If the two visuals match, click on the green button to approve.
- If they don’t match, click on the red button to reject.
- If you are not sure (sometimes the image is not clear), click on the Skip button.
NB! Don’t hesitate to use the skip button if you have any doubts. Since the goal is data verification, the output is only valuable if you’re confident that it’s correct.
You will be shown one image at a time. When you approve, reject, or skip the detection, you will be served the next image. However, if you make a mistake or change your mind, you can use the Back button within 10 seconds to go back to the previous image and make a correction. Note that you can only go back one step! When you move to the next image once more, your decision about the previous one will be locked and submitted.
In addition to some low-quality images, it might not always be obvious whether to approve or reject the detection. Here are some examples of tricky cases you might encounter. If you see any other cases where you're not sure what to do, choose skip. Also, feel free to grab a screenshot of it and email it to [email protected] so we can give you feedback and perhaps include it in the examples list. It might also be helpful to check our list of traffic signs that are currently detected.
- Sign is part of a bigger sign—approve
- Sign looks slightly different but means the same—approve
- Sign is tilted or rotated—approve
- Sign is a little bit hidden but it's clear which one it is—approve
- Sign is actually just a picture of a sign—approve (because the Verifier is related to teaching pattern recognition only; context will be given to the detections in later steps of our computer vision pipeline)
- Sign is self-made or someone has drawn on it but it looks "true enough" (so it's easy to see why the computer might not realize that it's fake)—approve (because the Verifier is related to teaching pattern recognition only; context will be given to the detections in later steps of our computer vision pipeline)
- Sign looks somewhat similar but actually isn't the same—reject
- Sign has the same meaning but looks totally different—reject
- Sign is blurred so you can't see what is on it—skip
To start verifying, you need to be logged in with your Mapillary account. Unless you’re already logged into the Mapillary web, you’ll be asked to log in before you start. If you don’t have an account yet, click on the link on the screen to create one. Once you’ve logged in, you will be shown the object that you’ll be verifying.
In the first version of the game, you can’t choose the object yourself but will be assigned one from a predefined set. However, you can share the link from the Share button at the top right so that you can invite your friends to verify the same object type and compete with you.
Start with some of these available tasks:
- Speed limit 50 MPH in the United States
- Speed limit 75 MPH in the United States
- Speed limit 30 km/h in Denmark
- Speed limit 50 km/h in Denmark
- Speed limit 70 km/h in Denmark
- Speed limit 110 km/h in Denmark
- Speed limit 20km/h in Germany
- Speed limit 60km/h in Germany
- Speed limit 100km/h in Germany
- Speed limit 120km/h in Germany
- Regulatory: End Of Buses Only
- Regulatory: Keep Right
- Regulatory: No Bicycles
- Regulatory: No Stopping
- Regulatory: Turn Right
- Warning: Falling Rocks Or Debris Left
- Warning: Loose Road Surface
- Warning: Roadworks
- Warning: Traffic Signals
To make this tool as fun as it is useful, it has been built as a game. You can see two numbers associated with your account as well as a leaderboard (on mobile, you might need to scroll down to see this).
The number that’s presented in the brackets is counting how many verifications you’ve submitted. You will see it increase it every time you click on the red or green button.
The other, more important number is given in bold font and that is your game score. You will get a point for every detection that someone else has verified in the same way as you did—i.e. if you approve a detection that already has been approved by another person, or if someone else rejects a detection that you have rejected as well (the order plays no role, both persons get a point when the image is similarly verified for the second time).
This point score is the basis of the leaderboard ranking. You will see the top 10 contributors and their scores so you can compare how you’re doing. If there haven’t been any matches between different people’s verifications yet, the leaderboard is empty. Keep submitting verifications and share the link to the task to get other people to do the same—that way, you’ll get a good head start and points will start landing on your account as others submit more verifications.
Comments
13 comments
A wrong sign for the task was displayed in the top right corner of the game, before I force refreshed the page with CTRL-F5. This happened in all the tasks, now its all fine.
Thanks for bringing this to our attention! I've sent this feedback to our developers. Keep the feedback coming here, or to [email protected] /Madelen
It looks for me, that this are all individual games.. you start with every game with 0 .. so there is no total points you can collect.. would be, my opinion, better, having one account and getting all points to one account and not for every game extra points.. what do you think ?
Thanks for the suggestion, Thorsten! Definitely something to think about as we develop the game further.
/Katrin
I have a question about a tricky case - for "No Left Turn in the United States" - some signs are combination no left turn and no U-turn in a single sign so they mean both. Should those be approved or not?
Hi Nina, thanks for writing about this! We consider the combination sign as a sign on its own - so the answer is that this sign should not be approved as a "No left turn" sign. I hope this helps for this and future cases - let us know if you ever have any other thoughts on this! /Madelen
Perfect, that's what I've been doing. Thanks!
Good afternoon. I found a link to "Verifier tool" in user messages but did not find a normal transition, you can show where it is, for the future.
There is a suggestion to add a place for a comment under the answer buttons, there are not unique cases and the user will be able to leave a comment on which it will be possible to isolate such pictures and double-check separately, there are signs on the rear door of trucks and you can always understand that he is on the car or on the trailer operator on the road next to another sign.
or as an option buttons with ready answers:
1. the picture is not a sign
2. the picture is reversed
3. In the picture, a sign with other data (for example at a reference speed of 110 km / h, 10 km / h) and leave the field for the correct value.
I apologize for the translation, google translit.
Hi georgisys,
Thanks for your feedback! I'm not sure I understand what it is that you didn't find. Could you explain some more, please? I'm sure we'll manage with help of Google Translate. :)
Good suggestions and we'll consider them for sure. Right now, we just collect yes/no data about the signs. If you're not sure, please skip the image. If you want to ask us about a specific example, please take a screenshot of it and email us at [email protected].
Katrin
Добрый день, я не мог найти ссылку на сайте, чтобы перейти к Verifier-tool, я случайно нашел ее в комментариях другого пользователя, теперь я уже нашел ее очень глубоко на вкладках.
Hi there,
I see what you mean now! Yes, so you can access verification tasks through the Tasker https://www.mapillary.com/app/task?menu=true, and we also share new tasks on our Twitter and Facebook channels every week so make sure to follow. :)
Katrin
Hi there,
wonderful idea!
I addes the tool as webapp to my android homescreen and did a lot of tasks, but the score keeps resetting to zero. Which is kind of demotivating
what happened there?
Michele
Please sign in to leave a comment.