The Verifier tool is a simple game-based tool for verifying Mapillary’s machine-generated object detections. Even the best algorithms always make some mistakes, so quality assurance by humans is a logical part of the workflow when using machine-generated data. The Verifier tool is a part of the Verification Projects toolset, which anyone can use for validating detections and checking for missed detections of chosen types of objects in their area of interest.
Tasks in the Verifier tool focus on a specific object class (such as stop signs or fire hydrants) and are about either validating object detections or checking whether any objects have been missed in images. This feedback will be used by Mapillary in the future to make the detection algorithms better (read more here). In addition, the detections that people reject will be removed, which improves the quality of the map data available on the platform.
To make the verification process more fun, the tool is built up as a game where you get points for the work you’ve done and compete with others on a leaderboard.
In this tutorial, you can get an overview of:
How it works
To start verifying, you need to be signed in with your Mapillary account on the Mapillary web. If you don’t have an account yet, click on the link on the screen to create one. Once you’ve signed in, you will be shown the object that you’re working with. What you need to do exactly depends on the task type: verifying detected objects or spotting missed objects.
Verifying detected objects
Your task is to verify if the object in the image has been correctly detected. You will be shown an image with a bounding box around the object that the algorithm has attempted to identify, together with a thumbnail or name of what this object has been identified as.
- If the object has been detected correctly, click on the green thumbs-up button to approve (or hit "E" on your keyboard).
- If it is not the correct object, click on the red thumbs-down button to reject(or hit "W" on your keyboard).
- If you are not sure (sometimes the image is not clear), click on the "Unsure" button (or hit "R" on your keyboard).
- If you want to see the whole image that this detection is from and get more context, click on the detection (or hit “F” on your keyboard). You can also zoom in and out on the image.
Note. Don’t hesitate to use the unsure-button if you have any doubts. Since the goal is data verification, it's important that you're confident about your decisions about each detection.
You will be shown one detection at a time. When you approve, reject, or click "Unsure", you will be served the next detection. However, if you make a mistake or change your mind, you can use the "Back" button (or hit "Q" on your keyboard) within 5 seconds to go back to the previous detection and change your decision. Note that you can only go back one step. When you move to the next detection once more, your decision about the previous one will be locked and submitted.
Spotting missed objects
Your task is to check if all objects of the type you’re working with have been annotated in the image. You’ll be shown crops of images together with bounding boxes of already identified objects (if there are any) and a counter telling you how many of that object we’ve already found in that image. You have the option to also view the whole image.
- If all objects of this type have been annotated, click on the green thumbs-up button to approve (or hit "E" on your keyboard).
- If one or more objects have been missed, click on the red thumbs-down button to reject(or hit "W" on your keyboard).
- If you are not sure (sometimes the image is not clear), click on the "Unsure" button (or hit "R" on your keyboard).
- If you want to see the whole image that this crop is from and get more context, click on the crop (or hit “F” on your keyboard). You can also zoom in and out on the image.
Note. Don’t hesitate to use the unsure-button if you have any doubts. Since the goal is data verification, it's important that you're confident about your decisions about each detection.
Each image will be shown to you in six subsequent crops so that it’s easier to see the contents. When you approve, reject, or click "Unsure", you will be served the next crop. However, if you make a mistake or change your mind, you can use the "Back" button (or hit "Q" on your keyboard) to go back and change your decision.
Within the set of six crops that belong to the same image, you can go back as many times as you want. Your votes will be submitted when you’ve done the last crop of the set and get served a new image. At that point, you will have only 5 seconds to go back to the previous image (but you’ll again have unlimited time to go back among the 6 crops of that image). Note that you can only go back one full image.
Examples of tricky cases
In addition to some low-quality images, it might not always be obvious whether to approve or reject the detection. Here are some examples of tricky cases you might encounter. If you see any other cases where you're not sure what to do, choose "Unsure". It might also be helpful to check our list of traffic signs and list of objects available as point features so you know which object classes our algorithms detect altogether. Note there are different tricky cases and instructions for verifying detected objects and spotting missed objects.
Verifying detected objects
- Traffic sign is part of a bigger sign—approve
- The rectangle is not drawn precisely around the object—approve
- Traffic sign looks slightly different but means the same—approve
- Traffic sign or object is tilted, rotated, or viewed from another angle—approve
- Traffic sign or object is a little bit hidden but it's clear which one it is—approve
- Traffic sign is actually just a picture of a sign—reject
- Traffic sign is self-made or someone has drawn on it but it looks "true enough" (so it's easy to see why the computer might not realize that it's fake)—reject
- Traffic sign looks somewhat similar but actually isn't the same—reject (this is where the lists of all traffic signs and detected objects might be handy, so you get an idea of what other objects something may be categorized as).
- Traffic sign has the same meaning but looks totally different—reject
- Traffic sign is blurred so you can't see what is on it—mark as unsure
Spotting missed objects
- Some objects are annotated but one or more have been missed at the same time—reject the image
- The object you are looking for is only partially visible in the image crop—reject the image
- An existing annotation does not look perfect (e.g. the bounding box doesn’t match the object precisely)—approve the image
The game
To make this tool as fun as it is useful, it has been built as a game. You can see two numbers associated with your account as well as a leaderboard (on mobile, you might need to scroll down to see this).
The number that’s presented in the brackets is counting how many detections or images you’ve voted on. You will see it increase every time you vote on a detection, or in the missed objects task every time you’ve voted on all six crops of an image.
The other, more important number is given in bold font and that is your game score. You will get a point for every detection or image that someone else has voted on in the same way as you did—i.e. if you approve something that has already been approved by another person, or if someone else rejects something that you have rejected as well (the order plays no role, both of you get a point when the detection or image is similarly voted on for the second time).
This point score is the basis of the leaderboard ranking. You will see the contributors and their scores so you can compare how you’re doing. If there haven’t been any matches between different people’s votes yet, the leaderboard is empty. Keep on playing to get a good head start and points will start landing on your account as others submit more votes.
Comments
16 comments
A wrong sign for the task was displayed in the top right corner of the game, before I force refreshed the page with CTRL-F5. This happened in all the tasks, now its all fine.
Thanks for bringing this to our attention! I've sent this feedback to our developers. Keep the feedback coming here, or to support@mapillary.com /Madelen
It looks for me, that this are all individual games.. you start with every game with 0 .. so there is no total points you can collect.. would be, my opinion, better, having one account and getting all points to one account and not for every game extra points.. what do you think ?
Thanks for the suggestion, Thorsten! Definitely something to think about as we develop the game further.
/Katrin
I have a question about a tricky case - for "No Left Turn in the United States" - some signs are combination no left turn and no U-turn in a single sign so they mean both. Should those be approved or not?
Hi Nina, thanks for writing about this! We consider the combination sign as a sign on its own - so the answer is that this sign should not be approved as a "No left turn" sign. I hope this helps for this and future cases - let us know if you ever have any other thoughts on this! /Madelen
Perfect, that's what I've been doing. Thanks!
Good afternoon. I found a link to "Verifier tool" in user messages but did not find a normal transition, you can show where it is, for the future.
There is a suggestion to add a place for a comment under the answer buttons, there are not unique cases and the user will be able to leave a comment on which it will be possible to isolate such pictures and double-check separately, there are signs on the rear door of trucks and you can always understand that he is on the car or on the trailer operator on the road next to another sign.
or as an option buttons with ready answers:
1. the picture is not a sign
2. the picture is reversed
3. In the picture, a sign with other data (for example at a reference speed of 110 km / h, 10 km / h) and leave the field for the correct value.
I apologize for the translation, google translit.
Hi georgisys,
Thanks for your feedback! I'm not sure I understand what it is that you didn't find. Could you explain some more, please? I'm sure we'll manage with help of Google Translate. :)
Good suggestions and we'll consider them for sure. Right now, we just collect yes/no data about the signs. If you're not sure, please skip the image. If you want to ask us about a specific example, please take a screenshot of it and email us at support@mapillary.com.
Katrin
Добрый день, я не мог найти ссылку на сайте, чтобы перейти к Verifier-tool, я случайно нашел ее в комментариях другого пользователя, теперь я уже нашел ее очень глубоко на вкладках.
Hi there,
I see what you mean now! Yes, so you can access verification tasks through the Tasker https://www.mapillary.com/app/task?menu=true, and we also share new tasks on our Twitter and Facebook channels every week so make sure to follow. :)
Katrin
Hi there,
wonderful idea!
I addes the tool as webapp to my android homescreen and did a lot of tasks, but the score keeps resetting to zero. Which is kind of demotivating
what happened there?
Michele
Whereas this tool seems promising, I didn't understand where this data is stored. I understand that manual votes serve as algorithm's training, but I am wondering if you store those manual data and if (hopefully!) that geolocated data could be added back to OSM.
We need some definitions for the object we are verifying. Mine was "bench" which I took to be a public place to sit down, like a park or sidewalk bench. However, some of the pictures had chairs by the roadway, private benches by the sidewalk, full picnic tables, benches on someone's private lawn, etc. If you mean publicly available places to sit, please say so somewhere in the details of how to verify those objects. Nice game - the only thing I was missing was the ability to magnify the image which Windows 10's Win+ + keys let me do on my PC.
I am truly amazed that this help page does NOT contain a link to the actual Verifier Tool! If you do a Google search for "mapillary verify" all you get is help pages like these, but nothing that leads to a page where you can actually verify.
Things that need to be fixed here:
Oh, and while I'm ranting anyway:
Please sign in to leave a comment.