The Verifier tool is a part of what we call Human in the Loop. It lets anyone contribute to the accuracy of Mapillary data by helping verify computer vision detections. These verifications will feed back into the system to help improve future detections. That’s where the name comes from—we’re introducing a human component into the learning loop of the machine.
Your task is to verify if the pattern in the image has been correctly detected. You will be shown an image with a red bounding box around the object that the algorithm has attempted to identify, together with a thumbnail of what this object has been identified as.
- If the two visuals match, click on the green button to approve.
- If they don’t match, click on the red button to reject.
- If you are not sure (sometimes the image is not clear), click on the Skip button.
NB! Don’t hesitate to use the skip button if you have any doubts. Since the goal is data verification, the output is only valuable if you’re confident that it’s correct.
You will be shown one image at a time. When you approve, reject, or skip the detection, you will be served the next image. However, if you make a mistake or change your mind, you can use the Back button within 10 seconds to go back to the previous image and make a correction. Note that you can only go back one step! When you move to the next image once more, your decision about the previous one will be locked and submitted.
In addition to some low-quality images, it might not always be obvious whether to approve or reject the detection. Here are some examples of tricky cases you might encounter. If you see any other cases where you're not sure what to do, choose skip. Also, feel free to grab a screenshot of it and email it to firstname.lastname@example.org so we can give you feedback and perhaps include it in the examples list. It might also be helpful to check our list of traffic signs that are currently detected.
- Sign is part of a bigger sign—approve
- Sign looks slightly different but means the same—approve
- Sign is tilted or rotated—approve
- Sign is a little bit hidden but it's clear which one it is—approve
- Sign is actually just a picture of a sign—approve (because the Verifier is related to teaching pattern recognition only; context will be given to the detections in later steps of our computer vision pipeline)
- Sign is self-made or someone has drawn on it but it looks "true enough" (so it's easy to see why the computer might not realize that it's fake)—approve (because the Verifier is related to teaching pattern recognition only; context will be given to the detections in later steps of our computer vision pipeline)
- Sign looks similar but doesn't mean the same—reject
- Sign has the same meaning but looks different—reject
- Sign is blurred so you can't see what is on it—skip
To start verifying, you need to be logged in with your Mapillary account. Unless you’re already logged into the Mapillary web, you’ll be asked to log in before you start. If you don’t have an account yet, click on the link on the screen to create one. Once you’ve logged in, you will be shown the object that you’ll be verifying.
In the first version of the game, you can’t choose the object yourself but will be assigned one from a predefined set. However, you can share the link from the Share button at the top right so that you can invite your friends to verify the same object type and compete with you.
Start with some of these available tasks:
- Speed limit 50 MPH in the United States
- Speed limit 75 MPH in the United States
- Speed limit 30 km/h in Denmark
- Speed limit 50 km/h in Denmark
- Speed limit 70 km/h in Denmark
- Speed limit 110 km/h in Denmark
- Speed limit 20km/h in Germany
- Speed limit 60km/h in Germany
- Speed limit 100km/h in Germany
- Speed limit 120km/h in Germany
- Regulatory: End Of Buses Only
- Regulatory: Keep Right
- Regulatory: No Bicycles
- Regulatory: No Stopping
- Regulatory: Turn Right
- Warning: Falling Rocks Or Debris Left
- Warning: Loose Road Surface
- Warning: Roadworks
- Warning: Traffic Signals
To make this tool as fun as it is useful, it has been built as a game. You can see two numbers associated with your account as well as a leaderboard (on mobile, you might need to scroll down to see this).
The number that’s presented in the brackets is counting how many verifications you’ve submitted. You will see it increase it every time you click on the red or green button.
The other, more important number is given in bold font and that is your game score. You will get a point for every detection that someone else has verified in the same way as you did—i.e. if you approve a detection that already has been approved by another person, or if someone else rejects a detection that you have rejected as well (the order plays no role, both persons get a point when the image is similarly verified for the second time).
This point score is the basis of the leaderboard ranking. You will see the top 10 contributors and their scores so you can compare how you’re doing. If there haven’t been any matches between different people’s verifications yet, the leaderboard is empty. Keep submitting verifications and share the link to the task to get other people to do the same—that way, you’ll get a good head start and points will start landing on your account as others submit more verifications.