GPS or the Global Positioning System is a global navigation system that provides geolocation and time information under the condition that there is an unobstructed line of sight to four or more GPS satellites.
Normally you use GPS when capturing images for Mapillary in order to assign location information to the images. Some capturing devices have GPS built in (like most smartphones and the Garmin Virb action camera), while others require that you record the GPS track separately with an external device (which can also be your phone via a GPS tracker app). In the latter case, you use certain tools (computer programs or scripts) to write the GPS information from the recorded track to the images you captured at the same time.
The locations of the Mapillary images on the map are only as accurate as the GPS of the device that was used (read more about Mapillary accuracy). And you won’t always be capturing in conditions where the “unobstructed line of sight to GPS satellites” requirement is fulfilled.
That explains why sometimes the green line on the map looks off compared to road or street you were on. Sometimes, especially in the midst of tall buildings or trees, you might have trouble achieving a GPS fix altogether. Try to move towards more open areas in that case. Sometimes it also helps to switch the GPS off and on again or restart your device.