<scratches chin> -- Homer: [drunk] Look, the thing about my family is there's five of us. Marge, Bart, Girl Bart, the one who doesn't talk, and the fat guy. How I loathe him. ---
http://vore.cc/post/32503374905/old-maps-vs-new-maps
Old Maps vs. New Maps
Via Gruber comes a link to some interesting research by Jason Matheson that—on the surface—shows how poor the data provided by iOS 6's Maps app for the Canadian province of Ontario.
This caught my attention for two reasons; the first is that I also live in Ontario, and my experience with the new Maps has, so far, been generally quite good. The second its that Matheson uses a reasonably objective test to see how good (or bad) Maps actually is at finding places in Ontario, which opens up a few interesting possibilities.
The highlights
This is a really long post, although it was fun to write, so here are the important points upfront:
Google data isn't much better than Apple data, at least as far as this test is concerned
Google always returns a result, while Apple only seems to return results when they are a precise match, possibly explaining why the latter's data appears to be less complete
This test doesn't really tell us anything about how accurate the maps are, however, both because of the small data set, and because of the test's nature
Background
Matheson used a geolocation class in iOS's Core Location framework that can be used to convert between terrestrial coordinates and place names. You can either give it the coordinates of a location on Earth and get back a number of known locations within a certain range of it, or provide a location name and get a list of coordinate pairs that match that name.
Under iOS 5.x, the Core Location framework piggybacks on Google's own APIs, but, in iOS 6, all the data is now provided by Apple.
Matheson's experiment shows that Apple fails to properly identify the vast majority of city names in Ontario. That is, indeed, a pretty big deal, but I want to point out that it doesn't necessarily mean that "these towns don't even appear on the map" anymore than not finding someone on a phone directory means that those people don't exist.
The difference is significant, because this experiment doesn't test the accuracy of the maps themselves, but, rather, of the metadata associated with them.
That's not to say that the information is unimportant, of course—if you need directions to someplace and Maps can't find it or sends you to Australia, the whole thing becomes utterly useless.
What about Google?
What struck me (and Gruber) about Matheson's work is the fact that he didn't take his research to the ultimate conclusion. He compares the results he receives from iOS 6 against known geocoded points, but—crucially—not against iOS 5, which relies on Google.
To me, that's the crucial test: If everyone is unhappy about the transition from Google- to Apple-provided data, this is a perfect opportunity to test how good both are based on a given sample.
Since I had an iOS 5 device (an iPhone 4) handy, I decided to give it a go and scanned through the same list of city names that Matheson used, running each as a query against the Google-powered iOS 5.1 APIs.
The results, available here as an Excel sheet, are, indeed quite interesting. Here are some highlights:
Unlike Apple, Google almost always returns at least some results. This is in line with my experience with all Google search properties, and is likely due to the fact that Google has discovered that an incorrect result is better than no result at all when it comes to keeping people on their Websites and continue to generate ad revenue.
Despite this fact, however, 784 of these results are outside Ontario (most, in fact, are outside of Canada—some as far away as China). This compares favourably to the 688 searches that returned no results in Matheson's iOS 6 experiment.
What about the others? Of the 2,000-odd queries run, about 900 of them returned results that were within 1km of the known geolocation that I was searching for. Some others were off by hundreds of clicks. It's hard for me to compare this number to Matheson's (it depends on his definition of "close enough"), but hopefully he will chime in at some point.
(Note—my calculations may be off here; my Excel skills are beyond poor, so it's entirely possible that my spherical law formula is incorrect, or that I misaligned the known geolocation points. This, however, doesn't affect the conclusion of the last paragraph—Google has no idea where those 784 towns I was looking for are, and that's worse than what Apple can do.)
What does this mean? Not much more than the fact that reverse geolocation is hard, particularly when the information fed to the search engine lacks in accuracy.
But wait, there's more
With these results in hand, I was curious whether the accuracy of either search engines (or both) could be improved. What makes geolocation so hard is that humans—particularly when they share a common cultural background—are insufferable nostalgia machines, and tend to name places the same way. There are dozens of Londons, Glasgows, Hamiltons, and even a handful of Torontos in North America, for example. By the same token, Ontario can be a city or a province—and different cultures will make the distinction between a city and a province in different ways.
All this conspires to making reverse geolookups easy to get wrong, which is why providing more information often helps search engines find places more easily.
To test this theory, I simply added the word "Canada" at the end of each of the queries, in the hopes that this would provide additional context to help narrow down the searches.
And I wasn't disappointed—but only as far as iOS 5 is concerned. The Google-powered API returns (Excel file) an 85 percent match at the provincial level (that is, it recognizes that a place is in Ontario 85 times out of a hundred), although still only 900 or so places are returned as being within 1km or so of the expected location.
On the surface, iOS 6 (Excel file) fares much worse, with only 65 percent of matches being reported in Ontario, and a large number of searches—a whopping 37 percent—returning no results at all.
This, however, doesn't tell the whole story. If we look at the standard deviation in the sets of results that both engines identify as being in Ontario, there's a very juicy bit of information to be had: Google's is 221, while Apple's is only 78—and the Apple results are top-heavy, with only a very tiny minority of locations reported as being far off the intended mark.
This is very important, because it confirms what I mentioned above: That Google will return a result, any result, no matter how far fetched. Thus, some of the locations it returns are hundreds or thousands of kilometers off target.
Apple, on the other hand, seems to be going for accuracy: either it returns a result that is spot-on, or it returns nothing.
Conclusions
As far as I can see, the data supports three conclusions:
Given our set of data, old Maps doesn't fare that much better than new Maps.
There seems to be a significant difference in the way the two companies approach the task of returning search results, with Google doing whatever it takes to get any result out, while Apple seems to prefer accuracy above all.
In the end, this is not really a particularly useful test insofar as determining the accuracy of Maps. At best, we get to see how good Apple is as at finding things, but with targets as big as whole towns we're unlikely to unearth any information that is really useful.
And this, more than anything else, outlines Apple's biggest hurdle: people keep finding mistakes in Maps because looking for them has almost become a sport. When was the last time that you saw users analyzing Google Maps with this ridiculous level of detail? Had Apple set the public's expectations better, we may be having a very different discussion at this point.
NOTE: This post hasn't been peer reviewed—if you spot something wrong, drop me a line on Twitter. Thanks!
| Reply via web post | Reply to sender | Reply to group | Start a New Topic | Messages in this topic (1) |
