Over the past few months I’ve been working on mapsnap, a program to automatically georeference old insurance maps. This post explains why these maps are interesting, how mapsnap works, and why this is worth doing.

I’m interested in history, cities, and maps, and in the United States that combination leads you very quickly to the Sanborn Insurance Maps. The Sanborn Map Company produced hundreds of thousands of detailed, block-by-block maps of every city in the United States from roughly 1870-1960. Their original goal was to help insurance assessment (Is this building made of brick or wood? Where are the water mains?) but today they’re valued for the unique view they provide into the history of urban spaces before the major urban renewal projects of the mid-20th century. You can read more about Sanborn maps at the Library of Congress, or watch this six-minute explainer video.

Here’s an example of one page from a Sanborn map (NY 1923 vol 1 p27):

You can zoom in to see more details:

There’s a lot of information here!

• The color shows what material the building is made out of: red is brick, yellow is wood frame.
• The numbers in front of each building (46, 48, 50, 52, etc.) are its address. This is a gold mine for sites like OldSF and OldNYC where you want to locate a photo using a historic address, which might not be the same as the current address.
• Businesses in a building are labeled, e.g. the Bakery at 50 Oak Street or the Italian Marionette Theatre (sounds fun!). “S.D.” means “Store and Dwelling,” i.e. a store on the ground floor with residential units above.
• The heights of the buildings are labeled. Reading right to left, there’s 50’, 40’, 40’, 40’, 44’, 66’.

As it turns out, the NYPL (and OldNYC) has a photo of this block! If you read the heights carefully, you can match them up to the buildings in the photo.

The street is alive with activity in the summer of 1933. It’s filled with pushcarts and vendors selling their goods under awnings. It’s a good thing this photo was taken, because this block no longer exists. It was demolished in 1950 to make way for the Alfred E. Smith Houses project.

I knew that there were large-scale “slum clearance” programs in the mid-20th century, but the sheer scale of the destruction is so much more striking when you see what was there before.

Accessing the Sanborn Maps

The Sanborn maps were very expensive, very limited-run books. Today, they’re in private collections and are still used in real estate and environmental law. But fortunately for us, many public libraries own Sanborn volumes as well. The NYPL has a collection of New York maps. And the Library of Congress has the largest collection of all. Since many of the Sanborn maps are old enough to have fallen out of copyright, the LoC has been able to scan around 400,000 of them and provide them for free online. No need to visit the library in person.

Generally you’re interested in a specific block, though, and finding all the maps of one particular block can be tedious. Google Maps has really raised our expectations about how easy it should be to find maps online.

To make Sanborn maps easier to use on a computer, the key step is to georeference them. This means aligning them with a modern map. This is a tedious but typically straightforward process. You find a point (maybe an intersection) on the Sanborn map, then find the same point on a web map. Two or three matched points establish the alignment. There’s a fabulous web site and community, OldInsuranceMaps.net (aka OIM), devoted entirely to georeferencing public-domain Sanborn maps.

When I learned about Sanborn maps and OIM, I georeferenced a few maps in my area of upstate New York and one in Brooklyn. I found it interesting at first but then increasingly tedious and time-consuming. As a software person, I started wondering: could a computer do this?

After a few months of going deeper on this problem than I ever intended, the answer is a qualified “yes.” It is, for the most part, possible to automaticallly georeference Sanborn maps. Some maps are harder than others and it doesn’t get everything right, but it generally does a good job.

My program to automatically georeference Sanborn maps is called mapsnap, and I’m excited to explain how it works!

Introducing mapsnap

We’re living in the era of AI and it’s natural to ask whether Claude or ChatGPT can just do this. I tried a few variations on this at first, from “here’s an image, find the transform” to “find the intersections in this image.” It didn’t work as well as I’d hoped, and ChatGPT at least would often try to write a Python program to do image processing, rather than just using its vision. I found that I was mentally tracing street labels to check whether its intersections were good. So why not just write a program to do that?

At its core, that’s how mapsnap works. It runs OCR over a Sanborn map to find the street labels. It uses those labels to find candidate intersections, and then it uses those intersections to generate a fit.

Let’s walk through those steps.

Step One: OCR

The first step is to detect street labels. I used EasyOCR for this. This is the only real “AI” in this project, and it’s pretty benign: EasyOCR is a text recognition model from 2020 that’s small and runs locally on your computer. I chose EasyOCR because it was, well, easy to set up, but it also performed well and was able to detect text at any angle, a key feature for maps where streets can run vertically or diagonally. I eventually came to appreciate that EasyOCR was very adaptable as well.

Here are the detections for a map in downtown Brooklyn:

Most of these are legitimate streets, though some (“BROOKLYN” and “BRIDGE”) are not.

For each detection, we get three things:

1. A street name.
2. A (rotated) rectangle containing that name.
3. A confidence score.

I don’t want to get into the weeds here, but this is very much not “vanilla” EasyOCR:

• mapsnap tries to figure out a good minimum street label size given the detections. Small text tends to be information like addresses and business names that we don’t need.
• I reworked the output decoder so that it could only emit real street names from a vocabulary of the city’s streets. Reads outside this vocabulary come out as low-confidence junk. The vocabulary can be pretty large: usually an entire county’s worth of roads is just fine.
• I retrained the top levels of the EasyOCR neural net to do a better job of recognizing the Sanborn font and ignore the noise that often accompanies street labels on these maps.

When I say “I” here, I really mean “Claude and I.”

Step Two: RANSAC on Extrapolated Intersections

Each street detection gives us quite a bit of information about the page. While it’s possible to do georeferencing directly from the streets (more on this in a future post), in practice it’s easier to work with intersections, just like the humans do on OldInsuranceMaps.

To get an intersection, we need two streets that aren’t parallel to each other. We’ll assume the streets are straight and go in the direction of the label on the Sanborn map. (What if they’re not? More on that soon.)

The circles here are the extrapolated intersections. We can get the latitude and longitude of the intersection from OpenStreetMap (OSM). The incorrect street detections tend not to produce intersections that exist in OSM, which is a helpful filter.

These pixel + lat/lng pairs give us “Ground Control Points” or “GCPs” as they’re known. OIM wants three GCPs to produce a georeference, but in a pinch it will let you get away with two. Two GCPs work fine, so long as you’re willing to assume that the map isn’t skewed and has a uniform scale. The Sanborn maps are very well-made, and this is typically a safe assumption.

Extrapolating all the streets produces a list of candidate intersections. Each pair of these produces a georeference. We need to pick a pair or calculate some kind of average.

In practice some of these GCPs will be bogus. In fact, a lot of them might be. There are a few reasons this could happen:

• mapsnap misreads a street name or detects text that isn’t really a street. Maybe “POST” is “POST OFFICE” and not “POST STREET”.
• The streets might curve before they intersect.
• We might misinterpret the street. It might be ambiguous whether “4TH” is “4TH STREET NORTH” or “4TH STREET SOUTH.”
• We might not have detected the street angle very precisely.
• A divided street or a street that “jogs” across another has multiple intersection points in the real world, and we might choose the wrong one.

Whatever the reason, the GCPs aren’t all trustworthy. Averaging in a situation like this tends to produce poor fits: a mix of good and bad comes out mediocre.

In statistics, you can mitigate this by using a robust metric like the median rather than the mean. mapsnap uses a related technique from the 1980s called RANSAC. Here’s how it works:

• We try each pair of GCPs. Together, these georeference the map.
• We run all the street detections through that model and see how far away they are from the real street in OSM, both in distance and angle.
• If they’re close, we’ve got an “inlier.” If they’re far apart, we’ve got an “outlier.”
• We choose the pair of GCPs that produces the most inliers.

In the intersections image above, the two GCPs we choose are blue (PIERREPONT x HENRY and PIERREPONT x CLINTON). The ones we didn’t choose are red and yellow. The street labels that are “inliers” under this model are yellow (HENRY, MONROE, CLINTON, PIERREPONT, etc.) and the ones that are outliers are gray (POST, BRIDGE, EAGLE). These outliers are mostly bad reads.

This pair of GCPs produces an excellent fit, good enough that you can line up individual buildings across the nearly hundred year gap between the old and new map:

This isn’t textbook RANSAC, but it’s in the same general spirit. The beauty of this system is that it can tolerate a lot of noise (>50%!) so long as the noise isn’t self-consistent. When there’s even a nugget of signal, RANSAC does a pretty good job of finding it.

Where this fails

Street OCR and RANSAC work well when there are street labels, when those labels are clear and unambiguous, when the streets are straight, and when they haven’t changed since the Sanborn map was made. That’s often the case but not always:

• EasyOCR has trouble with short street names like “1ST” or “E.”
• It sometimes has trouble picking up cardinal prefixes like “N 4TH ST” vs. “S 4TH ST” or stitching together multi-word street names.
• When streets curve, as in hilly or suburban areas, extrapolating the labels to find intersections doesn’t work.
• When there aren’t many streets (say in a waterfront area or in a rail yard), mapsnap has nothing to latch onto.
• When the streets have been renamed (looking at you, Queens) or changed due to highway building or other urban renewal projects, this approach doesn’t work at all.

My goal is to georeference at least 90% of pages in the Library of Congress’s Sanborn collection without making major mistakes. To do that, mapsnap has to handle at least some of these tricky cases. The next posts will look at some of the other information Sanborn maps give us to aid in georeferencing. If you can’t wait for those, or you want to run it yourself, check out the mapsnap repo and the full LLM-generated How it Works page.