Mapping the Windows of New York
I’m a huge Windows of New York fan. After I moved to New York just over a year ago I made it a goal of mine to see every building José illustrated.
Last week I saw he posted 371 Broome, and I thought I just had to hack with this data and imagery some how.
I decided I would map out all the windows that had been posted, and in the marker popups, embed a google street view map that looks at the window. Happy Sunday, I guess? Anyways, this is how I did it.
First thing I needed to do was to get the data. Fortunately, the markup on the site is straightforward and I could write a simple scraper.
This was the first iteration –
Essentially, it makes an HTTP request to the website, parses the HTML, and pulls out the important information (image, address and neighborhood).
Note: I stored the results in redis, thinking if this were to ship we’d run a daily cron job to check if there is a new window, parse the page again and store the results in redis. For simplicity’s sake, I’ll skip that here.
First problem: we need the latitude and longitude to place the markers on the map. Fortunately, this isn’t a unique problem and there are a ton of services that do this for us (the process of taking an address and turning it into a lat/long is called geocoding). So, we need a geocoder.
Google’s got a really simple one, so I updated the scraper to utilize that.
Problem is, though, the addresses on the site look like this: “― 208 WEST 17TH ST. ―” and querying google’s geocoding service with a string like that will result in an error.
Instead, we need to strip out all non-ascii characters and also remove any whitespace around the address string. An updated gist is below:
Much better.
Since we now have the address string from the site, a formatted_address (which is part of the google maps geocoding service response; it’s the proper address string, i.e. “208 W 17th St, New York, NY 10011”) and a lat and lng, we can put the markers on a map!
First thing we need to do is to make the flask app. As you see in app.py below, it’s really straightforward — when folks request the root path, we create an instance of our Scraper class and scrape the page.
Next, we need to create an index.html page to show the user. As you can see, we take the global windows variable passed to the page’s context from flask and use that to loop over the windows and render them on the page. We also add a map.
Per the design, we also need to grab the inline styles that is the text color and background color of each window. We can use the cssutils Python package to do so. It’s updated in the scraper below.
We’re almost there! Check out what we’ve got –
Finally, we’ve got to add the markers and popups to the map. Let’s do that in two steps. First, we’ll add the markers.
Modify the inline Javascript to iterate over the marker objects and add them to the map, like so:
Woot! Even closer to the finished thing –
Alright, let’s add the popups. To embed a google street map, we need to embed an iframe with our API key, a location parameter with the latitude and longitude, and a few parameters for the UI.
The inline Javascript should be updated to this:
Basically, we create new popups for each marker and embed the Google street view as an iframe like mentioned above. We add some click listeners to the windows, so when clicking a window we open the associated popup and highlight the window (make the image and text opaque). We also focus the active window, ensuring it is scrolled in to view. When closing the popup, we unhighlight the window.
As you can see, it actually works decently well! I’ve noticed a few shortcomings though –
- The Google street view images are occasionally outdated and the window illustrations are for newer buildings.
- Embedding a street view requires a lat/long, but there’s no way to specify an inside or outside picture. Occasionally we see a street view image that is inside of a store.
- Sometimes the street view images point to the wrong building and you need to move it around to see the proper window.
Anyways, it was a fun little project to hack on last Sunday evening. I’m not going to do anything with it, but tweeted about it yesterday and thought I’d share my process on how it was built.
High five.
