Well, we’ve sent in our Citytracking grant report to the Knight News Challenge team. I’m still working on a write-up of the last few months of Citytracking activity (since my last update post), and we’re planning to publish the actual grant report for other people to read on the blog. But! While you wait patiently, here’s a nice photo of a giant map out in the world.
The photo was sent to us by Martin von Wyss, in Victoria, Australia. We were proud to display a large (upside down) watercolor map of London in the map gallery at GeoNext, the conference in Australia for location-based technology & business:
State of the cartography
Once maps were special. They were collected, copied, encrypted, used to plan wars and proved wisdom and influence. The map was the muse for artists like Johannes Vermeer and Jasper Johns who made them the focus of paintings.
Now maps are commonplace and banal. As publishers fade away, map shops close and road atlases are replaced by GPS directions, do maps continue to fascinate artists?
A new exhibit of recent works puts the art back into cartography and coincides with the second GeoNext conference. Maps of big data, novel interpretations, emotions and ideas remind us that it isn’t the process or media that matter when a big, stationery, canvas is required.
Following up on our April post, here’s another round-up of Watercolor, Toner and Terrain out there on teh internets.
Maps on Things
Today we launched Watercolor New York, the first print in a series of watercolor cities on Jen Bekman’s 20×200 site. We’ll be releasing more cities over the coming months, so keep an eye out. It’s a thrill for Stamen to be amongst the many artists already represented on 20×200, and lovely to see the watercolors in 30×40 inches of high resolution glory!
Along this theme, an enterprising chap named Sergey has taken the time to assemble map tiles into 3 posters of New York, London and Amsterdam, and popped them up as PNGs, PDFs or to buy as a print on a service called Red Bubble. Here’s to your first million, Sergey!
Ian Johnson painted some white Vans sneakers with Toner cities significant to him. (When can I get a pair, Ian?)
Riding on the Satellite Eyes wave of attention, maps popped up around the place on people’s desktops, and in the news…
Information about Cities
“”well, somebody press the Easy button and let’s ship it! made with @LeafletJS and @stamen in 5 minutes.”
While it’s not exactly clear from his picture what he’s shipping—I suspect something beer-related—it’s fantastic to hear how easy it is to get up and running with our maps. There are more and more projects that are using CityTracking to share information.
Danny & Tom from Leeds Art Map have used Toner on their map of art and cultural happenings in Leeds. There’s an interesting echo with CityTracking in their project’s aims: “The project draws inspiration from Jacques Rancière’s notion of equality of intelligence. Rancière rejects the conventional relationship between master and student, specifically how this manifests as an ‘active teacher – passive receiver’ dynamic. This relationship, according to Rancière, encourages a superiority of those in the know over those who are not and need emancipating, thus perpetuating inequality. Instead, ordinary people should always have an assumption of intelligence, therefore fostering a relationship between bodies that initiates from a point of equality.”
Jack Reed has built a site that displays any and all health inspections for Atlanta restaurants on a Toner map. The service connects directly into the Georgia Department of Public Health, so viewers can see inspections for restaurants in their area. A simple map interface makes the GDPH information more available than it was before.
“I read a quote now lost in the mists of Twitter that not knowing how to program in the 21st century is like living in the Middle Ages and not knowing how to swing a sword. I think that analogy goes extra for journalists.”
We were chuffed at his lovely “tribute to Stamen’s amazing watercolor map tiles and great open-source tools such as Polymaps that make creating custom interactive maps a pleasure,” Ansel Adams in L.A...
There are also individuals within big organizations and government beginning to use the maps, which is super exciting. Last week, Code for America announced a new initiative called 311 Labs, a platform to experiment with 311 data and communicate with each other about their work. One of the showcase projects, The Daily Brief, displays 311 tickets on Toner…
Amongst its myriad mandates, the Federal Communications Commission (FCC) also registers antenna structures across the USA. You can see the location of the antennas being registered on either Toner, Terrain, or a couple of other maps styles not in CityTracking, called Nightvision and Light.
The studio was thrilled when we found out that venerable Sanborn mapping company, founded in 1866, had chosen the Terrain tileset on a recent project (even though the project was in response to a sad event). Sanborn’s local Denver community was devastated by the recent Waldo Canyon Fire, and Sanborn employees responded by creating a stunning map of aerial photography of the fire damage, and a Colorado-wide map of the state’s susceptibility to wildfires.
All in all, an amazing couple of months for CityTracking! Do let us know if you have other examples out there…
Visual changelog for Toner 2012:
Toner v2 (2012 and 2011) uses High Road for more sophisticated roads and tucks San Francisco’s punky park mohawk under the water:
There is a new “lite” version with toned-down contrast, better for printing out analog style or overlaying polygons client-side:
We added reservoirs!
And better about showing walking and biking paths through meadows and woods:
Back in the urban grid, we’ve added subways and building footprints:
Speaking of buildings, big ones get added as you zoom in, until you get to the most detailed zooms, where all buildings are drawn:
We cleaned up labels so they don’t overlap as much:
And added city labels worldwide:
And now draw 漢字 and other non-Latin scripts correctly:
Easy-to-use tiles: Stamen now hosts easy to embed Toner tiles with CC-BY-3.0 license from maps.stamen.com, thanks to the Knight Foundation and our Citytracking.org grant! No server hardware or software setup needed, just start using the tiles in your favorite web mapping API client side. You can still roll your own tiles using the data and setup readme’s in the Github repo. Read more »
More international: Plays better outside of the United States! Now displays local names in non-Latin writing scripts (like Japanese and Arabic) and better accent marks in Europe. We optimized the road symbology to more places world wide. Issue 30
More Toner flavors: Introduces specific flavors of Toner optimized for map sandwiches, easy to integrate with for your custom map stories: toner-standard (toner), toner-hybrid-with-labels (toner-hybrid), toner-hybrid-only-lines (toner-lines), toner-hybrid-only-labels (toner-labels), toner-no-labels (toner-background). Issue 10.
Easy to read stylesheets: General stylesheet cleanup, consolidation. Restructured all the OSM roads using High Roads. Now uses Postgres views by zoom level, making it much easier to design what big, medium, and small roads should look like consistently between layers while abstracting the data part. Similar approach is taken for water bodies using Imposm tables. Issue 9
More content: Added reservoirs, state boundaries, and more. Map now zooms to 19+, important when you’re inventory mapping stories at the city block level where locations along a street or buildings/venues are helpful. Before, Toner stopped at zoom 18, but often when you’re looking at street-level incidents (as in Dotspotting.org), you need more detail Issue 18. Along with that, the transition between bold black roads and cased white roads now starts at zoom 18 and carries thru to zoom 19+ (Issue 17). This preserves the strong contrast of Toner, but also allows better use as a background map visually at these zooms so your story points stays in focus (and consumes much less ink if you print the maps using a service like Stamen’s Field Papers.
Urban wayfinding: At these most detailed zooms building footprints help us orient to the built landscape so we start adding those progressively in starting around zoom 14 (big airport terminals and convention center sized buildings) but most noticeably at zoom 16+. We also show metro (subway) stations now, helping navigate by landmarks in big cities like New York, London, and Tokyo. Issue 16, Issue 48, Issue 40.
Improved graphic styling of country boundary lines at zooms 8 and 9. Issue 27
Added state boundary lines at the city and regional zooms. Important for places like Washington DC where a metropolitan area sprawls across multiple admin-1 jurisdictions. Made sure they stack above the water and made upstream changes in OSM master data to allow for boundaries in the water that aren’t indicator level to be not shown in Toner when using newest OSM Issue 11, Issue 24, Issue 7, Issue 6, and Issue 50.
Added support for international Unicode (UTF-8) labels from OSM by re-authoring fonts. Primarily seen in street labels and park names. Issue 30
Removed map label overlap by manually adjusting the Dymo output around other map features like bodies of waters, country labels, and state labels. Issue 34, Issue 35. Version 3 will address remaining occasional placement funk and overlap of marine labels.
Easier to read street labels at zooms 17+. Issue 25
Parks are now tucked under the water in the street-level maps. This is a OSM pecularity where some parks are mapped to the shoreline and others extend out into the water. As these are black-and-white maps, we take a shortcut by making a transparent pattern with the black stipples. When it’s over the water, the black park is still drawn, but the water is also black so win-win. Issue 12
Added reservoirs to the “inland water” aka “lakes” symbolization. Removed smaller lakes at zooms 8 to 12. Since the water is solid black, these tiny lakes attracted undue attention. Instead, they are now progressively added on each zoom in. This reduces visual noise in the map. Issue 23, Issue 45, Issue 39
For lake labels, similar progressive approach but with a slight zoom delay. Issue 44
Tunnel street labels are now grey to match their grey linework. Issue 22
Added subway icons at zooms 18 and 19, helpful for city wayfinding. Issue 26
Added data import scripts to PostGIS, still rough.
Updated the Readme.md Issue 36
A beta version of Toner v2 was released in late 2011. The final v2 release mostly focuses on airport icons and making map labels more legibile (less overlap).
The first thing I thought after we hung a copy of London’s Kerning, a printed map showing only the street names in London, in the studio, was: I want one of those for the rest of the world. “How hard can it be to just (people here love that) show the streets?”
Toner: just streets
Terrain: just streets
Terrain: just labels
Terrain: just background
We’ll follow up with some more detail about how to incorporate these into existing projects soon—many of the styles have transparent backgrounds, for example, so they can be used as layers on other maps—but for now: enjoy!
We’ve got some new additions to Toner, the black and white style that Geraldine started and that Nathaniel and Mike have been gradually improving this year. There are some fairly significant changes to the cartography stack all the way through, which you can read about in detail at the project’s visual changelog on GitHub. And of course everything’s open source and available for download as per the terms of the Citytracking grant.
We promised to do this work in public, so here goes. One significant thing we’ve decided to do is to keep older versions of the project around, so that we (and, hopefully, you) can compare the different versions of the maps. So when Nathaniel talks in his post about “shaving San Francisco’s Mohawk” from how it looked in 2010:
to how it looks in 2012, with a better coastline:
you can see it in situ. It’s pretty simple to change the urls:
The changes can be fairly visually dramatic, as in the addition of non-Roman scripts to places like Tokyo:
The thing about designing maps is that you’re never designing just one view. For one thing, it’s important to account for all the different zoom levels: it’s about showing more as you zoom in, but it’s also about showing different things at different scales. Choices need to be made at every level about the thicknesses of streets, which buildings to show, which city name to show, and so forth. Different places have different characteristics spatially; some are more dense than others, and you have to keep the whole system in mind. These two versions of the zoom into DC, from different years, gives a sense of the range of choices involved:
I’m not aware of any other mapping projects that let you look back in time as a design evolves this way.
Tile Farm, our map rendering server, has undergone a quiet overhaul during the past three months to deal with wild swings in demand. In March, we released the worldwide Watercolor maps to bring the number of visual designs at maps.stamen.com to three. With great press and kind words comes unprecedented demand, so I’ve been working with Tethr’s Aaron Huslage to change the way we use geographic data to handle the load. We’ve learned a few things along the way that will be important features of Stamen’s future mapping work.
The general theme of our optimization work has been to speed up response times and shrink problems. Tiles were being rendered reliably most of the time, but the overall experience was sluggish and unresponsive. Sometimes, the watercolor tiles would fail altogether. System load was weirdly high, and we didn’t have a reliable way to understand what we were seeing.
Repair #1: Postgres and Imposm
Looking into the system, Aaron noticed that several common map feature queries were taking an unusually-long amount of time for Postgres to process, even after performing basic database tunings to use memory and cache more effectively. Our first significant move was to attempt to shrink the cost of those queries, typically for large features at low zoom levels.
We found queries for lakes, forests, riverbanks and similar features that introduced massive overhead due to the number and complexity of their shapes. After doing a bit of research into simplifying and filtering geometries, I remembered Oliver Tonnhofer’s Imposm mentioned by AJ Ashton in his 2011 State Of The Map presentation on Tile Mill. Imposm is an OpenStreetMap data importer and an alternative to the older Osm2pgsql, and in addition to solving our needs for simpler, fewer shapes through its GeneralizedTable feature, it also answered a bunch of questions we didn’t know we had yet.
The first unexpected bonus from using Imposm was that we could create special, custom tables for exactly the data we’d need for any particular selection of data. Instead of the resource-sucking Postgres views onto generic point/line/polygon tables offered by High Road, Imposm allows us to create a larger number of rendering-specific tables with no more than the exact data we’d need.
The second unexpected bonus is that Imposm is written completely in Python, so adding new post-processing steps to the data is trivial as it’s imported into the database. For example, we’re starting to get rid of long, confusing regular expressions for abbreviating street names and replacing them with procedural code that can be more easily customized and shared.
The nicest pleasant surprise from Imposm is that it’s actually much faster to run than Osm2pgsql, thanks to a concurrent design that divides the import into parallel processes.
Repair #2: More, smaller Gunicorns
Tile Farm uses a WSGI server called Gunicorn to host instances of TileStache. A herd of gunicorn processes are hidden behind the webserver Nginx, which protects them and delivers requests. This is a common arrangement, and in our first design for Tile Farm we had a single Gunicorn configuration with all of our TileStache settings. With a rush of visitors to the site upon launch and the unique design of the Watercolor layer, we had to perform a number of fast, targeted interventions to improve performance and keep up with sudden demand. Watercolor in particular presented some debugging challenges, and we found that it was important to test changes to it in isolation from the other styles.
After working with a separate Gunicorn configuration for just Watercolor on a different port, it was clearly going to be easier to extend the same setup to the remaining Toner and Terrain styles, each of which is actually built up from a number of sub-styles composited together after rendering in Mapnik. The TileStache configuration is now split, with one for Toner, one for Terrain, one for Watercolor and a few others. Instead of interfering with the entire service when performing maintenance we can now modify individual styles and lower the extensive cost of restarts. Unlike Apache processes, Gunicorns can often be hard to kill and they don’t go down easy, so it’s important to be able to target them more narrowly. The primary drawback to this approach is complexity: each Gunicorn/TileStache setup runs on a separate port, governed by a separate startup script with its own logfile and other reporting. Another drawback is hundreds of persistent connections from Mapnik to PostGIS left open at all times, which Aaron assures me is weird and wrong. Still, the ability to isolate problems and even permit limited crashiness in construction areas has been liberating.
Repair #3: The Gunicorn Slayer
I mentioned above that Watercolor has its own special snowflake problems. Processes rendering these maps would eventually become unresponsive, not replying to requests and not giving up their slots to be replaced by their parent Gunicorn server. We looked at a number of possible causes and fixed each: watercolor texture bitmaps are now loaded globally instead of per request, and we no longer use metatiles since the CPU cost of Watercolor is linear with bitmap size. Debugging this was confusing, then frustrating, then finally a waste of time. We settled on a more drastic solution and built the Gunicorn Slayer.
The Gunicorn Slayer is inspired by the Netflix Chaos Monkey, “a wild monkey with a weapon in your data center to randomly shoot down instances and chew through cables.” Admittedly less chaotic, ours borrows a strategy from Logan’s Run and retires any Gunicorn process older than a few hours.
This last fix still feels somewhat dirty since we were unable to determine the root cause of the problem, though if we were to revisit it all again an excellent post from Pinterest’s engineering blog gives some great hints on repurposing process titles and POSIX signals to gain some visibility into a running process.
Then And Now
Maintaining reliability for a project like maps.stamen.com is always going to be a long game of catch-up. Our goal has been to make the project slightly easier to ignore every day, by increasing reliability and finding and fixing trouble spots. The fix with the single largest impact has been a move from generic to bespoke OSM data through Imposm, followed closely by moving to a more-and-smaller approach with the rendering servers.
Nathaniel showcased nearly two years of City Tracking in Boston last Monday at the MIT/Knight Story and the Algorithm conference. You can read about this years’ winners here, as well about the changing nature of the grant: shorter cycles, more opportunities to apply, that kind of thing. It’s worth a look.
The City Tracking project started off with Dotspotting.org, which allows mapping of data spreadsheets. The project still sees active use, but after we made the Toner tiles available for download we started to notice that more people were expressing interest in the background maps than in the ability to put dots on them.
Responding to public feedback while the project happened was something we were interested in from the start, and http://maps.stamen.com is the result: a browsable, embeddable, and otherwise immediately usable map of the whole world that can be used in Google Maps, Modest Maps, and other mapping APIs without having to download all of OpenStreetMap or tinkering with servers and technical code.
Nathaniel closed with a project we launched in March showing how climate change can be made personal on the street level, instead of the usual course brush strokes, with Climate Central’s surging seas project.
These thin slices of big data are bite size morsels of aha. We hope you like them!
This morning when I got into work, I noticed a small uptick in the bug reports that we get via maps.stamen.com. Curiously, quite a few of them were centered around England. I suppose I had thought that we would have generated most of the English tiles by now, so could expect smooth sailing there. Then, someone gave the game away:
“A blue tile in the center of London that unfortunately shows up on my desktop background when using Satellite Eyes.”
Friend-of-Stamen, Tom Taylor, announced almost 24 hours ago that he had “made a thing and wrote a blog post about it.” That thing is Satellite Eyes, which changes your Mac desktop wallpaper to a map of where you are. Tom has incorporated Watercolor, Toner and Terrain (in the U.S.) as well as the lovely Bing Aerial map as options.
After many of us here at the Studio had installed the app, we began to notice that our tile farm was… well… smoking. Since Tom is English, we suspected that many other English people around England had also installed the app, and were happily playing with the preferences that let you switch between map styles and zoom levels. We took to the graphs:
Even though this is all Tom’s fault, it’s also a good thing! Far better to respond to actual demand than to try to optimize prematurely. So, we’ve increased capacity by spreading some of the some of the watercolor rendering load into EC2, and are working on re-creating those “underwater” tiles you might have noticed around the map.
Thanks, Tom! Excellent work!