Sunday, March 18, 2012

Custom Map Tiles (Part 3 - TileStache)


In my previous post all the map tiles were pre-rendered and stored in the filesystem. Although that approach brings more flexibility it also has some disadvantages: storage and time.

Imagine that we want to store tiles for every zoom level covering the whole world. We would be talking probably about Pentabytes of information. Even if we just stored all the zoom levels for some countries it would be probably lots of Terabytes. Space is cheap, but we're talking about a whole new level here.

Probably even worse than the storage, it would take ages to render all of the required tiles. Without enough computational power we could be looking at months (or more) to render all the tiles. 

So, in this post, I'm going to present a realistic alternative: using a Tile Server to generate and cache map tiles on-the-fly. I'm going to use TileStache for this.

I like to consider TileStache as the spiritual successor of TileCache, probably the most famous tile server around. Anyway, I prefer TileStache because it's easier to use and more modern, supporting for instance the MBTiles/UTFGrid specs that I've shown previously.

So, in this post I'll:
    • Install Python 2.7
    • Install Mapnik
    • Install TileStache
    • Configure the server
    • Replace the URLs in the client (from my previous post)
    • Try it out

Install Python 2.7

Install the 32 bit version of Python 2.7. Although you may install other versions I strongly advise against it. Direct link to the installer here.


Installing  mapnik:
TileMill uses Mapnik to render the map images, and we'll also use it with TileStache.

To install Mapnik 2.0 in Windows just follow this post to the letter. It includes a direct link to the binaries and installation steps.

The last step of the installation is to try out the map generation. The following output should be generated.


If this is what you see, all is well. Otherwise, Houston we have a problem.

PS: Don't forget to add all the environment variables/paths described in the page. In my case I've added:

PATH
c:\mapnik-2.0.1rc0\lib;
C:\Python27;
C:\Python27\Scripts

PYTHONPATH
c:\mapnik-2.0.1rc0\python\2.7\site-packages


Installing  TileStache
We could do this the hard way or easy way. I'll opt for the easy way.

Download and install Python Setup tools from here:
This includes a tool called "easy_install". We'll eventually run:
easy_install tilestache
But we need to previously install the TileStache dependencies. According to the "easy install" package page for TileStache it depends on ModestMaps, PIL and SimpleJSON so a :
easy_install PIL
easy_install ModestMaps
easy_install SimpleJSON
should do the trick.


UPDATE: 08/07/2012
I forgot to mention a dependency on Werkzeug. It's a web-server included with TileStache that's required for this example. Thanks Bashton :)




After installing TileStache is run using the "tilestache-server.py" script, inside the scripts folder. Run it in the TileStache main folder using a command window.





Then open a browser window on "http://localhost:8080" and it should display a welcome message.




TileStache by default searches for a "tilestache.cfg" file. The default one has a "osm" layer defined as a proxy for the OpenStreetMap tiles.
"osm":
{
    "provider": {"name": "proxy", "provider": "OPENSTREETMAP"}
},



If we place a url on the browser as:
"http://localhost:8080/osm/0/0/0.png" a tile with the whole world should be displayed.

Remember, we're not generating anything (yet). We're just using a proxy to OpenStreetMap own tiles.


Now we want to generate our own tiles. As I said previously we'll render the tiles using Mapnik and thus create a new .cfg file named "Test.cfg" with the following content:
{
  "layers": 
  {
    "freguesias":
    {
        "provider": {"name": "mapnik", "mapfile": "C:\\tilestachedemo\\style.xml"}
    }
  }
}

The layer name will be used on the url to fetch the tiles. I've named it "freguesias" because of my previous post but name it as you please.

Couldn't get any simpler than this. The map info itself will be provided by the style.xml file used by Mapnik. To create the style.xml file we could read the whole spec and write the file by hand, which could be tedious. Fortunately TileMill allows use to export the Carto language to a working Mapnik XML.

I'll export the project from my previous post using the "Mapnik XML" option from the top right menu of TileMill. Create a "style.xml" file with this content in some folder (the same referenced from the .cfg file).

Now, let's run the server in the folder of our "test.cfg" like:






Now I'll change the javascript from my previous post so that the tilelayers are directed to TileStache.

The tiles property will thus be modified from:
tiles: ['http://localhost/tiles/freguesias/1.0.0/Freguesias/{z}/{x}/{y}.png']
to
tiles: ['http://localhost:8080/freguesias/{z}/{x}/{y}.png']
If we now open the webpage the output should be the same as before. The main difference is that the tiles are now generated on the fly, and we're not limited to the zoom levels that we've exported previously. The disadvantage is that it takes some time to render each tile and, although the browser might cache them, each new client will make the tilestache server compute tiles. So, it's essential that we cache the tiles at the server. TileStache supports many different type of caches but I'll use one of the simplest: "Disk".

Inside the .cfg let's add a cache section (before "layers")
"cache":
  {
    "name": "Disk",
    "path": "file://C:/temp/cache"
  },
Pretty simple, the cache will be at the filesystem. It will be pretty easy to check if it's working or not. Before starting we can create the "c:\temp\cache folder. Then, while we navigate the map, the tiles will be computed and stored in the filesystem at the path that we supplied. For example:

Now, we're just missing something that we had previously: support for UTFGrids. TileStache supports them and we just have to create a specific section for it. The full .cfg will be:
{
  "cache":
  {
    "name": "Disk",
    "path": "file://C:/temp/cache"
  },

  "layers": 
  {
    "freguesias":
    {
        "provider": {"name": "mapnik", "mapfile": "C:\\tilestachedemo\\style.xml"}
    },

    "freguesias_utfgrid":
    {
        "provider": {
            "class": "TileStache.Goodies.Providers.MapnikGrid:Provider",
            "kwargs":
            {
                "mapfile": "C:\\tilestachedemo\\style.xml",
                "fields":["DICOFRE", "FREGUESIA"],
                "layer_index": 1,
                "scale": 4
            }
        }
    }
  }
}
Now, change the map client so that the initialization is:
var tilejson = {
        tilejson: '1.0.0',
        tiles: ['http://localhost:8080/freguesias/{z}/{x}/{y}.png']
        grids: ['http://localhost:8080/freguesias_utfgrid/{z}/{x}/{y}.json'],
        formatter: function (options, data) { return "CODE: " + data.DICOFRE }

    map = new L.Map('map');
    map.addLayer(new wax.leaf.connector(tilejson));
    map.setView(new L.LatLng(39.5,-5.0), 6);

    wax.leaf.interaction(map, tilejson);

Lets try the demo again. The code of the parish should appear on the top left corner when the mouse is over the layer.


For now this post will end this series on Custom Map Tiles. I'll probably revisit this topic later to show how to create tiles programatically using C# and GDI+ without resorting to any tool.

13 comments:

  1. Do you have an online demo of the leaflet map? I can't seem to get this to work, especially with jsonp.

    ReplyDelete
  2. I don't have this demo on-line. I've got another from a different blog post:
    http://build-failed.blogspot.com.es/2012/04/maps-and-boardgames-part-2-using-server.html

    The direct link for the demo is:
    http://psousa.net/demos/maps-and-boardgames-part-2/demo3.html

    It's also using Leaflet and Wax, so should be similar enough.

    ReplyDelete
  3. Great tutorials. I'm a .NET developer working with the range of open-source mapping tools out there. Have been trying to get into TileMill, etc. lately but having continual technical frustration getting it all running under Windows/IIS.

    Your tutorials have been a great help.

    ReplyDelete
  4. Hi Pedro,

    Really good tutorial. However I've encountered a slight problem. When executing the tilestache-server.py file (from c:/Python27/scripts) I get a whole heap errors (see below). It would be greatly appreciated if you could help me diagnose the problem. Thanks in advance.

    Traceback (most recent call last):
    File "C:\Python27\Scripts\tilestache-server.py", line 5, in
    pkg_resources.run_script('tilestache==1.38.0', 'tilestache-server.py')
    File "C:\Python27\lib\site-packages\pkg_resources.py", line 489, in run_script

    self.require(requires)[0].run_script(script_name, ns)
    File "C:\Python27\lib\site-packages\pkg_resources.py", line 1207, in run_scrip
    t
    execfile(script_filename, namespace, namespace)
    File "c:\python27\lib\site-packages\tilestache-1.38.0-py2.7.egg\EGG-INFO\scrip
    ts\tilestache-server.py", line 48, in
    from werkzeug.serving import run_simple
    ImportError: No module named werkzeug.serving

    ReplyDelete
    Replies
    1. Hey bashton, I'm getting the same error as you. I went back and did easy_install Werkzeug but still getting the above error - how did you go about it?

      Delete
    2. Ando,

      I'm going to update the post with the missing steps.

      Delete
    3. Thanks Pedro - Great article. My issue turned out to be a dodgy style.cfg file after I rectified the Werkzeug issue mentioned by Bashton. Cheers

      Delete
  5. Ok. I figured it out. After reading the TileStache documentation it seems werkzeug (http://werkzeug.pocoo.org/) is also an optional dependency.

    All working now!

    ReplyDelete
  6. You're 100% right. Werkzeug is totally required as it's the web server used by default by TileStache. I forgot this step on the post... Sorry for the misinformation :)

    Anyway, thanks for the feedback, I'm going to update the post

    ReplyDelete
  7. This tutorial is very good, but there are a lot of information missing.

    Everything is installed and I had a lot of weird error when executing
    1. "C:\Python27\Scripts>python tilestache-server.py"
    Result: asked me for a config file

    2. I added one:
    python tilestache-server.py -c C:\TileStache-1.48.2\scripts\test.cfg
    Result:

    Error loading Tilestache config:
    Traceback (most recent call last):
    File "tilestache-server.py", line 5, in
    pkg_resources.run_script('tilestache==1.48.2', 'tilestache-server.py')
    File "c:\Python27\lib\site-packages\pkg_resources.py", line 489, in run_script
    self.require(requires)[0].run_script(script_name, ns)
    File "c:\Python27\lib\site-packages\pkg_resources.py", line 1207, in run_script
    execfile(script_filename, namespace, namespace)
    File "c:\python27\lib\site-packages\tilestache-1.48.2-py2.7.egg\EGG-INFO\scripts\tilestache-server.py", line 55, in
    app = TileStache.WSGITileServer(config=options.file, autoreload=True)
    File "c:\Python27\lib\site-packages\tilestache-1.48.2-py2.7.egg\TileStache\__init__.py", line 343, in __init__
    self.config = parseConfigfile(config)
    File "c:\Python27\lib\site-packages\tilestache-1.48.2-py2.7.egg\TileStache\__init__.py", line 95, in parseConfigfile
    config_dict = json_load(urlopen(configpath))
    File "c:\Python27\lib\urllib.py", line 84, in urlopen
    return opener.open(url)
    File "c:\Python27\lib\urllib.py", line 202, in open
    return self.open_unknown(fullurl, data)
    File "c:\Python27\lib\urllib.py", line 214, in open_unknown
    raise IOError, ('url error', 'unknown url type', type)
    IOError: [Errno url error] unknown url type: 'c'ache-1.48.2\scripts\test.


    Not very pretty.
    I looked in tilestache-server.py and this is what I found to correct it.

    1.1: In C:\python27\tilestache-server.py, it calls another tilestache-server.py, but this time in "C:\Python27\Lib\site-packages\tilestache-1.48.2-py2.7.egg\EGG-INFO\scripts"

    1.2 In this second tilestache-server.py, there are a couple of useful comments, such as: "By default the script looks for a config file named tilestache.cfg in the current directory and then serves tiles on http://127.0.0.1:8080/."

    Solution:
    Step1: The file you have downloaded to install TileCache with easy_install, unzip it, inside there is a "tilestache.cfg"

    Step2: Move this file in "C:\Python27\Scripts", it is where "tilestache-server.py" is

    Step3: In cmd, "C:\Python27\Scripts>python tilestache-server.py"

    Now you can go in your browser and try: http://localhost:8080

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Hi Pedro,
    thanks for your tutorials, but i still have some problems.
    my *.cfg file is "
    "example":
    {
    "provider": {"name": "mapnik", "mapfile": "/home/******/osm/style/test.xml"}
    } ,
    but i find it cannot connect the postgresql,the renderd tile is just a blue background,no other things.
    Thanks!

    ReplyDelete