Parsing UTFGrid data to polygons

I've talked previously about UTFGrid. It provides, as described by Development Seed, "a standard, scalable way of encoding data for hundreds or thousands of features alongside your map tiles".

So, to sum it up, if we have this image tile for the hexagons of one of my previous posts.

We would have a corresponding json file with this UTFGrid

         !!!!!!!!!!!!!!!!!###################$$$$$$$$$$$$$$%%
            !!!!!!!!!!!!!!!!####################$$$$$$$$$$$$$$%%
             !!!!!!!!!!!!!!######################$$$$$$$$$$$$%%%
             !!!!!!!!!!!!!!######################$$$$$$$$$$$$%%%
             &&&&&&&&&&&&&&######################''''''''''''%%%
            &&&&&&&&&&&&&&&&####################''''''''''''''%%
            &&&&&&&&&&&&&&&&&###################''''''''''''''%%
           &&&&&&&&&&&&&&&&&&##################''''''''''''''''%
           &&&&&&&&&&&&&&&&&&&#################'''''''''''''''''
          &&&&&&&&&&&&&&&&&&&&################''''''''''''''''''
         &&&&&&&&&&&&&&&&&&&&&&##############'''''''''''''''''''
         &&&&&&&&&&&&&&&&&&&&&&##############'''''''''''''''''''
        &&&&&&&&&&&&&&&&&&&&&&&&############''''''''''''''''''''
        &&&&&&&&&&&&&&&&&&&&&&&&############''''''''''''''''''''
        &&&&&&&&&&&&&&&&&&&&&&&&((((((((((((''''''''''''''''''''
         &&&&&&&&&&&&&&&&&&&&&&(((((((((((((('''''''''''''''''''
          &&&&&&&&&&&&&&&&&&&&&(((((((((((((((''''''''''''''''''
          &&&&&&&&&&&&&&&&&&&&((((((((((((((((''''''''''''''''''
           &&&&&&&&&&&&&&&&&&(((((((((((((((((('''''''''''''''')
           &&&&&&&&&&&&&&&&&&(((((((((((((((((('''''''''''''''')
            &&&&&&&&&&&&&&&&((((((((((((((((((((''''''''''''''))
             &&&&&&&&&&&&&&&((((((((((((((((((((('''''''''''''))
             &&&&&&&&&&&&&&(((((((((((((((((((((('''''''''''')))
             **&&&&&&&&&&&&(((((((((((((((((((((('''''''''''+)))
             ***************(((((((((((((((((((((+++++++++++++))
            ****************((((((((((((((((((((++++++++++++++))
           ******************((((((((((((((((((++++++++++++++++)
           ******************((((((((((((((((((++++++++++++++++)
          ********************((((((((((((((((++++++++++++++++++
          *********************(((((((((((((((++++++++++++++++++
         **********************((((((((((((((+++++++++++++++++++
         ***********************(((((((((((((+++++++++++++++++++
        ************************((((((((((((++++++++++++++++++++
        ************************(((((((((((,++++++++++++++++++++
         ***********************,,,,,,,,,,,,,+++++++++++++++++++
         **********************,,,,,,,,,,,,,,+++++++++++++++++++
          ********************,,,,,,,,,,,,,,,,++++++++++++++++++
          ********************,,,,,,,,,,,,,,,,++++++++++++++++++
           ******************,,,,,,,,,,,,,,,,,,++++++++++++++++-
            *****************,,,,,,,,,,,,,,,,,,,++++++++++++++--
            ****************,,,,,,,,,,,,,,,,,,,,++++++++++++++--
             **************,,,,,,,,,,,,,,,,,,,,,,++++++++++++---
             **************,,,,,,,,,,,,,,,,,,,,,,++++++++++++---
             ..............,,,,,,,,,,,,,,,,,,,,,,////////////---
            ................,,,,,,,,,,,,,,,,,,,,//////////////--
            .................,,,,,,,,,,,,,,,,,,,//////////////--
           ..................,,,,,,,,,,,,,,,,,,////////////////-
           ...................,,,,,,,,,,,,,,,,,/////////////////
          ....................,,,,,,,,,,,,,,,,//////////////////
         ......................,,,,,,,,,,,,,,///////////////////
         ......................,,,,,,,,,,,,,,///////////////////
        ........................,,,,,,,,,,,,////////////////////
        ........................,,,,,,,,,,,,////////////////////
        ........................000000000000////////////////////
         ......................00000000000000///////////////////
          .....................000000000000000//////////////////
          ....................0000000000000000//////////////////
           ...................00000000000000000////////////////1
           ..................000000000000000000////////////////1
            ................00000000000000000000//////////////11
            ................00000000000000000000//////////////11
             ..............0000000000000000000000////////////111
              .............0000000000000000000000//////////22111
                          00000000000000000000000222222222222111

Entity Framework 5 and Spatial Data (Part 2 - Comparison with NHibernate Spatial)

Part 1 - Introduction
Part 2 - Comparison with NHibernate Spatial

As promised in my previous post I'm going to do a comparison between NHibernate Spatial and Entity Framework.
 
I'll start by making a pros/cons list for both libraries, but just focusing on items that may differ between them.

 NHibernate Spatial

Pros:

• True abstraction from the database with IGeoAPI/NetTopologySuite
• Works with .NET 3.5 and newer

Cons:

• No longer actively developed
• Querying with LINQ/QueryOver doesn't work in many cases
• Some spatial operators are not implemented

Entity Framework 5 Beta 2 Spatial

Pros:

• Setup is a breeze
• Querying with LINQ works flawlessly.
• Natively integrated with Entity Framework

Cons:

• The DbGeography/DbGeometry types belong to System.Data.Spatial. Therefore, we're always tied to the database model, even if just conceptually.
• Requires .NET 4.5 (at least in the Beta 2. I don't know if support for 4.0 is on the road)

Entity Framework 5 and Spatial Data (Part 1 - Introduction)

Part 1 - Introduction
Part 2 - Comparison with NHibernate Spatial

I've blogged a lot on NHibernate Spatial. Although it works nicely, it seems to have been left for dead. It no longer compiles without modifying the source code and has many missing features and bugs.

On the other hand, Entity Framework is finally receiving native Spatial Support on version 5 (it's about time). Although it still is in Beta 2, I believe it's time to check it out and to see how it fancies itself. I'm going to create an example using it against a SQL Server database.

I'm going to use the following model:

• There are 18 districts, each defined by a polygon
• The are about 140.000 places, each defined by a geographic point

I've already used this model on a previous blog post.

I'll create two separate projects:

• Class library with the model
• Console application with some spatial operations

Maps and Boardgames (Part 3 - Client Side drawing + Optimizing)

Part 1 - Introduction
Part 2 - Using server tiles
Part 3 - Client side drawing

In my previous post I've shown how one may interact with an hexagon layer on top of a Leaflet map.  The particular thing in that example is that it used hexagons embedded in the server-side tile images (created using TileMill). This has some pros and cons:

Pros:
- great performance
- works on most browsers (even pre-HTML5)
- less "visual noise", as the map is rendered at the same time as the hexagons (heck, they're in the same image tiles)

Cons:
- by itself it's difficult to have client-side behavior. It's possible using something like the UTFGrid technology, but that brings some additional complexity to the table and some limitations (I've used this approach in my previous post).
- the hexagons are static. So, if for example, we want to generate the hexagons with a different size or color or change them dynamically we're kind of screwed.

In this post I'm going to try a different approach, and return to the HTML5 Canvas implementation that I've started in the first post of this series.

Maps and Boardgames (Part 2 - Using server tiles)

Part 1 - Introduction
Part 2 - Using server tiles
Part 3 - Client side drawing

In my previous post I've created an hexagon layer in client-side using HTML5 Canvas. In this post I'm going to generate the hexagons on the server and display them on a map. So, the roadmap for this post will be:

• Generate the hexagons  in C#
• Store them as spatial data in SQL Server 2008 using NHibernate Spatial
• Use QuantumGIS (or any other GIS tool for that mater) to read that data and tweak it 
• Use Tilemill to generate a layer of .mbtiles, including UTFGrid data of the hexagons
• Extract the .mbtiles file to image files
• Use the data on a Leaflet map.
• Create some basic interaction on the map

If you want you can skip to the end result.

Now, we've got two options for this. We can create hexagons that overlay perfectly on top of the map, all of them having the same pixel size, or all have the same edge size in meters, regardless of where they are on the world.

Maps and Boardgames (Part 1 - Introduction)

Part 1 - Introduction
Part 2 - Using server tiles
Part 3 - Client side drawing

Hi all.

If you have seen some of my posts you've probably suspected that I like spatial stuff :). I would like to make an experiment where I'm going to make some kind of boardgame engine on top of a map (probably Leaflet).

It will be a little bit useless but should be a fun exercise to apply some of the stuff that I've been talking about, like html5 canvas, tile maps, leaflet, UTFGrids, etc.

NHibernate Spatial (Part 4 - Querying)

Part 1 - Introduction
Part 2 - First Demo
Part 3 - Improving
Part 4 - Querying 
Part 5 - NHibernate 3.3.1+

In my previous NH Spatial post I've shown a concrete end-to-end example of how to use NHibernate Spatial on an ASP.NET MVC project.
Anyway, there's still much to be said about using this library and I've received some hits on my blog from people who were searching for how to do specific spatial operations with NH Spatial. So, this blog post will be pretty much practical, showing some spatial queries using NH Spatial on SQL Server 2008 R2, taking into account some variables in the process:

• Geography (always assuming SRID 4326)  and Geometry data types
• HQL and LINQ
• NHibernate version

Custom Map Tiles (Part 3 - TileStache)

Part 1 - MapCruncher
Part 2 - TileMill
Part 3 - TileStache
Part 4 - Programmatically
Part 5 - Offline maps on iOS

In my previous post all the map tiles were pre-rendered and stored in the filesystem. Although that approach brings more flexibility it also has some disadvantages: storage and time.

Imagine that we want to store tiles for every zoom level covering the whole world. We would be talking probably about Pentabytes of information. Even if we just stored all the zoom levels for some countries it would be probably lots of Terabytes. Space is cheap, but we're talking about a whole new level here.

Probably even worse than the storage, it would take ages to render all of the required tiles. Without enough computational power we could be looking at months (or more) to render all the tiles. 

So, in this post, I'm going to present a realistic alternative: using a Tile Server to generate and cache map tiles on-the-fly. I'm going to use TileStache for this.

Custom Map Tiles (Part 2 - TileMill)

Part 1 - MapCruncher
Part 2 - TileMill
Part 3 - TileStache
Part 4 - Programmatically
Part 5 - Offline maps on iOS

In my previous post I've shown a really simple way of overlaying custom map tiles on top of Bing Maps using MapCruncher.

One of the problems with that approach is that it only works for raster files. If one needs to do more complex stuff like creating tiles with data from shapefiles, kml or databases then a different method/tool must be used.

We have three challenges here:

1. How do we create a raster map representation of our raw data?
2. How do we split that image in tiles?
3. How do we serve those tiles to the map client (like Bing/Google maps)? 

I'm going to use a tool that simplifies this process without sacrificing any quality on the process: TileMill. TileMill is, like stated in their homepage, "an application for making beautiful maps". It uses a Mapnik core to render the maps, and provides a very simple UI and a CSS-like language to style them. I could praise the application a little bit more but I prefer to show you.

Custom Map Tiles (Part 1 - MapCruncher)

Part 1 - MapCruncher
Part 2 - TileMill
Part 3 - TileStache
Part 4 - Programmatically
Part 5 - Offline maps on iOS

When drawing overlays on top of a map one can choose between two different methods: custom map tiles or using the map API.

On this series of posts I'm going to show what are custom map tiles, how overlaying them compares to using the map API in client-side and finally how to interact with the tiles.

ASP.NET MVC 4 (Part 2 - Mobile Features)

In its newer release ASP.NET MVC has brought better support for mobile devices. How so?

ASP.NET MVC 4 (Part 1 - Web API)

Since ASP.NET MVC was launched it became my favorite framework for developing web applications. I believe that Microsoft pushed all the right buttons with it: really well conceived, extensible, modular and testable, with every iteration of ASP.NET MVC bringing lots of great improvements.

Now, MVC4 is in beta, and as before many new features have been introduced. The official whitepaper with the release notes may be read here.

One of the new features is called "ASP.NET Web API" which is, according to Microsoft description:

"(...) a new framework for creating HTTP services that can reach a broad range of clients including browsers and mobile devices. ASP.NET Web API is also an ideal platform for building RESTful services."

Visual Studio 11, .NET 4.5 and MVC 4

I've been wanting to try out the Developer preview of Visual Studio 11 and .NET 4.5 for some time. The problem is that new technologies and products have appeared/evolved at an incredible pace and it's difficult to keep oneself on track with all the novelties, specially when time has become such a limited asset.

Anyway, today is the day. It's .NET day! I've already installed:

• Visual Studio 11
• .NET 4.5 (C# 5.0)
• MVC 4.0
• SQL Server 2012

There are so many new things to try out, but I'm particularly interested in :

• the new C# 5.0 features, like the "async" and "await" keywords
• native WebSockets in .NET
• The WebAPI in MVC 4.0
• The new Mobile stuff of MVC 4.0
• Asynchronous controllers with Tasks and the async keyword (hell yeah)
• The new spatial features of SQL Server 2012
• The new distribution features, like packaging javascript files, minification(?), MSBuild 4.5 features
• etc, etc, etc

I'm not sure where to start, but I would like to do something that I haven't done before: besides doing regular tutorial posts I would like to accompany them with video-tutorials. Not exactly sure what will come out of this but, in the worst case scenario, should be a fun experiment :)

Anyway, gotta hurry, Visual Studio 11 expires in 127 days :P

Leaflet and HTML5

When one thinks of web map platforms Google Maps and Bing Maps are the two obvious choices. They're both free as long as you comply with some ground rules (i.e, terms of use) and don't exceed their usage limit. Unfortunately many applications can't adhere to those restrictions, leaving them with two options: purchase a commercial license or use a free alternative.

The most common free alternative is, without a doubt, OpenLayers. It provides an incredibly complete API (much more than Google and Bing) and integration with varied data sources (see examples here). The problem with OpenLayers, in my opinion, is it complexity. I'm also not particularly fond of the map control itself, but that's probably just personal taste.

Anyway, recently appeared a new kid on the block: Leaflet. It is a free lightweight mapping solution that provides a rich and extensible API, very much focused on HTML5. Also, it includes something that a lot of developers are asking for in Bing Maps V7: a client-side canvas layer. In this post I'm going to show how it works and develop a working prototype.

Update (21/01/2013): I've added a working example here

Bing Maps and HTML5 (Part 4 - Final optimization)

Part 1 - Basic
Part 2 - Canvas
Part 3 - Optimizing
Part 4 - Final Optimization

In Part 3 of this "Bing Maps and HTML5" series I optimized the drawing of the Canvas quite a bit. The time taken to draw the canvas with 140.000 points was cut from 800ms to 75ms.

Anyway, and as I said previously, my target was (and is) the 40ms mark (1000/25) for a full canvas render with all the points. Why? Because I want the data to be continuously drawn while the map is zoomed/panned, and it has to be fluid.
After profiling my code, the bottleneck is still the function where wgs84 coordinates are converted to pixel coordinates.

Update 26/07/2012: I've added a working example here. I've also added a zip file with the project here.
I modified the demo so that the data is not fetched from a node.js but from a separate js file instead.

NHibernate Spatial (Part 3)

Part 1 - Introduction
Part 2 - First Demo
Part 3 - Improving
Part 4 - Querying
Part 5 - NHibernate 3.3.1+

This is the third part of a series of posts regarding a simple example using NHibernate Spatial.


Actually, the title is kind of misleading as this post is just the wrap-up of the previous one, and I will not do much more regarding NHibernate Spatial (at least on this post). I'll just do the web page for the user interaction.

As I said, I'll create a simple map that, when clicked on  a specific point, paints the district that contains it. Not particularly useful but should be easy enough to understand and adapt to a more realistic scenario.

NHibernate Spatial (Part 2)

Part 1 - Introduction
Part 2 - First Demo
Part 3 - Improving
Part 4 - Querying
Part 5 - NHibernate 3.3.1+

Well, now that we've got a working NHibernate Spatial dll (from my previous post) let's put it to some good use.

I want to create a relatively simple example that uses spatial data with NH Spatial. For this tutorial I'm going to create a SQL Server 2008 database table with all of the Portuguese districts. Each district will have it's area stored in the database.

 There should be 18 polygons which fill the entire country and do not overlap (obviously).



I'll display a Bing map on a webpage. When the user clicks on a point of the map the district that contains that point will be fetched and the polygon will be displayed on the map with a certain color.



All of the queries will be done with NHibernate using NHibernate Spatial for the GIS operations, in this case the intersection between the click-coordinate and the corresponding district area.

NHibernate Spatial (Part 1 - Introduction)

Part 1 - Introduction
Part 2 - First Demo
Part 3 - Improving
Part 4 - Querying
Part 5 - NHibernate 3.3.1+

NHibernate is a nice, tested and mature ORM. I've used it extensively since version 1.2 and it just keeps getting better and better.
It obviously provides many challenges, most of them regarding it's learning curve and some limitations, like handling spatial data at the database

Nowadays several Databases provide spatial extensions. The most notorious are probably PostGIS, Oracle Spatial and more recently SQL Server 2008. NHibernate does not natively handle this type of data but fortunately there's a library that fills this gap quite nicely: NHibernate Spatial Extensions.

In this post (or series of posts) I'm going to do a complete tutorial on setting up a SQL Server 2008 R2 database with spatial columns, mapping the tables to domain objects using NHibernate Spatial and querying them with HQL and QueryOver.

KineticJS

HTML5 Canvas is freaking awesome and, although HTML5 has many improvements over HTML4, Canvas is probably one of the strongest and my personal favorite.

One of the things that I like about Canvas is that it provides a really nice low-level API, allowing for a great degree of control over the drawing. The problem is that, without using an external library, some simple tasks are a little bit cumbersome and hard to maintain.

For example, let's assume we want to create a simple page like:

• Two boxes, one red and one blue.
• Each box will show an alert when clicked upon.

Bing Maps and HTML5 (Part 3 - Optimizing)

Part 1 - Basic
Part 2 - Canvas
Part 3 - Optimizing
Part 4 - Final Optimization

In my previous post I've created an example where about 140.000 points are retrieved from a MongoDB database and drawn on top of a Bing Maps using Html5 canvas.

If you recall, it took about 800+ ms to create and display the canvas above the map when all the points were visible. For the current scenario it's acceptable, as the drawing only occurs after the view changed event has ended. The problem is that I would like to have the drawing being continuously drawn during the view changed events. So, and for the motion to be fluid, each canvas image has to be drawn in less than 40 ms(1000/25).
Obviously it depends on the CPU of the client machine. My laptop is pretty average, so it will be a realistic benchmark. I'll post my attempts to achieve the 40ms target on my computer, starting with the current post, where I try to solve the most problematic operation.