Sunday, June 14, 2015

Game-development Log (14. CDN Improvements)


I haven't been really active on the development of this project as I've been occupied with other stuff. Regardless, I'm now focused on returning to active development on it.

Interestingly enough what actually triggered my return was a couple of Azure announcements last week on the CDN front. First, some context:

Although I like Azure quite a lot its CDN offering has been quite lacking to say the least. The community was quite vocal in requesting some essential features but Microsoft neglected to provide any updates or expected delivery dates, as seen here: http://feedback.azure.com/forums/169397-cdn

For example, the top voted feature request was the ability to force content to be refreshed, which is, IMHO, completely essential for a CDN offering:

http://feedback.azure.com/forums/169397-cdn/suggestions/556307-ability-to-force-the-cdn-to-refresh-any-cached-con

The feature was requested 5 years ago, eventually marked as "planned", and no further update was provided by Microsoft, similarly to the other requested features.

Well, all of this until last week, when Microsoft apparently woke up.

First they've provided feedback on most of the feature requests and provided an expectation around release dates. Not ideal (as most of these features are late by a few years) but positive nevertheless.

Then, the icing on the top of the cake was this post:

https://azure.microsoft.com/blog/2015/06/04/announcing-custom-origin-support-for-azure-cdn/

Basically Microsoft shipped three awesome features for the CDN:

I'll just copy&paste from that post:
Custom Origins Supported
Azure CDN can now be used with any origin. Previously, Azure CDN only supported a limited set of Azure Services (i.e. Web Apps, Storage, Cloud Services and Media Services) and you only had the ability to create a CDN endpoint for an Azure Service that was in your Azure Subscription. With this recent update, you can now create a CDN endpoint for any origin you like. This includes the ability to create an origin in your own data center, an origin provided by third party cloud providers, etc. and gives you the flexibility to use any origin you like with Azure CDN!
Multiple CDN Endpoints with the Same Origin
Several of you may have tried to create multiple CDN endpoints for the same origin and found this wasn’t possible due to restrictions. We have now removed the restrictions and you now have the ability to create multiple endpoints for the same origin URL. This provides you a more refined control over content management and can be used to improve performance as multiple host names can be used to access assets from the same origin.
Save Content in any Origin Folder
Previously, when you created a CDN endpoint for cloud services you were required to use “/cdn/” as the default origin path. For example, if the path for your cloud service washttp://strasbourg.cloudapp.net you were required to use http://strasbourg.cloudapp.net/cdn/ as the root path to get content from your origin when you created a CDN endpoint. This restriction has been removed and you can store content in any folder. Using the previous example, you can now use http://strasbourg.cloudapp.net/as the root path to get content from your origin.
These might seem minor changes but let me explain how they positively affect this project:


Multiple CDN Endpoints with the Same Origin

 Completely related with my blog-post on improving tile-loading at the browser (http://build-failed.blogspot.pt/2015/03/improve-tile-loading-at-browser.html), a really simple performance improvement relies on having multiple urls for the map-tiles.

This technique is called domain sharding. Different domains are used to fetch the same information, thus bypassing the "same-domain" browser limitation. This limitation is implemented differently on the various browsers, but all include a hard-limit on the number of requests that can be done concurrently to the same url. With this approach I'm basically quadrupling this number.

I've created 4 different CDN urls:

  • http://az710822.vo.msecnd.net/imagetiles/{quadkey}.jpg 
  • http://az768596.vo.msecnd.net/imagetiles/{quadkey}.jpg
  • http://az769152.vo.msecnd.net/imagetiles/{quadkey}.jpg
  • http://az769848.vo.msecnd.net/imagetiles/{quadkey}.jpg

All of them are pointing to the same blob storage containg all the tiles. The difference is that, on client-side, the url used to request each tile is dependent on the last digit of the quadkey that identifies the tile.

Thus:
  • http://az710822.vo.msecnd.net/imagetiles/{quadkey}.jpg   (quadkeys ending in 0)
  • http://az768596.vo.msecnd.net/imagetiles/{quadkey}.jpg   (quadkeys ending in 1)
  • http://az769152.vo.msecnd.net/imagetiles/{quadkey}.jpg   (quadkeys ending in 2)
  • http://az769848.vo.msecnd.net/imagetiles/{quadkey}.jpg   (quadkeys ending in 3)
Example:



Notice the different urls for each tile.

The interesting part is that, cost-wise, this approach doesn't affect my Azure bill as the set of cached tiles for each CDN has no duplicates, thus not requiring additional disk space.

Save Content in any Origin Folder

Serving custom tiles is a challenge, particularly due to the ridiculous amount of tiles required to serve higher zoom levels. The math is simple: for each zoom level you need the following number of tiles:
4^zoom level

zoom 0 = 4^0 => 1 tile
zoom 1 = 4^1 => 4 tiles
zoom 2 = 4^2 => 16 tiles
...
zoom 20 = 4^20 => 1.099.511.627.776 tiles  (yes, we're talking about trillions here).

This is challenging both in terms of disk space and time spent to generate the tiles. So, a common approach is to pre-render lower zoom levels (ex: 0-10) and serve the higher zoom level dynamically on demand (also, its also quite desirable to cache these while serving them).

For this project I'm pre-generating the tiles up to the zoom level 12 (which is still a reasonable number) and dynamically generating higher zoom levels.

Initially I did setup a CDN to the service that is dynamically generating the tiles but it was quite limiting as it only worked with Cloud-Services, required the /cdn suffix and didn't work well with the routing I had, requiring me to create a custom route with query-string parameters which, although supported by the CDN, wasn't working properly. So, eventually I gave up and was serving the dynamic tiles directly from my webapi hosted directly from a WebProject.

Example:
http://tile-win.azurewebsites.net/13/3923/3085.jpg

With the new Azure feature I can now point a CDN to my existing webapi. Thus I can obtain the same image from the CDN.
http://az768968.vo.msecnd.net/13/3923/3085.jpg
So, what's the performance comparison?



Also, not CDN related

On this update I also:

  • Completely refactored my loading logic. Now I have separate projects for each step, namely:
    • Converting Geographical data to Vector Tiles
    • Generating Image Tiles from Vector Tiles
    • Pushing Tiles to Azure Blob Storage 
This change was really important as it allows me to streamline the creation and publication of the tiles going forward.
  • Fixed one of the most tricky bugs I had on the loading logic.
I had this problem:


As I explained on one of the posts in this series, I split the geographical data into manageable chunks. When processing the individual tiles some artifacts would be visible on the seams between those chunks.

I detected this problem a long time ago but I didn't have an immediate solution for it. With the loader refactor I managed to find an elegant solution for the problem. The same area now appears as:

Note: Eventually will still appear incorrect on the main website due to the CDN caching, which is currently setup to about 1 week.


Next steps:

  • Improvement to game mechanics
Posted by at

2 comments:

  1. giggiozMay 17, 2016 at 2:16 PM

    Impressive work, impressive dedication, impressive skills.
    Great work and thanks for sharing all your techs, it's a gold mine.

    ReplyDelete

Subscribe to: Post Comments (Atom)