Ask HN: Performance optimizations for Geospatial applications
I am in the process of creating a mobile app that consumes geospatial data intensively. I am displaying a map of points of interest and upon user interaction (pinching, panning) with the map, I query the database for the new region. As you may infer this is very computationally expensive as I could potentially be making thousands of geospatial queries per minute. My current DB is RethinkDB, however as explained in Uber's tech stack article they use Redis for caching geospatial data. Does anyone have any insight into how to accomplish this? Are there any alternatives you recommend? Thanks
1 comments
I don't know exactly what kind of data you are using but here are a few things -
Often with geodata, people viewing a map of the whole country are interested in a different subset of the data than people viewing a small local area. For instance, when I view Texas I don't need the database to tell me where every street is - only the major rivers and highways. This is called layering. Consider 3-5 zoom levels and filter out the data that your user is going to care about at each, then in your indexing / storage scheme make it easy for the database to filter down to an appropriate dataset for the zoom level.
Also, fetch an area 4 times too big from the server and cache it so as they pan around in the local area you don't have to keep querying.
If you are rendering maps on the server side, and hand rolling that, consider using WMTS rather than WMS to serve your raster maps. WMTS tiles can be cached much more effectively than WMS tiles.
Just because something can be a spatial query doesn't mean it has to be. Spatial queries are relatively heavy. If your users are mostly operating inside one country, feel free to segment your data by country and only query the database for data for the countries on the screen - much lighter than applying a bounding box to a bunch of polygons.
Often with geodata, people viewing a map of the whole country are interested in a different subset of the data than people viewing a small local area. For instance, when I view Texas I don't need the database to tell me where every street is - only the major rivers and highways. This is called layering. Consider 3-5 zoom levels and filter out the data that your user is going to care about at each, then in your indexing / storage scheme make it easy for the database to filter down to an appropriate dataset for the zoom level.
Also, fetch an area 4 times too big from the server and cache it so as they pan around in the local area you don't have to keep querying.
If you are rendering maps on the server side, and hand rolling that, consider using WMTS rather than WMS to serve your raster maps. WMTS tiles can be cached much more effectively than WMS tiles.
Just because something can be a spatial query doesn't mean it has to be. Spatial queries are relatively heavy. If your users are mostly operating inside one country, feel free to segment your data by country and only query the database for data for the countries on the screen - much lighter than applying a bounding box to a bunch of polygons.
Well I have a simple GeoJSON point object with lat/long coordinates. The document also stores a score of sorts and what I plan to do is limit the results to a certain number say 20 (although still not quite sure) and then sort by the score. This way people will only view the most interesting locations no matter if they are looking at a street level view or on a national level.
The way rethinkDB advises is to query by intersections of circles. So on the client I calculate the center of the map, and then the radius based on the edges of the screen. Then I send that to construct the circle and get all points within the circle. I am just unsure how I can organize these results and cache them within Redis (or a better alternative) for ease of retrieval.
The way rethinkDB advises is to query by intersections of circles. So on the client I calculate the center of the map, and then the radius based on the edges of the screen. Then I send that to construct the circle and get all points within the circle. I am just unsure how I can organize these results and cache them within Redis (or a better alternative) for ease of retrieval.
Okay, so presumably you are rendering this geojson in something like a google maps data layer, or some other kind of mapping library.
That means if you give it some GeoJSON which describes some points outside the normal screen bounding box you then that is okay - stuff on screen gets shown and stuff off screen doesn't get shown until you pan to it.
So far so good.
Now request and a circle 4 times too big and every time they move the screen, test if the ouside of the screen is outside that circle yet. Don't send another request till it is. Now you have a lot less requests for panning around.
But you want to cache, and all these very unique circles are hard to cache for right?
So instead of fetching a custom circle, divide up the world into a bunch of mathmatically calculated static squares [1]. When they view an area decide what square they are inside and query that. Like the 4xbig circle if you are feching some area outside the current view then it's around for when they pan. But critically - you can cache the result for a query for a square!
Hope this helps?
I haven't done this with rethinkdb.
[1] http://www.ra.ethz.ch/cdstore/www6/technical/paper130/pyrami...
That means if you give it some GeoJSON which describes some points outside the normal screen bounding box you then that is okay - stuff on screen gets shown and stuff off screen doesn't get shown until you pan to it.
So far so good.
Now request and a circle 4 times too big and every time they move the screen, test if the ouside of the screen is outside that circle yet. Don't send another request till it is. Now you have a lot less requests for panning around.
But you want to cache, and all these very unique circles are hard to cache for right?
So instead of fetching a custom circle, divide up the world into a bunch of mathmatically calculated static squares [1]. When they view an area decide what square they are inside and query that. Like the 4xbig circle if you are feching some area outside the current view then it's around for when they pan. But critically - you can cache the result for a query for a square!
Hope this helps?
I haven't done this with rethinkdb.
[1] http://www.ra.ethz.ch/cdstore/www6/technical/paper130/pyrami...
