Protomaps Blog

New CLI feature: pmtiles merge

Tue, 16 Dec 2025 00:00:00 +0000

The pmtiles CLI now includes the highly requested merge command for combining multiple tilesets into a single archive.

As an example use case: the Mapterhorn project distributes several archives:

planet.pmtiles for the full planet terrain mosaic, from zoom levels 0 to 12.
6-20-28.pmtiles, 6-21-29.pmtiles, etc for zoom 13 and above in specific areas. These archives can contain up to zoom level 17.

These archives are split to ensure each one is at most a few hundred gigabytes: the total size of all tilesets is over 2 TB.

If you need to combine both low resolution and high resolution tiles into a single archive for one area like Interlaken, you can accomplish this now with pmtiles extract followed by pmtiles merge:

pmtiles extract \
 --bbox=7.74,46.60,7.96,46.75 \
 https://download.mapterhorn.com/planet.pmtiles \
 planet.pmtiles

pmtiles extract \
 --bbox=7.74,46.60,7.96,46.75 \
 https://download.mapterhorn.com/6-33-22.pmtiles \
 6-33-22.pmtiles

pmtiles merge planet.pmtiles 6-33-22.pmtiles interlaken.pmtiles

This results in a single archive, hostable as a single file on cloud storage, enabling smooth zoom from 0 to 17.

Requirements

The merge command currently works for both raster and vector tilesets.

All input archives must be clustered.
All input archives must be the same tile type. It’s not possible to combine raster with vector tiles, or PNG with JPEG, for example.
All input archives must have the same compression method.
For now, inputs must be disjoint with 0 overlapping tiles. The easiest way to satisfy this is with non-overlapping zoom levels.

Disjoint inputs

The current implementation of pmtiles merge does not work with overlapping inputs. The PMTiles CLI is a tool for reading and writing with general tile archives, not a tool for editing individual tiles.

Consider two tilesets that contain zoom 0 to zoom 12 of two different regions of the world, like a square region around Tokyo and Paris. All tilesets that form a complete pyramid contain tile 0,0,0:

 0, 0, 0 0, 0, 0 
┌───────────────┐ ┌───────────────┐
│ ┌───┐ │ │ │
│ │ P │ │ │ │
│ └───┘ │ │ │
│ │ │ ┌───┐│
│ │ │ │ T ││
│ │ │ └───┘│
└───────────────┘ └───────────────┘

The logical merging of the two tilesets should result in a brand new 0,0,0 tile combining the geometry (vector), or compositing the pixels (raster):

 0, 0, 0 
┌───────────────┐
│ ┌───┐ │
│ │ P │ │
│ └───┘ │
│ ┌───┐│
│ │ T ││
│ └───┘│
└───────────────┘

However, combining geometry or image compositing operations are both outside the current scope of the PMTiles CLI.

Recommendations

If your project requires merging different tilesets that are overlapping, it’s recommended to instead pre-generate or extract the tilesets from the source data.

For raster archives, use the rio-pmtiles command line tool to generate a PMTiles directly from a GeoTIFF or other GDAL-readable source.
For vector basemaps, use pmtiles extract with a multipolygon input.
For other vector datasets, use tippecanoe on the combined source data directly or use a Planetiler profile like protomaps/basemaps on combined source data.

Future work

A future enhancement to pmtiles merge will allow overlapping input tilesets only in the case for vector tilesets with non-overlapping input layers.

This is especially useful for combining overlay data like bike lanes or choropleths with basemap tiles into a single archive, which can halve the number of requests for a mapping application and speed up the map browsing experience.

Mapterhorn - Terrain for Web Mapping

Tue, 02 Sep 2025 00:00:00 +0000

The Protomaps project is the PMTiles format, its tooling, and a 120GB basemap vector cartographic tileset created from OpenStreetMap and other open data sources. PMTiles is not limited to storing vector data - it’s also used for raster data, like scans of historical paper maps.

Mapping apps often don’t just need to show vectors of buildings, boundaries and places. Some apps need elevation data, since interesting places on Earth aren’t flat! The Mapterhorn project fulfills this with a new independent open data product - it’s Protomaps for Terrain.

Project Inspiration

Mapterhorn’s inspiration is the Mapzen Joerd project. Joerd is available as tiles from AWS Open Data, and was originally created for the Tangram map renderer, but works with MapLibre GL as well. It’s built from a collection of digital elevation models (DEMs) processed into a single tileset using batch jobs on Amazon Web Services.

{
 type: 'raster-dem',
 tiles: ["https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"],
 maxzoom: 13,
 encoding: 'terrarium',
 attribution: "<a href='https://github.com/tilezen/joerd/tree/master'>Joerd</a>"
}

Mapterhorn’s goals are similar to Joerd - create a global, easy-to-use terrain tileset, with an initial focus on European DEMs.

Key project differences

Mapterhorn’s design differs from Joerd and other open data projects in these ways:

Focus on interactive web visualization - The end product is sliced into tiles at conventional sizes like 512x512 pixels for direct usage in 2D and 3D web maps. Tiles are stored in the terrarium encoding which MapLibre GL supports.
Ease of tileset recreation - Instead of being tied to AWS, Mapterhorn can be reproduced from scratch using a single powerful machine, either a desktop or a rented server. This means the pipeline can be customized with different data if your project requires more detail in certain countries. The full pipeline is open source on GitHub.
Ease of deployment - Mapterhorn distributes the end product as static PMTiles archives, which can be directly read from cloud storage to map libraries in browsers. This is used to visualize the Mapterhorn tileset on mapterhorn.com as well as the pmtiles.io viewer.

Using the PMTiles format for distribution means you can use the pmtiles extract CLI on a planet archive. To extract only the area surrounding the Matterhorn, try this:

pmtiles extract \
 --bbox=7.510659,45.897669,7.799642,46.04662 \
 https://download.mapterhorn.com/planet.pmtiles \
 planet.pmtiles

Project Future

Oliver Wipfli, a former official coordinator of the MapLibre project is leading the development of Mapterhorn. The initial phases of the project are supported by an NLnet grant. If you’re a company or organization that needs high resolution terrain for web visualization, start a discussion on GitHub!

A Tour of the New pmtiles.io

Thu, 15 May 2025 00:00:00 +0000

pmtiles.io is an open source viewer for PMTiles tilesets. It’s a great way to inspect tilesets before you publish them on your site or app. It’s also a simple way to share GIS datasets without coding anything - just upload your PMTiles to any cloud storage, set up CORS, and deep link to the file on pmtiles.io using embedded URL parameters.

pmtiles.io got a re-write! It now features:

Light UI mode
support for viewing TileJSON (ZXY endpoints) in addition to PMTiles. For example, inspect the vector tiles hosted at vector.openstreetmap.org.
Drag-and-drop of local files
Inspect raster tilesets

At the top of the site are three main tabs for Map, Archive and Tile.

The Map view

The primary view for looking at tilesets and JSON metadata. Zoom in, pan around, and deep link to a specific view using URLs. Powered by MapLibre GL JS. View vector features, color-coded by layer, labeled by their name tag:

Check Inspect Features to hover an individual feature’s tags:

The popup also shows geographic coordinates of the cursor, and includes a button linking to the Tile View.

The Tile View

The Tile view is for map developers that need to drill down into the details of vector geometry. View a single tile in isolation and scale it up to see detailed geometry, without swapping in new tiles. It’s a simple SVG rendering of a single tile powered by D3.js.

Use the top navigation to browse parent, child and neighbor tiles:

The Archive Inspector View

The Archive Inspector is for implementers of the PMTiles specification, and visualizes the low-level internals of a tileset as a tree structure.

Opening the inspector shows the root directory of the archive, the size of tiles or leaf directories, and summary statistics in the header.

Clicking a tile entry opens the Tile View, and clicking a leaf entry opens another directory panel.

On the right, the current directory is visualized on a globe, with tile entries represented as squares, and run-length encoded duplicate tiles represented as space-filling curves.

You don’t need to have a tileset ready to try the new pmtiles.io - a variety of example datasets are included:

v4.pmtiles - vector basemap, the daily build from OpenStreetMap.
flowers.pmtiles - raster overlay, a zoom 20+ small area of aerial drone imagery.
places.pmtiles - POIs derived from Meta and Microsoft places, distributed by the Overture Maps project.

Covering the Planet with Compressed Bitmaps

Mon, 27 Jan 2025 00:00:00 +0000

This is the first post explaining how the pmtiles extract command works to slice a smaller tileset from a larger one. Protomaps Builds contains a daily planet tileset - if you want just Canada, or South America, you can pass pmtiles extract a GeoJSON to get only that region.

The first step in extracting a slice is computing a tile covering for the target region. A covering is the set of all tiles that touch the edge of the region, or lie completely within it. Coverings are also useful for finding if two tilesets overlap. If you’re building tools for the PMTiles format, this covering technique can make your program faster and use less memory.

Polygon Rasterization

Covering a region with tiles is the same task as polygon rasterization in computer graphics. Start with a polygon like a triangle, then find all the pixels to fill the shape on a screen.

Classical algorithms perform this with scanlines or other algorithms - see this interactive demo to explore methods. For computer graphics, the output is a framebuffer of pixels.

Working with map tiles results in much larger effective screen sizes: the current maximum zoom of the Protomaps basemaps is 15, which is 32768 x 32768 pixels. A region covering for a continent where each entry is stored as (Z,X,Y) in a Set or Hashmap consumes 100+ megabytes of memory.

Data Structure

Instead of holding a Set or Hashmap of pixels, we use a Bitmap - each “pixel” is a single boolean bit in a large array, like a black and white framebuffer image. We use the Hilbert TileID as a 64-bit integer index into the bitmap.

Plain in-memory bitmaps are inefficient, though, because the domain of possible values is large: addressing zoom level 15 addresses over one billion tiles, and storing an array of zeros to set the (0,0) pixel bit at z15 would take up 100+ megabytes.

Roaring Bitmaps are a popular implementation of compressed bitmaps. These are especially good for storing regions on space-filling curves because areas occupy continuous parts of the curve, so can be run-length encoded by Roaring.

Roaring Bitmaps are a core part of the software ecosystem around the PMTiles format, but not part of the format itself. It is cheap and easy to construct an in-memory Roaring Bitmap over a set of PMTiles directory entries:

tile_id uint64, offset uint32, length uint32, run_length uint32

roaring = new Roaring()
roaring.addRange(tile_id, tile_id + run_length)

Algorithm

Roaring Bitmaps compactly represent tile coverings, but do not address how to efficiently compute those coverings.

For the PMTiles Tile ID addressing scheme, we start with finding the boundary covering using classical methods, and then exploit the nature of the Hilbert curve: it has has the special property of no jumps - unlike other addressing schemes such as Morton (z-order), or Row/Column major, there are zero discontinuities in the Hilbert Curve. (The Peano curve also shares this property.)

This means that if the curve leaves the region and later enters the region, the intermediate positions on the curve between those two points are either all outside or all inside.

To fill in the interior polygon, iterate over the boundary bitmap and test every gap for whether or not the first tile lies on the interior of the polygon. If it does, add every tile up to the next boundary tile.

Results

The following table shows the growth in RAM usage of the program attached below, which covers the South American continent at different zoom levels using both techniques.

Zoom	Hashmap (Memory)	Hashmap (Time)	Roaring (Memory)	Roaring (Time)
13	54 MB	150 ms	3 MB	7 ms
14	207 MB	754 ms	4 MB	13 ms
15	823 MB	3000 ms	4 MB	25 ms
16	3295 MB	24000 ms	4 MB	47 ms
17	13176 MB	103000 ms	5 MB	95 ms

For the simple hashmap representation of tile coordinates, the memory and runtime grow exponentially as Z increases, because each additional zoom level addresses 4x more tiles. At zoom level 17, it consumes a whopping 13GB of RAM and takes two minutes to compute.

Roaring bitmaps and Hilbert IDs computes the same result in 100 milliseconds and stores the result in 5 megabytes, even at zoom level 17.

Code

A complete, runnable implementation of the compressed bitmap covering algorithm, with comparison to classical techniques, using the paulmach/orb library:

package main

import (
 "fmt"
 "github.com/RoaringBitmap/roaring/roaring64"
 "github.com/paulmach/orb"
 "github.com/paulmach/orb/geojson"
 "github.com/paulmach/orb/maptile"
 "github.com/paulmach/orb/maptile/tilecover"
 "github.com/paulmach/orb/planar"
 "github.com/paulmach/orb/project"
 "github.com/protomaps/go-pmtiles/pmtiles"
 "io/ioutil"
 "runtime"
 "time"
)

func BitmapCovering(zoom uint8, polygon orb.Polygon) (*roaring64.Bitmap, *roaring64.Bitmap) {
 // compute the covering of the boundary linear ring
 boundarySet := roaring64.New()
 for _, ring := range polygon {
 boundaryTiles, _ := tilecover.Geometry(orb.LineString(ring), maptile.Zoom(zoom))
 for tile := range boundaryTiles {
 boundarySet.Add(pmtiles.ZxyToID(uint8(tile.Z), tile.X, tile.Y))
 }
 }

 // Prepare the polygon in Web Mercator for point-in-polygon testing
 polygonProjected := project.Polygon(polygon.Clone(), project.WGS84.ToMercator)

 interiorSet := roaring64.New()
 i := boundarySet.Iterator()
 for i.HasNext() {
 id := i.Next()

 // For every gap between tile IDs in the boundary, check if the next tile
 // is inside or outside using point-in-polygon
 if !boundarySet.Contains(id+1) && i.HasNext() {
 z, x, y := pmtiles.IDToZxy(id + 1)
 tile := maptile.New(x, y, maptile.Zoom(z))
 if planar.PolygonContains(polygonProjected, project.Point(tile.Center(), project.WGS84.ToMercator)) {
 interiorSet.AddRange(id+1, i.PeekNext())
 }
 }
 }

 return boundarySet, interiorSet
}

func main() {
 dat, _ := ioutil.ReadFile("southamerica.geojson")
 geom, _ := geojson.UnmarshalGeometry(dat)
 polygon := geom.Geometry().(orb.Polygon)

 var m runtime.MemStats
 zoom := uint8(15)

 start := time.Now()
 if false {
 boundary, interior := BitmapCovering(zoom, polygon)
 fmt.Println("covering size", boundary.GetCardinality()+interior.GetCardinality())
 } else {
 covering, _ := tilecover.Polygon(polygon, maptile.Zoom(zoom))
 fmt.Println("covering size", len(covering))
 }
 runtime.ReadMemStats(&m)
 fmt.Println("megabytes", m.Alloc/1024/1024)
 fmt.Println("took", time.Since(start))
}

Input polygon southamerica.geojson:

{"type":"Polygon","coordinates":[[[-62.57541,14.915208],[-72.821509,14.915208],[-72.821509,14.915208],[-82.904973,6.291524],[-85.995066,-3.289539],[-73.959965,-23.278034],[-81.441244,-50.744789],[-72.821509,-57.406184],[-63.551229,-56.429744],[-58.184224,-42.141627],[-35.57775,-17.022015],[-31.186565,-5.235684],[-44.197485,2.400227],[-62.57541,14.915208]]]}

What is a basemap, anyway?

Mon, 20 Jan 2025 00:00:00 +0000

The Protomaps project consists of:

The open PMTiles format;
The tools and ecosystem for creating and publishing PMTiles;
A basemap PMTiles created from OpenStreetMap data and other open sources.

Mapping apps need background context to help humans understand where things are. The base layer has water bodies, buildings and roads. You could use Google Maps and Apple maps for those, but Protomaps is open source!

Two key aspects of a basemap are labeling and generalization. Text labels help users identify well-known places. Generalization is the choice of features that appear at zoomed-out overview levels. It’s what makes a basemap different from merely a data dump of OpenStreetMap. Some data necessary for generalization will never be part of OSM, such as importance rankings from Natural Earth.

Basemap Origins

The Protomaps basemap is inspired by a long legacy of web mapping projects.

Legendary San Francisco design studio Stamen created OSM basemaps as part of a Knight News Challenge grant. These basemaps were built with OpenStreetMap, PostGIS and Mapnik.

The Toner basemap was especially suited as a background to data visualization overlays, and was a popular style that strayed from the “road atlas” look of early web maps.

The Watercolor map took the raster slippy map rendering stack to its limit, and pushed the boundaries of what was recognizable as a basemap, while using familiar OpenStreetMap data.

The original code for rendering Toner is available on GitHub at the toner-carto and watercolor repositories.

Mapzen

The Protomaps basemap is primarily derived from the Tilezen and Tangram projects that were developed at Mapzen from 2013 to 2018.

The Bubble Wrap style was a general-purpose basemap that included icons for points of interest.

Refill was not a single map, but a family of mono or duo-chrome maps in the vein of Toner that could be customized for different label densities:

You can read more about the cartography behind Tilezen on Mapzen’s former blog:

Moving Web Maps Forward

One vital aspect of the Tilezen project is its open license - it’s available under permissive FOSS licenses, and is now a Linux Foundation project. This provides a solid foundation to build a basemap, instead of having to start from scratch.

Key parts of Tilezen that are re-used or replaced in Protomaps are:

PMTiles is an alternative to the Tapalcatl storage system. A copy of the Tilezen dataset in PMTiles format is available for viewing and download.
The Protomaps Basemap Layers Version 4 are backwards-compatible with Tilezen Layers, with the exception of a few helper tags.
The Tilezen build pipeline is replaced with a Planetiler profile that runs in 2 hours on a fast machine, instead of days on AWS.

Protomaps Basemap

The Protomaps basemap also adopts MapLibre as its frontend rendering library, instead of the Mapzen Tangram project, because of its features like map rotation and globe mode, and its ecosystem of plugins.

For MapLibre we’ve had to design an entire new set of styles for the Tilezen tileset. The maps are designed in collaboration with Geraldine Sarmiento - you can check out her portfolio, including her work on the Stamen and Mapzen styles, at her portfolio site.

Rather than needing to maintain and update many independent styles, the Protomaps basemap is one parameterized style in TypeScript that can be customized using simple palette objects - think of it like a color scheme or paint job, applied to the same map “skeleton.” This lets different visual styles share common logic for the ordering and relative prominence of features.

The maps include:

A default light flavor. It includes icons originally developed for Mapzen’s Bubble Wrap style.

A Dark mode variant of the light flavor.

Data visualization flavors, inspired by Refill. Designed for overlaying data or even “sandwiching” data.

Project Status

The implementation of the above map styles is still a work in progress, but sufficient for a wide range of mapping use cases like data visualization! You can follow along at:

the interactive Basemaps Layers documentation.
the basemaps repo on GitHub, and the basemaps-assets repo for fonts and sprites.

This work is supported by an NLNet Foundation NGI0 grant.

Slicing OpenStreetMap Just Got Fresher

Fri, 29 Nov 2024 00:00:00 +0000

Developing software for OpenStreetMap data all starts with the PBF - it’s nodes, ways and relations in the raw, native OSM format. You can make tiled maps, PostGIS databases, or GeoJSON out of PBFs, but those conversions lose some tagging and metadata. With PBF, you get pure OpenStreetMap!

The global OSM.org website provides a weekly copy of the entire planet as one ~80GB PBF. And there’s convenient sites like Geofabrik Downloads for getting just one country of data. These are amazing free resources, with one catch: if you’re contributing data to OSM, those sources are instantly out of date once you upload.

Protomaps Extracts

Protomaps Extracts (previously hosted at protomaps.com/extracts) aims to solve this - it’s a web portal to download minute-updated extracts of OSM for any area, up to 100 million nodes. Well, that’s how it used to be, because this is the last post about Protomaps Extracts here! The service has been migrated to an OpenStreetMap US Community Project, at the url slice.openstreetmap.us, and has:

A flashy new logo and domain
a new, 100% open source backend and UI, with no API keys or logins required
a new GitHub organization SliceOSM outside of the Protomaps umbrella project

New project: SliceOSM

Think about OpenStreetMap as a big pizza party. You can’t eat a whole pizza yourself: you just want a a slice of your local city or country. And with minutely updates, if you add your own toppings, like pineapple or anchovies, your friends get those on their pizza slice too!

By migrating into the OpenStreetMap US nonprofit, this service:

Brings the tool to a wider audience outside of just Protomaps map users
Increases the open source maintenance “bus factor”
In the future, can seamlessly integrate with other OSMUS initiatives such as OSMCha, OpenTrailMap, and MapRoulette

Providing this free service is also supported by hosting credits from the AWS Open Data program.

Project Structure

SliceOSM is built on the OSM Express database for replicating the OpenStreetMap planet by the minute and spatial indexing via S2 cells. It’s intended to succeed Michal Migurski’s original Metro Extracts as well as Mapzen’s Extracts on Demand.

The project is designed to be modular and re-usable, with:

sliceosm-frontend built on Terra Draw, OSM Americana and MapLibre GL
sliceosm-api for wrapping OSM Express calls in a go web server

Project Alternatives

SliceOSM is designed for city to country-sized slices, made by human editors. If you need larger areas, bulk slices, or automated use, some alternatives you should look into are:

using osmium-tool on a weekly planet file
using Geofabrik Downloads for daily countries

PMTiles on More Platforms

Mon, 28 Oct 2024 00:00:00 +0000

The PMTiles format is a cloud-native way to visualize map data in the browser, directly from cloud storage. Browser decoding via the TypeScript implementation is ideal for most use cases, and accelerated deployment options run on AWS and Cloudflare.

Here’s some highlights of new platforms and integrations in 2024:

Style Editor

Maputnik is a visual editor for MapLibre Style JSON that’s been migrated to the MapLibre GitHub umbrella organization. It’s the most full-featured way to tweak colors and label appearance. An experimental fork of Maputnik adds PMTiles sources directly to the editor, so you can customize maps on cloud-hosted datasets without any server or proxy setup!

Try it at editor.protomaps.com.

OpenLayers

The ol-pmtiles library has reached major version 1.0. New features include TypeScript support and custom non-HTTP data sources, such as local files. This library can be bundled with your OpenLayers application to display both raster or vector tilesets using the built-in tile display classes.

See example page and code for OpenLayers vector tile integration.

AWS CloudFormation

PMtiles is designed to run on plain S3-like storage, and has an identical developer experience for small neighborhood maps, scaling up to 100GB+ planet archives. For websites with lots of traffic, it’s best to use PMTiels in conjunction with a CDN integration, like with AWS Lambda and Cloudfront, to serve edge-cached ZXY tiles to viewers, eliminating the overhead of client-side decoding.

Previously the method to accomplish this on AWS was to click in the console for IAM role setup, lambda function deployment and CloudFront configuration. This was too time-consuming and error-prone.

The automation options we considered were Terraform, CDK and CloudFormation. CloudFormation ended up being the simplest choice, since anyone access to the AWS console can one-click upload a single YAML and go straight from a private S3 bucket to CDN tiles - no CLI required!

Thanks to contributors Carlos Honrado for kicking this off and Charlie McGrady for reviewing this feature.

Google Cloud and Azure

Along with AWS and CLoudflare, two new platforms have support for serverless decoding: Google Cloud and Microsoft Azure. This is a perfect fit for projects already built on GCP or Azure and leveraging blob storage.

Both of these deployment methods use the go-pmtiles Docker image to run on a scale-to-zero container platform, either Cloud Run or Azure Container Apps. Most importantly, the authentication with private storage is built-in and doesn’t require manually managing credentials.

Check out the guides for accelerating PMTiles on Google Cloud and Microsoft Azure.

Server

There’s plenty of reasons not to use any of the major cloud providers - you might simply want to run a world map on a server in your basement, on an airplane, or on an affordable VPS.

The docs for running the pmtiles CLI have been revamped, with details on pmtiles serve and setting up CORS and remote or local archives. Serving tiles this way is great for development too; for production use it’s best behind a reverse proxy like Nginx or Caddy.

Caddy

Additionally there’s now a first-class Caddy plugin for PMTiles, so you don’t need to keep any separate process running, the PMTiles decoding is embedded directly into the webserver! Configuration is directly in the Caddyfile:

 handle_path /tiles/* {
 pmtiles_proxy {
 bucket https://mybucket.com
 cache_size 256
 public_url https://example.com/tiles
 }
 }

See the docs on downloading a build with the plugin.

Thanks to NLnet and Radically Open Security for funding a security audit of the go-pmtiles server implementation as part of the NGI Zero Core grant program.

You Might Not Want PMTiles

Wed, 22 May 2024 00:00:00 +0000

PMTiles is the open archive format that underlies the Protomaps ecosystem. It’s a single file that enables deploying an entire world map as a static artifact, making mapping accessible to the widest audience of front-end developers and neogeographers.

However, PMTiles isn’t the right solution for all mapping applications.

PMTiles is made for web-based viewing of large, mostly static datasets., for example:

base maps built from OpenStreetMap data.
An interactive exploration of building ages in Seoul by Hanbyul Jo.
A climate change story map of USDA gardening zones by NPR.

All these have a few things in common:

The experience is built on the web platform, instead of a local desktop application.
The total information to be explored is over a few megabytes - more than can be loaded at once for a pleasant website experience.
The dataset changes at most daily, or never.

If your application doesn’t have these three properties, there are simpler alternatives to PMTiles.

GeoJSON

If you’re building a web-based map of static information, but your data is small, you should serve it as a single GeoJSON file.

With MapLibre it’s as easy as adding a GeoJSON source:

sources: {
 "geojson-marker": {
 "type": "geojson",
 "data": {
 "type": "Feature",
 "geometry": {
 "type": "Point",
 "coordinates": [12.550343, 55.665957]
 },
 "properties": {
 }
 }
 }
}

This saves you the hassle of converting your data into tiles, and you can use all the same map styling and interaction techniques as tiled data.

PostGIS

If you’re building a web-based map for a sizeable dataset, but your dataset is dynamic, with frequent user updates, you should store your features in a transactional database.

It’s possible to update a PMTiles file regularly, but it requires re-uploading the file each time on storage. This is fine for hourly or daily updates: any higher frequency uses an uneconomical amount of data transfer.

PostGIS is the industry standard transactional database for geographical features. Popular ways to fetch tiled data from PostGIS are pg_tileserv, martin and the raw ST_asMVT function.

A major challenge for web maps, including PostGIS-based ones, is how to generalize data for lower-zoom tiles. One method is to selectively drop attribute data at zoom levels to make lower zoom overviews lighter.

PostGIS on Supabase

If PostGIS is the right fit, but standing up a database looks intimidating, take a look at Supabase, a hosted PostgreSQL platform with authentication and client APIs built in.

I’ve developed a demo application using PL/pgSQL functions that call the ST_asMVT function. This function is called directly from the browser and passed into MapLibre using addProtocol. Browse the the Overture Places dataset for Singapore, fetched straight from the PostGIS database. The example code is on GitHub at supabase-vector-tile.

Overture Places in Singapore - live PostGIS tiles from Supabase

GeoParquet

If you’re exploring large, static datasets, but don’t need publishing on the web, you can forgo tiles and directly visualize files using desktop software.

Tiling with tools like tippecanoe requires computing generalized overview tiles in advance, and is designed for fetching small, optimized pieces of data over the Internet. If the network is not the bottleneck, and you have your dataset locally, QGIS is an excellent, open-source solution for visualization and mapmaking.

Along with GeoJSON and FlatGeobuf, GeoParquet is a new format that can store large datasets efficiently and is interoperable with open source data tools. GeoParquet 1.0.0 is supported in GDAL version 3.8.0 and above.

Lonboard

Tools like Lonboard enable visualization of GeoParquet in Jupyter notebooks. It’s possible to publish these on the web using hosted notebooks, though transferring tens or hundreds of megabytes leads to more waiting than a tiled map experience. For local data, though, GeoParquet + Lonboard is a great solution for exploratory data analysis that saves you the trouble of converting to a network-optimized, tiled format.

Lonboard Examples - Development Seed Documentation

Protomaps receives NLnet open source grant

Wed, 21 Feb 2024 00:00:00 +0000

The Protomaps open source project is a recipient of a NLnet open source grant! This grant is part of the NGI (Next Generation Internet) initiative of the European Commission.

Motivation

The goals of the NGI Zero Core initiative are to develop alternatives and improvements to core internet hardware, software and protocols, in order to remove gatekeepers, choke points and surveillance capabilities on the web.

Maps are everywhere on the Internet. Think of any popular retail site, travel search or local government portal. The infrastructure of the web, from languages like JavaScript, front-end frameworks, and databases, is overwhelmingly free software. Maps are one of the few remaining choke points of an otherwise open web. Developers are locked-in to proprietary vendors like Google Maps; worse, all web traffic is leaked to third party APIs, creating problems for privacy and GDPR compliance.

The Protomaps project isn’t a hypothetical idea - you can use it right now to deploy a world map as a single file, with none of the typical complications of self-hosted networked software.

Our NLnet grant will focus on enhancing key parts of Protomaps to advance maps on the open web, resulting in software distributed under a standard FOSS license, to be built upon and reused freely.

Maps for the Public Sector

The focus of our initial grant is enhancing the viability of Protomaps for public sector use. This aligns with European mandates for open source software in government, and hopefully helps erode de facto monopolies on public sector GIS by large corporations.

The general-purpose “base layer” component of Protomaps is downstream of OpenStreetMap. This grant will make OSM-based apps simple to deploy for local governments, including key data layers like points of interest. The current daily build channel lets public improvements to OSM appear on tilesets within 24 hours.

Localization

A key requirement of public sector use is localization: displaying maps in the local language, or even multiple written languages. Multilingual audiences, especially in countries with many official languages like Belgium, are underserved by technology companies that focus on commercial use in North America and the English-speaking world. Notably, The MapLibre GL map rendering library, an open source fork of Mapbox GL JS, does not support writing systems of South and Southeast Asia. This presents a barrier to open source adoption for users in countries like Nepal and India.

Project contributor Oliver Wipfli has created a proof-of-concept Nepali map on GitHub:

You can see Brandon’s presentations at State of the Map 2021 and FOSS4G Asia 2023 covering localization for map rendering.

Security & Privacy

The basic Protomaps deployment pattern is static files, taking advantage of HTTP Range Requests. Since this design requires no specialized server software, it can rely on mature, secure HTTP servers like nginx or Caddy. Cloud storage deployment options can satisfy requirements for placing data in specific jurisdictions.

As part of the NLnet grant program, the Protomaps open source system is undergoing a security audit by a third party firm. This will cover the server and command line tools, as well as documented best practices for public sector deployments.

Collaborators

The best part of the NLnet grant program: it’s open to individuals or companies anywhere in the world, not just those in the EU! Our small team of independent developers for this grant:

Brandon Liu - lead developer of Protomaps project, based in Taipei, Taiwan - maintainer of PMTiles open format and ecosystem.
Oliver Wipfli - contributor to MapLibre project, based in Zürich, Switzerland; researching WebGL-accelerated multilingual text for maps.

You can view a list of hundreds of NLnet-funded projects on their website.

Transitioning Protomaps from Open Core to Open Source

Fri, 16 Feb 2024 00:00:00 +0000

The Protomaps Project has gained several new repositories in the past year, including the serverless implementations for AWS and Cloudflare, and a new basemaps repository, which generates a full planet PMTiles archive from OpenStreetMap. A free daily build of the planet output is available on maps.protomaps.com/builds.

These additional components complete the transition from an open core project into an open source one, distributed under a standard BSD License.

What is Open Core?

There’s a paradox at the heart of open source development:

The best FOSS projects have maintainers working on them full-time as their job.
Giving away software for free is a very bad way to make a living.

At one extreme, FOSS developers are employed directly by large corporations with an unrelated business model. The open source project is a necessary, but not sufficient, part of the company’s overall product. That’s why Vercel develops Next.JS, and Facebook funds development of ZStandard. These are success stories where developers are paid to work on FOSS. But continued investment in the project is at the whims of the patron companies. It’s a fit for mature infrastructure projects, but not early-stage ones that attempt to change some status quo.

At the other extreme are early-stage companies and bootstrapped developers. A strategy for funding FOSS development, especially in the developer tools space, is open core - use a fully open source project to attract adoption, which funnels into sales of a premium, proprietary product.

Tailwind UI is a proprietary bundle of templates by the creators of Tailwind CSS.
Sidekiq Pro is a a flat-fee, enhanced version of the Sidekiq Ruby job queue.

FOSS vs. SaaS

Another strategy is to run a paid, hosted version of a fully open source project. This is the approach used by Zulip, Supabase and Tailscale. SaaS is, bar none, the best business model for monetizing software, for a few key reasons:

Adoption: Onboarding is instant with no installation, and software can be updated behind-the-scenes.
Digital Rights Management (DRM): Access can be terminated if users don’t pay.
Price Discrimination: Products are segmented into tiers so companies with higher willingness to pay, pay more.

As I described in Rethinking the Free Tier for Maps, SaaS as a path to financial sustainablity is not a good fit for the Protomaps project, because unlike a database-backed web service, the fundamental design of Protomaps makes it as easy to deploy as possible - a single file on cloud storage. Acquiring the high-ticket customers necessary to build a SaaS business demands convincing businesses that operating Protomaps is difficult. It simply isn’t, and I give you all the tools to do it in 20 minutes.

The Open Core Experiment

Protomaps is not “VCware”. It’s the result of 4 years of bootstrapped, full-time independent developer work, and ensuring its continued development for another 4 years and more cannot be achieved by burning investor money. It must experiment with a balance between user-supported commercialization and its open source ecosystem.

The core of Protomaps is the PMTiles format and ecosystem, which is an open spec in the public domain. Dozens of map apps use the format to serve their own datasets, like the Washington Post and the National Zoning Atlas.

The basemap, or background underlay map, is a necessary complement to your own data. Popular solutions for basemaps are subscription APIs from Google and Esri. The first Protomaps commercial product offering was a one-time basemap download - not an API, not a subscription, just a download with a big Buy Now button:

This enables companies to deploy a self-contained mapping solution for one flat fee, on the cloud infrastructure they already use. The total costs end up 10 to 100x less than a hosted solution like Google Maps, and it works forever.

Leaving Open Core

The Protomaps Store is closed now, and everything previously proprietary is open source on GitHub. It had some potential for sustaining the development of the open source project, but suffers key drawbacks of open core:

It splits development effort between proprietary parts and open source parts. For a single full-time developer, this draws attention away from the the higher-impact FOSS project.
The proprietary nature and “bus factor” of being an independent bootstrapped project presents a risk to potential commercial adopters.
It alienates users. Maintainers waste time explaining which parts are open and which are not, and contributions that overlap with the proprietary components are excluded.
It creates a licensing mess, requiring specialized terms on how data artifacts can be stored and distributed to re-introduce scarcity.

Offering a hosted SaaS is crucial for web apps that don’t need the flexibility of controlling the data on their own storage. The Protomaps API now accepts payments via GitHub Sponsors. This centers the open source code over the inherently proprietary API: at any point, you are free to exit the API and run Protomaps yourself, and your monetary contribution continues to sustain the project.

You can also cache any results from the tiles. Please don’t heavily scrape the API though, that’s inefficient; you can download the whole file for free at maps.protomaps.com/builds, or bulk extract a region using the pmtiles CLI.

New Commercial Distribution

The daily build of the Protomaps basemap is created from up-to-date OpenStreetMap data, and distributed as an ODbL Produced Work. This ensures that map viewers can contribute data into OSM and have free access to an updated map within 24 hours.

Alongside this free build is a new Protomaps Commercial Distribution built from the Daylight Map subset of OSM, which can be a better fit for public-facing deployments by companies. Access to this is granted via the second tier of GitHub Sponsors. Update: The Commercial Distribution is currently paused until later in 2024!

You can buy it once, copy it to your own cloud storage, and never pay for it again. A monthly sponsorship gives you access to new versions.

Any previous customer of the Protomaps Store has complimentary access to the Commercial Distribution for one year.

Protomaps Access

The third product tier available through GitHub Sponsors is Access - direct communications with the project lead developer, via your company’s Slack or Teams. This includes:

Evaluation of private datasets and deployment infrastructure
Recommendations on geospatial architecture, even if it’s not Protomaps: PostGIS, OpenLayers and more
Ensures knowledge is accessible to your entire team

There’s no long-term commitment for Access - a single month can work as a training course for your org on how to simplify your geospatial apps. Many companies have already adopted Protomaps into critical parts of their product, and Access is the best way of ensuring continued investment in that infrastructure.

What is a clustered PMTiles archive?

Wed, 17 Jan 2024 00:00:00 +0000

One detail in the PMTiles version 3 specification is a boolean flag called clustered. Popular tools like the pmtiles CLI, tippecanoe and Planetiler always create clustered archives. PMTiles is an open specification in the public domain, so this post is to aid developers in implementing this optional feature.

Clustered

The term clustered comes from relational databases: the PostgreSQL manual defines that “When a table is clustered, it is physically reordered based on the index information.”

A PostgreSQL index may be created over any column or set of columns; the directories of a PMTiles archive are the analogue of a database index, where the lookup key is the Hilbert TileID. Setting the clustered property of a PMTiles archive means that the ordering of the tile data on disk matches the directories. Since directories must be sorted by Hilbert TileID, the tiles must also be in Hilbert order.

De-duplication

Ordering the tile contents in PMTiles has one small catch - tiles are allowed to be de-duplicated. This is essential for representing the ocean on basemaps, otherwise 70% of the archive would be repetitive water tiles.

A toy example of a tileset with 4 tiles of length=100, 2 of which are the same tile contents:

 ┌───┐┌───┐
TileID=1, offset=0, │ A ││ O │
TileID=2, offset=200 (ocean tile) └───┘└───┘
TileID=3, offset=100 ┌───┐┌───┐
TileID=4, offset=200 (ocean tile) │ O ││ B │ 
 └───┘└───┘

Because the tile with offset=200 appears twice, it’s ambiguous exactly where it should go relative to other tiles. To resolve this, the PMTiles spec defines clustered as only having backward references to lower offsets in the file, never jumping forward to a higher offset.

 ┌───┐┌───┐
TileID=1, offset=0, │ A ││ O │
TileID=2, offset=100 (ocean tile) └───┘└───┘
TileID=3, offset=200 ┌───┐┌───┐
TileID=4, offset=100 (ocean tile) │ O ││ B │ 
 └───┘└───┘

A properly clustered archive has the first tile contents at offset 0, and all unique tile contents are contiguous for the entire file.

Locality

The historical motivation for clustering database tables was to improve locality. When most servers used magnetic hard drives, disk access to nearby parts of a file was much faster than far-away parts. PMTiles is designed to work with storage systems like S3: cloud blob store APIs don’t exhibit a huge difference in latency depending on data locality. But if you’re accessing a .pmtiles on disk, or on a magnetic hard drive, the seek time should be lower for a typical task like requesting multiple tiles in one area.

Directory delta encoding

A planet-scale tileset like the Protomaps basemap addresses over one billion tiles. Storing this directory using a naive encoding like TileID(uint64),Offset(uint64),Length(uint64) is about 20 gigabytes of raw data. Since the map user can zoom and pan anywhere on the map, and parts of the directory must be fetched on-demand, it’s key to make this encoding as small as possible. The directory encoding delta-encodes TileIDs; if an archive is clustered, it can also effectively delta-encode offsets, since entries are contiguous (excluding the small % of de-duplicated tiles).

Another way to view this optimization is as a compromise between sparse and dense directories. A sparse representation of the directory is an associative map between TileID and offset+length:

{
 1:[0,100],
 2:[100,100],
 3:[200,100],
 4:[100,100]
}

On the other hand, a dense representation of the directory is an array, where the index in the array is the TileID:

[[0,100],[100,100],[200,100],[100,100]]

Whether a sparse or dense data structure is optimal depends on the data being stored. A Canada wildfires tileset may be widely dispersed, where a sparse structure is better; or they might be OSM basemaps for a populous city that blanket an entire area of continuous tiles. A dense array-like format would ideal, but using it for wildfires means storing many directories full of zero or null entries.

PMTiles uses a sparse directory layout, but by delta-encoding clustered entries, the on-disk bytes are much like a dense layout where only the offset is stored.

Determinism

A more significant advantage of clustered archives is that they make archive layout deterministic: running a program on the same input twice should produce byte-for-byte identical output. Storage engines like SQLite, used in MBTiles, can create different output bytes based on order of operations or library version. Tileset creation is computationally expensive, so programs generate tiles on parallel CPU threads and may finish in indeterminate order. By defining a single, canonical ordering of tiles, detecting data changes in PMTiles works with checksums like MD5, and secure verification of outputs is possible with SHA-256 or Blake3.

Mike Barry has recently enhanced Planetiler’s determinism by ensuring vector tile contents like polygon ring ordering and tag ordering are consistent between runs.

pmtiles verify

The pmtiles CLI now includes a verify operation that validates the contiguous Hilbert ordering of all tile contents, with no forward references.

Recommendations

If your only use case is deploying a PMTiles file on S3 for web map visualization, clustered archives aren’t essential: they only minimize the space taken up by the directory.

Clustered archives are mandatory for more advanced operations on PMTiles, such as extracting a subset of tiles from a larger archive, which will be the subject of a future post.

Serverless Maps - Now Open Source

Mon, 06 Mar 2023 00:00:00 +0000

Accelerating Maps

Last year I described Protomaps Mantle, a commercial solution for self hosting planet-scale maps without running any servers. It’s built on top of the open, cloud optimized PMTiles format, and adds the option of a content delivery network in front to accelerate map tiles to users anywhere in the world.

The code components of Mantle are now open source, under the same permissive licenses as the rest of the Protomaps software stack.

It’s also been renamed: you can find it all at the serverless/ directory of the PMTiles repository, and get started with the docs at the new Protomaps Documentation site: CDN Integration.

A summary of features:

AWS Lambda and Cloudflare Workers functions for turning Z/X/Y URLs into PMTiles range requests to private storage buckets. Under 10kb compressed, with copy-and-paste deployment.
Simple, 20-minute setup guides on installing in your own AWS or Cloudflare account.
Tested in production at scale: On AWS, response times average 100 ms for cache hits, 200 ms for misses. Cloudflare is significantly slower, but is improving as Cloudflare’s R2 offering matures.

Used in Production

Felt uses Protomaps with a CDN to serve user uploaded tilesets created with their open source tile generator, Tippecanoe.

OpenInfraMap uses the Protomaps Cloudflare integration to store and visualize rarely seen infrastructure from the OpenStreetMap dataset.

The Washington Post uses the Protomaps integration with AWS Lambda and Cloudfront to affordably power interactive data stories.

Cost Comparison

A web application with 50,000 map loads using the Google Maps API will cost $350 per month.

A similar amount of traffic running through a Protomaps CDN install on Cloudflare, with an average of 20 tiles per session, costs fifty cents in Cloudflare Workers request fees.

Free Datasets

Protomaps-on-CDN enables serving any tiled data at scale - vector map tiles, raster tiles, or even RGB-encoded terrain data, all from a single serverless function. Here’s two free example datasets to try with your own install:

Vector Tiles (zooms 0-10)

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

Live Example Page

Link to download the dataset

A global vector tile dataset based on OpenStreetMap data. The organization of layers and tags is described at Basemap Layers, and open source styles are provided for Leaflet as well as MapLibre GL vector rendering. It’s based on OpenStreetMap data only, so you can do anything you’d like with the data, subject to the ODbL Produced Work attribution requrements.

Terrain RGB

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

Live Example page

A global terrain RGB dataset, packaged from the Mapzen Joerd project. This is designed to work with MapLibre GL JS for global hillshading.

See the Joerd README for attribution requirements.

Copy these datasets!

You can hotlink directly to these datasets, because they’re stored in public buckets on Cloudflare R2 and include Access-Control-Allow-Origin: * headers. But be aware that these URLs might change in the future.

A solution is to copy the PMTiles files to your own S3 or R2 bucket, so they’ll last forever, under your own control, and at minimal cost. Unlike a map SaaS, you can cache and remix these datasets however you like.

Finally, if you’d just like to use the Protomaps-hosted deployment - it happens to run on Cloudflare - you can register for an API key. It’s free for non-commercial use, and commercial use is available to our GitHub Sponsors.

Happy mapping!

What's new in PMTiles V3

Mon, 31 Oct 2022 00:00:00 +0000

PMTiles is a single-file archive format for map tiles, optimized for the cloud. Think about it like MBTiles, where the database can live on another computer or static storage like S3; or as a minimal alternative to Cloud Optimized GeoTIFFs for any tiled data - remote sensing readings, photographs, or vector GIS features.

Why adopt PMTiles? Companies like Felt, a collaborative mapmaking app, are using PMTiles for user-uploaded datasets - eliminating the need to run map tile servers at all.

Spec version 3

Read the specification on GitHub

In its first year of existence, PMTiles focused on being the simplest possible implementation of the HTTP Byte Range read strategy. PMTiles V3 is a revision that makes the retrieval and storage of tiles not just simple but also efficient. Minimizing archive size and the number of intermediate requests has a direct effect on the latency of tile requests and ultimately the end user experience of viewing a map on the web.

File Structure

97% smaller overhead - Spec version 2 would always issue a 512 kilobyte initial request; version 3 reduces this to 16 kilobytes. What remains the same is that nearly any map tile can be retrieved in at most two additional requests.
Unlimited metadata - version 2 had a hard cap on the amount of JSON metadata of about 300 kilobytes; version 3 removes this limit. This is essential for tools like tippecanoe to store detailed column statistics. Essential archive information, such as tile type and compression methods, are stored in a binary header separate from application metadata.
Hilbert tile IDs - tiles internally are addressed by a single 64-bit Hilbert tile ID instead of Z/X/Y. See the blog post on Tile IDs for details.
Archive ordering - An optional clustered mode enforces that tile contents are laid out in Tile ID order.
Compressed directories and metadata - Directories used to fetch offsets of tile data consume about 10% the space of those in version 2. See the blog post on compressed directories for details.

JavaScript

Compression - The TypeScript pmtiles library now includes a decompressor - fflate - to allow reading compressed vector tile archives directly in the browser. This reduces the size and latency of vector tiles by as much as 70%.
Tile Cancellation - All JavaScript plugins now support tile cancellation, meaning quick zooming across many levels will interrupt the loading of tiles that are never shown. This has a significant effect on the perceived user experience, as tiles at the end of a animation will appear earlier.
ETag support - clients can detect when files change on static storage by reading the ETag HTTP header. This means that PMTiles-based map applications can update datasets in place at low frequency without running into caching problems.

Inspector app

PMTiles on GitHub now hosts an open source inspector for local or remote archives. View an archive hosted on your cloud storage (CORS required) - or drag and drop a file from your computer - no server required.

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

Leaflet

For raster tiles, there is first-class support for loading PNG or JPG image archives into Leaflet via the tiny (7 kilobytes!) PMTiles library like this:

const p = new pmtiles.PMTiles('example.pmtiles')
pmtiles.leafletRasterLayer(p).addTo(map)

For vector tiles, you’ll need to use protomaps.js, the from-scratch renderer built for vector rendering and labeling using plain Canvas. It’s only about 32 kilobytes - a fraction of the size of an alternative like MapLibre GL JS - and now supports V3 archives.

MapLibre GL JS

The MapLibre protocol plugin has a new, simpler API; specifying the archive under a source url will automatically infer the archive’s minzoom and maxzoom.

"sources": {
 "example_source": {
 "type": "vector",
 "url": "pmtiles://https://example.com/example.pmtiles",
 }
}

Python

pmtiles/python on GitHub

Python libraries are now modular and can have data sources swapped out. A PMTiles file can be read from disk, or a custom function can be provided to grab byte ranges from AWS via the boto library, Google Cloud, or any other blob data source.
Python command line utilities have been deprecated as the first-class tooling for creating and working with PMTiles.

Go

go-pmtiles on GitHub

The greatest obstacle to adopting PMTiles for many users was the need to have a working Python 3 installation on your computer.

The official PMTiles tooling is now a single-file executable you can download at GitHub Releases.

Example for converting an MBTiles archive:

pmtiles convert input.mbtiles output.pmtiles

This will spit out some facts on the internals of your archive:

tippecanoe ne_10m_admin_1_states_provinces.geojsonseq -o ne_10m_admin_1_states_provinces.mbtiles -z8
pmtiles convert ne_10m_admin_1_states_provinces.mbtiles ne_10m_admin_1_states_provinces.pmtiles
...
# of addressed tiles: 40560
# of tile entries (after RLE): 20733
# of tile contents: 18933
Root dir bytes: 57
Leaves dir bytes: 53570
Num leaf dirs: 6
Total dir bytes: 53627
Average leaf dir bytes: 8928
Average bytes per addressed tile: 1.32
Finished in 444.930625ms

The above shows that the sample dataset - Admin 1 boundaries from Natural Earth has more than 50% redundant tiles. Although about 40,000 tiles are addresses by the archive, only 19,000 tiles are stored.

On average, only 1.3 bytes or 11 bits is needed per tile in the directory index after compression!

To upgrade your PMTiles V2 archive to V3:

pmtiles convert input_v2.pmtiles output_v3.pmtiles

Inspect a PMTiles V3 archive:

pmtiles show file://. output.pmtiles

Uploading your archive to cloud storage, once you’ve put your credentials in environment variables:

pmtiles upload LOCAL.pmtiles "s3://BUCKET_NAME?endpoint=https://example.com&region=region" REMOTE.pmtiles

Ecosystem

Bringing PMTiles support to OpenLayers (GitHub issue #3).
Luke Seelenbinder has started a implementation of PMTiles in Rust.

Free Downloads

Finally, you can download OpenStreetMap-derived, up-to-the minute basemap tilesets from protomaps.com/downloads, now only delivered in the V3 format. Small-area downloads are perfect for your hyper-local mapping project that will work forever, hosted on storage like GitHub Pages or S3.

PMTiles version 3: Disk Layout and Compressed Directories

Fri, 12 Aug 2022 00:00:00 +0000

PMTiles is a single-file archive format for pyramids of tiled map data. The last post described the new Entry struct to compact repetitive data in-memory; the next step is to further shrink directories for storage on disk and transfer over the Internet.

PMTiles is designed to substitute for APIs like this:

https://example.s3-like-storage.com/tileset/{z}/{x}/{y}.pbf

One storage pattern is to store each tile as its own object, relying on cloud storage’s filesystem-like paths:

 ┌───┐ 
┌────────┐ ┌───▶│███│ /tileset/1/0/0.pbf
│ │───┘ └───┘ 
│ │ ┌───┐ 
│ S3 │───────▶│███│ /tileset/1/0/1.pbf
│ │ └───┘ 
│ │───┐ ┌───┐ 
└────────┘ └───▶│███│ /tileset/1/1/1.pbf
 └───┘

But this approach doesn’t scale up to planet-sized datasets, since millions of unique tiles can take days to upload.

A PMTiles archive is a single file upload:

https://example.s3-like-storage.com/tileset.pmtiles

The information mapping a Z,X,Y coordinate to the address of the tile must be stored in the file itself via an embedded Directory sector. Making interactive maps fast and affordable means making this directory as small as possible.


┌────────┐ /tileset.pmtiles 
│ │ ┌─────────────────────┐
│ │ │1,0,0 ┌───┐┌───┐┌───┐│
│ S3 │───────▶ │1,0,1 │███││███││███││
│ │ │1,1,1 └───┘└───┘└───┘│
│ │ └─────────────────────┘
└────────┘

PMTiles v2 punts on compression completely; it has a 1:1 relationship between the file layout and in-memory data structures. Waiting for half a megabyte of directory data for every map is slow, but the implementation remains dead simple and has been good enough to prove out the design across diverse environments like Cloudflare Workers.

The goal of the next specification version is not just to gzip directories and call it a day, but hand-tune a custom compression codec specific to map datasets. Projects like Apache Parquet combine multiple compression techniques for arbitrary non-tiled data; our approach will look more like the domain-specific compression for PNG images, but tuned to map tiles instead of RGB pixels.

Disk Layout

PMTiles v2 did not enforce any particular ordering for tile contents in the archive, so it’s easy to generate archives with multi-threaded programs like Tippecanoe. v3 adds an optional header field clustered: a boolean indicating the disk layout of tile contents is ordered by Hilbert TileID, analogous to FlatGeobuf’s indexed layout. A clustered archive enables optimized clients to batch tile requests for lower latency, inspired by Markus Tremmel’s COMTiles project.

clustered=false clustered=true 
──────────────▶ ▼ ┌───┐ ┌───┐ ▲
─────▶ ───────▶ └─┘ ┌─┘ └─┐ └─┘
──▶ ──────────▶ ┌─┐ └─┐ ┌─┘ ┌─┐
──────────▶ ──▶ │ └───┘ └───┘ │
──────────────▶ └─┐ ┌─────┐ ┌─┘
───────▶ ─────▶ ┌─┘ └─┐ ┌─┘ └─┐
────▶ ────────▶ │ ┌─┐ │ │ ┌─┐ │
───────▶ ─────▶ └─┘ └─┘ └─┘ └─┘

Test Dataset

Our starting example is a global vector basemap tileset. It addresses 357,913,941 individual tiles, or every tile on every zoom level between 0 and 14. (It includes both an earth and ocean layer, so there are no holes.) After Hilbert run-length encoding, 40,884,468 Entry records remain.

A direct serialization of these records to disk is 40884468 * 24 bytes or 981.2 MB. Simple gzip compression reduces this to 305.4 MB, but we should be able to do better.

Varint Encoding

A web-optimized tileset should have individual tiles under a megabyte in size, so 32 bits for Length is overkill. We replace the fixed-size records with a stream of unsigned Varints. We also chop off unnecessary high bits used in TileId, RunLength and Offset.

This step reduces the 981.2 MB directory to 526.4 MB, or 53.6% of the original size.

 TileId RL Offset Len
┌───────────┬─────┬──────────┬────┐ ┌─────┬───┬───┬────┐ 
│ 100 │ 1 │ 0 │2200│ │ 100 │ 1 │ 0 │2200│ 
├───────────┼─────┼──────────┼────┤ ├─────┴┬──┴┬──┴───┬┴───┐ 
│ 101 │ 1 │ 2200 │2300│ │ 101 │ 1 │ 2200 │2300│ 
├───────────┼─────┼──────────┼────┤ ──────▶ ├──────┼───┼──────┴─┬──┴─┐ 
│ 103 │ 1 │ 4500 │2000│ │ 103 │ 1 │ 4500 │2000│ 
├───────────┼─────┼──────────┼────┤ ├──────┴┬──┴┬───────┴┬───┴┐
│ 104 │ 1 │ 6500 │1900│ │ 104 │ 1 │ 6500 │1900│
└───────────┴─────┴──────────┴────┘ └───────┴───┴────────┴────┘

Delta Encoding of TileID + Offset

Because a directory is sorted by ascending TileID, we can store deltas between consecutive entries instead of large numbers.

In a clustered archive, the physical layout of tile data will mostly match TileID order. Where tile contents are contiguous, we can keep Length while replacing Offset with 0, since the Length implies the delta to the next Offset.

Since this delta encoding makes values small, the varint step above should be even more effective.

These two encodings reduce the 526.4 MB directory to 243.2 MB, or 24.8% of the original size.

┌─────┬───┬───┬────┐ ┌─────┬───┬───┬────┐
│ 100 │ 1 │ 0 │2200│ │ 100 │ 1 │ 0 │2200│
├─────┴┬──┴┬──┴───┬┴───┐ ├───┬─┴─┬─┴─┬─┴──┬─┘
│ 101 │ 1 │ 2200 │2300│ │ 1 │ 1 │ 0 │2300│ 
├──────┼───┼──────┴─┬──┴─┐ ──────▶ ├───┼───┼───┼────┤ 
│ 103 │ 1 │ 4500 │2000│ │ 2 │ 1 │ 0 │2000│ 
├──────┴┬──┴┬───────┴┬───┴┐ ├───┼───┼───┼────┤ 
│ 104 │ 1 │ 6500 │1900│ │ 1 │ 1 │ 0 │1900│ 
└───────┴───┴────────┴────┘ └───┴───┴───┴────┘

Column transpose

Instead of storing each entry in order, we transpose the values to a columnar layout.

┌─────┬───┬───┬────┐ ┌─────┬───┬───┬───┐ 
│ 100 │ 1 │ 0 │2200│ │ 100 │ 1 │ 2 │ 1 │ 
├───┬─┴─┬─┴─┬─┴──┬─┘ ├───┬─┴─┬─┴─┬─┴─┬─┘ 
│ 1 │ 1 │ 0 │2300│ │ 1 │ 1 │ 1 │ 1 │ 
├───┼───┼───┼────┤ ──────▶ ├───┼───┼───┼───┤ 
│ 2 │ 1 │ 0 │2000│ │ 0 │ 0 │ 0 │ 0 │ 
├───┼───┼───┼────┤ ├───┴┬──┴─┬─┴──┬┴───┐
│ 1 │ 1 │ 0 │1900│ │2200│2300│2000│1900│
└───┴───┴───┴────┘ └────┴────┴────┴────┘

This step in isolation does not reduce the size of our directory. However, sparse geographic datasets will have repeated deltas of 1, RunLength=0 and Offset zeroed in the first step, which aids in the next compression step.

General-purpose compression

Finally, a general purpose compression algorithm like gzip is applied to the transposed directory.

This step reduces our 243.2 MB directory size to 91.6 MB, 9.3% of the original size. Without the column transpose above, the result is 102.0 MB.

Compressors like Brotli and Zstandard improve on gzip and are supported by the spec for when they’re widely available in browsers.

Conclusions + Next Steps

Our real-world, planet-scale dataset can address over 350 million individual tiles in just 91.6 megabytes, beating generic compression by a factor of 3.

The finishing touches to header design and directory partitoning are under discussion on GitHub and will be presented at the FOSS4G 2022 conference in Firenze, Italy, along with a richer tool ecosystem for PMTiles.

PMTiles version 3: Hilbert Tile IDs and Run-Length Encoding

Tue, 09 Aug 2022 00:00:00 +0000

PMTiles is a single-file archive format for pyramids of tile data, like those used to power interactive maps. It’s designed to make storage and serving of planet-scale tiled maps simple using only affordable S3-like storage and HTTP Range Requests.

The physical world depicted in maps is infinitely detailed, but also not random like white noise. By exploiting redundancy in the physical world, we can make specialized formats for GIS more compact and efficient.

The universe is sparse at all scales: dense clusters of unique features, with wide swaths of similarity

One step towards this in the first public version of PMTiles (specification version 2) was tile deduplication. The set of all OpenStreetMap PNG tiles will have many references to the same blue square representing the ocean.

Deploying PMTiles v2 for further real-world use cases and datasets has resulted in a few refinements to the design, the first of which is related to the internal tile addressing format for version 3. The new specification is being finalized and discussed on GitHub in time for the FOSS4G conference happening this year in Firenze, Italy.

Old vs. New Design

Refresher: the existing Entry record in PMTiles version 2 looks like this, inspired by the popular MBTiles SQLite-based format:

uint8 Z
uint24 X
uint24 Y
uint48 Offset
uint32 Length

Example: the entry Z=8 X=40 Y=87 points to byte range offset=4240340 length=32836 in a PMTiles file, which is a 32.7 kB PNG image near Vancouver, British Columbia:

The v3 design simplifies the Entry struct to fewer and more standard column types:

uint64 TileId
uint32 RunLength
uint64 Offset
uint32 Length

The equivalent of the above in the new design would be TileId=36052 RunLength=1 Offset=4240340 Length=32836.

TileID

The TileId 36052 corresponds to the Z,X,Y position of 8,40,87. The calculation of ID uses a pyramid of Hilbert curves starting at TileId=0 for zoom level 0. The next zoom level, a 2x2 square, occupies the next four IDs in the ID space TileId=(1,2,3,4), the next level being the next 16 IDs, and so on.

In the example British Columbia tile above, the Z,X,Y position of 8,40,87 corresponds to the 14207th Hilbert curve position on level 8, which starts at TileID=21845.

The TileId is a convenient 64-bit integer that packs low zoom levels into the least significant bits. This makes it efficient to store in bitmap representations like Roaring, variable-length integer encodings, or constrained runtimes like JavaScript. The maximum representable tile zoom level in JavaScript numbers is 28.

RunLength

PMTiles v3 replaces Z,X,Y with the single TileID field, and also adds a new field RunLength.

The previous PMTiles design deduplicates repetitive tiles like blue ocean squares via multiple entries pointing to the same data. However, it can repeat the same entry millions of times for large regions of oceans, which cover 70% of the planet.

Ocean tiles are not only repetitive, but sparse and often contiguous in Hilbert space. This entry:

TileID=2578427,RunLength=107977,Offset=3650651795,Length=42

means that the 44 byte vector tile with a single square in the layer ocean is repeated over 100,000 times, starting at Z,X,Y=11,285,1311 and ending at 11,19,1304. This single contiguous sequence is represented by the yellow-green gradient in the planet image below:

What previously required 107,977 separate directory entries, or 1.8 megabytes of storage, can now be repesented by a single 24 byte entry. This specific case of a 75,000x compression ratio is useful to minimize storage costs and transfer latency of map tiles, which can help developers build and deploy faster, more flexible maps.

The next post will address the design of compressed directories in PMTiles version 3.

Contiguous Hilbert runs

Serverless Maps using Lambda or Cloudflare Workers

Fri, 08 Apr 2022 00:00:00 +0000

Note: Mantle has been open-sourced and renamed, please see the latest blog post!

In my last post, I described the downsides of the existing “pay-as-you-go” business model for maps. Organizations searching for a map solution weigh the benefits and drawbacks of build vs. buy. Buying a hosted SaaS — and letting someone else manage infrastructure for you — makes sense when DIY map infrastructure looks like this diagram:

In this post, I’ll introduce Mantle, the commercial offering of the Protomaps system. It’s the same backend that powers the interactive Leaflet maps on this site, like at Protomaps Downloads. It enables scalable hosting of map data with two moving parts:

Runs in your own cloud account

Mantle is a commercial product bundling the open source components of the Protomaps ecosystem into a planet-scale, deployment-friendly solution for organizations.

It has a mere two components:

A serverless app that runs on either AWS Lambda @ Edge or Cloudflare Workers.
A global basemap vector tile dataset derived from OpenStreetMap and delivered as a single 62 GB PMTiles archive that lives on S3 or compatible storage.

With this combination, you get near-infinite storage space and scaling, edge caching for low latency, and security with SSL and CORS, powered by your existing cloud account and CDN.

Mantle is not a SaaS

I’ll emphasize here that this solution is not Software as a Service (SaaS). It is licensed for a flat fee and self-hosted. It lives in your own AWS or Cloudflare account, but involves zero server management, databases or certificates.

The software industry has shifted to hosted services because the diversity of on-premise installation induces fractal complexity. “Serverless” computing offers an elegant path forward: Mantle takes less than an hour to set up in your own account, and deployment of the CDN code is merely pasting a JavaScript snippet:

Differentiation in Maps

The market for map APIs is crowded. Google Maps pioneered selling a hosted API to developers, and has maintained its lead in data completeness and quality. Rivals to the Google Maps API such as MapBox and even AWS and Microsoft offer more affordable plans, but stick to the pay-as-you-go business model with a focus on the most profitable market segments.

While the Protomaps system has a lot of catching up to do in cartographic refinement, it has a key disruptive advantage over incumbents. A Mantle installation enables you to serve unlimited map tile traffic, incurring only the costs of the underlying storage and CDN. Unlike hosted services, you’re free to cache, transform, and re-combine the data as you see fit for your application with no vendor lock-in.

To illustrate the cost savings, every additional 1,000 users that load a map on Google Maps costs 7 USD. An additional million hits to Cloudflare Workers costs fifteen cents. Even taking into account the additional cost of storage and bandwidth, there’s a 10x to 100x difference in total costs.

Pricing

The introductory pricing for Mantle is a flat, per-site license of 950 USD a year, for integration into one End Product. You can continue to run Mantle after your one-year expires, but won’t have access to updated software, datasets or support.

Funding Open Source

Mantle is a capstone product for the ecosystem of Protomaps tools, covering a spectrum of technolology from spatial databases to TypeScript map rendering, as well as free, ODbL packaged geodata at Protomaps Downloads. All software components are published under commercial-friendly, permissive BSD licenses, and you can assemble a production-ready system from them.

Protomaps is currently a single-person, bootstrapped business, which removes the constraints of a typical tech startup. My intent is for the majority of mapping users — hobbyists, NGOs, corporations, everything in between — to use it completely for free. A first commercial offering is the entry point for businesses to support the development of this open source ecosystem, influence its future, and get expert support for their own environments and datasets.

Get Started

Mantle has been tested by early users for a few months, and I’m gradually onboarding new customers with personalized support. If you’d like to take advantage of the introductory pricing, fill out this short form and I’ll get in contact via email.

Rethinking the Free Tier for Maps

Wed, 23 Mar 2022 00:00:00 +0000

Protomaps is a new mapping system, and usually that implies a new map API - an HTTP service, hosted by a company somewhere Out There on the Web. Your typical interactive web map talks to an API using these conventions:

The world is split into a pyramid of square tiles, with objects simplified to one of many pre-determined zoom levels.
The browser fetches an image or other data tile as the user zooms and pans.

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

Front-ends like Leaflet or OpenLayers are mature libraries for consuming tiles, but leave unanswered where the tiles come from and whether or not those tiles cost money.

The Status Quo: Consumption Pricing

Map tiles from an API are just one of many developer Software-as-a-Service (SaaS) products, which onboard new customers like this:

Sign up and generate an API key.
Embed that key on your site, which you can use for free up to a certain limit.
Once you’ve exceeded that limit, you’re billed on a pay-as-you-go pricing model.

This pay-as-you-go model is also called consumption pricing. Good examples of this model in the developer market are Sentry for error tracking and Twilio for SMS. Google applied this model to Maps, and it is infamously expensive:

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

In theory, consumption pricing is good for both vendors and customers. It lets companies capture mindshare via a free tier: developers, more than anyone else, love free stuff, especially when learning new tools for personal side projects. It also lets them charge much, much more for use cases with high willingness to pay, such as heavily trafficked retail websites, popular social media applications, or enterprise deployments.

Why SaaS does not serve all map use cases

Most developer services that come to mind fit the consumption pricing model perfectly - indie projects pay nothing, startup companies buy a medium-sized plan, and heavy usage is priced in after clicking the Contact Us button. However, a lot of mapping use cases do not fit the consumption pricing model.

Firstly, many useful applications of maps, even those that are important infrastructure for society, have high traffic but low ability to pay, like civic technology projects, environment & climate, public health, journalism and information visualization. Projects are often built on the free tier of a map API but discover they can’t afford a higher tier after launch.

Secondly, the SaaS model means the vendor must track usage, which means applications must always connect to the public Internet. Developers could cache, proxy or scrape map data for offline use, but this is always forbidden by terms of service - see the Google Maps TOS as an example - because it contradicts the pay-as-you-go business model. Entire classes of applications are impossible due to policy, not technical feasibility. Mapping applications are always dependent on a third party, and exposed to changes in pricing or companies going offline on an Incredible Journey.

The narrow market served by pay-as-you-go pricing means that map vendors must cater to the most profitable commercial ventures, ignore exciting but unlucrative use cases, and enforce artificial scarcity of information.

Lastly, vendors rely on convincing customers that the service they provide is too difficult and complex to run themselves. Metered SaaS reshapes the architecture of systems to fit a business model. It creates strong disincentives for the creation of simple, modular software that can be adapted and re-used.

Can APIs be totally free?

OpenStreetMap is thought of as a “free” map service, due to ubiquitous confusion between the cartographic product served from openstreetmap.org, and the OpenStreetMap dataset. You’ve probably seen an embedded OSM map that looks like this:

OSM through its Foundation pays to host this for free, but for the primary purpose of aiding map editing and data creation, not widespread public embedding. The Tile Usage Policy plainly states: “OpenStreetMap data is free for everyone to use. Our tile servers are not.”

There does exist alternate map services like maps.stamen.com that provide a set of pre-rendered map images. But free services ultimately depend on the a sponsor willing to commit to the long-term cost of data transfer, because bandwidth costs are not trivial.

Holding the Bandwidth Bag

Free map hosting is difficult for the same reasons image and video hosting is difficult: letting anyone embed assets anywhere on the web leads to exploding bandwidth costs. Bandwidth, while a commodity, is not free - AWS charges about $0.09 per gigabyte.

Developers might believe that companies like Cloudflare offer unlimited bandwidth, but this is limited to HTML content and not media as described in section 2.8 of their Terms of Use. Rented bare-metal servers can come with unmetered network connections, but the total transfer is limited by speed, not quantity - incoming requests can saturate a single network interface but no further.

Slippy maps are sensitive to latency, so ideally a map can be served from a content delivery network, minimizing the time to serve a map to the end user. CDNs themselves follow consumption models, but strong competition between major players drives pricing down over time.

How Protomaps Is Different

Protomaps, runs a public API on the fast Cloudflare CDN, which you can use for free up to a soft cap of 1,000,000 requests per month. Like everywhere else, you can start by signing up for an account which will generate an associated API key.

But here’s a key difference: If I change this free tier in the future, I go out of business, you outgrow the service, or you just don’t want to depend on an external service anymore, you can export the map data and host it yourself for a limited area via Protomaps Downloads. This is much faster than scraping the API, and wraps the map into a tidy PMTiles package that lives on S3 with no intermediate server.

Obviously, there is no way for me to ever know how much you’re using a downloaded map, and thus no way for me to charge you based on that! You’ll still pay for your own storage and bandwidth, but those will be orders of magnitude cheaper than typical metered map APIs. Your map will work forever, even offline, because it transforms the map from a live service into just another asset like an image or video.

If you’re excited about the possibilities here for your project, you can contact me at brandon@protomaps.com.

The Surprising Anatomy of a One-Man Map Tech Stack

Thu, 06 Jan 2022 00:00:00 +0000

Protomaps is a independent project to build a new map of the world. This scale of mission demands a wide range of novel frontend and backend components, like an open source spatial database, a serverless tile archive format, and a vector map renderer. There’s also a web application — the one you’re looking at now — with subsystems to process background tasks, ingest metrics and manage objects on cloud storage. Finally, there’s the core map engine for cartographic generalization and tiling of OpenStreetMap data, which you can access at Protomaps Downloads.

Bringing a viable product to market, in addition to publishing open source software, creates some special pressures for a one-man shop. Chief among those is to make conservative technology choices. Building on stable tech means I can spend my risk budget on unique parts of the project instead of libraries; it means my open source components can be adopted by others with minimum friction and mature independently of my own use case. Rust and WebAssembly are exciting for maps, but a bootstrapped company can’t afford multi-year investments in every emerging low-level technology. In this context, some Protomaps choices raise eyebrows among web and GIS developers, so I’ll document a few of them here.

C++

C++ is the primary language at the heart of the Protomaps mapping system.

C++ isn’t valued here for any particular properties of the language itself. In aspects like consistent programming conventions or package management, it feels miserable compared to newer languages.

If you’re working with the nitty-gritty bits of geodata and computational geometry, C++ should be your first choice, because it’s what the vast majority of the ecosystem is already written in. You’ll have access to decades of battle-tested, robust libraries like glm for vector and matrix operations, GDAL/OGR for reading and writing geospatial data, S2 Geometry for spatial indexing on the sphere, and Clipper for computational geometry.

If you’re using a different language, you can write a from-scratch alternative to those libraries — which for each would be a multi-year investment without immediate payoff — or use wrappers, which might offer a mismatched API or lead you down a debugging hole that would not exist if you just interfaced with C or C++. In addition, a core goal of Protomaps is to make tooling that works at all cartographic scales, from a neighborhood to the entire planet. Using C++ and powerful libraries directly is the best way to efficiently process hundreds of millions of geographic features.

LMDB

All serious data storage and manipulation inside Protomaps uses Symas LMDB.

Most developers haven’t heard of LMDB, yet it’s the rock-solid foundation that contributes the most to making Protomaps work at scale. It’s an open source, multiprocess and fully ACID storage engine with some unique constraints, such as a single write transaction happening at a time. These constraints might limit it for general web applications but are perfect for Protomaps geodata, in which the only writes come from the OpenStreetMap replication stream.

Making LMDB Spatial

LMDB is an embedded storage engine for binary blobs, and is optimized for 64-bit integer keys. It has special functions for key-prefix cursor traversal. Protomaps uses spatial indexing schemes like S2 Cells where geographic features are bucketed into cells identified by a 64-bit integer, and cells share binary prefixes based on parent-child relationships.

cells on the hilbert curve by @enjalot

Memory-Mapping Matters

LMDB is memory-mapped as a fundamental fact of its design. The relevance of this beyond an implementation detail is not immediately obvious, but is revealed when paired with a system such as Cap’n Proto or FlatBuffers. Geographic features can be indexed by compact 64-bit keys, and attributes or geometry accessed with a pointer into virtual memory — a zero-copy operation — instead of wasting memory and CPU cycles on a deserialization step.

Geodata access patterns exhibit locality in geographic space or ID-space; arranging IDs on space-filling curves means high cache hit rates with zero application code, since virtual memory paging is handled by kernels.

LMDB: Highly Underrrated

LMDB has some mindshare overlap with storage engines like LevelDB and RocksDB, but does not have the implicit clout of being born within Google or Facebook, despite its superior B-Tree design for read-heavy applications. The API and documentation unapologetically assume you know what you are doing, and have a deep understanding of trade-offs for your problem space, such as mandatory sorted insertion order for fast write performance with MDB_APPEND. Its mmap-based, zero-copy design precludes compression, for example; for Protomaps, random I/O latency is much more critical to the product than space savings, and disks are cheap relative to other resources.

You can watch my talk on OSM Express at State of the Map and check out the source code on GitHub.
Mozilla’s Design Review: Key/Value Storage explores storage engines including LMDB.
LMDB author Howard Chu’s fiddle performance and technical deep dive at CMU Databaseology.

Go

Protomaps uses Go extensively as a complement to C++.

This is surprising to many developers who perceive Go as an alternative to C++. Go doesn’t have many mature low-level libraries for computational geometry. Whether libraries could even be competitive with C++ is an open question, given Go’s memory model and approach to generic programming.

Networking, HTTP and concurrency are solved problems in C++ via libraries like libasio but anything but easy to implement, especially as a solo developer. These are all trifling matters in Go to the point where they are first-class parts of the standard library. Go programs inside Protomaps avoid external libraries whenever possible and average about 100 lines of code. A typical example is to listen for HTTP requests, unmarshal JSON in the request body, and fork/exec a C++ program in a goroutine, reporting progress through a channel.

go-pmtiles is a library to read, write and serve the PMTiles serverless tile archive format.

Other Likes

The Protomaps web application uses Django, mostly for its batteries-included auth and administration modules, and good-enough server rendered HTML. Scripting and lightweight geoprocessing is in Python, via the excellent Shapely and Rasterio libraries. SQLite stores relational data like users and metrics. esbuild makes all TypeScript development a breeze.

Docker: Nope

source: flickr

Given the breadth of software needed to make maps happen, it is surprising to developers that I consciously avoid Docker and its ecosystem.

Docker is a container format and runtime. It is incredibly useful if you are building Heroku or dotCloud, since building a Platform-as-a-Service (PaaS) demands multi-tenant isolation. Protomaps is not building a PaaS, nor are most companies. Companies often use Docker as a glorified static linker, and in the process introduce redundant abstractions and unknown-unknown failure modes.

Docker for many in-house use cases solves a cultural problem rather than a technical one: the historic separation of software development and operations, where “code” is tossed over the wall for SREs to deploy and carry pagers for.

As a one-man shop doing both software development and operations, I can decree a priori that all programs target a specific Ubuntu, depend on a set of packages from the OS, and run with a unprivileged user and designated port via a systemd configuration. My day-to-day development is still on macOS, but this is treated as a environment secondary to the production one.

Revenge of Docker

It’s likely that a PaaS that Protomaps adopts is built on Docker, and might require some Docker incantations. It’s also possible that software Protomaps distributes to customers will be packaged via Docker, although statically linked, architecture-specific binaries are a better solution (goreleaser is my current choice for this).

PostgreSQL: Nope

source: flickr

Protomaps does not use PostgreSQL, despite it being the de-facto standard for storing and querying GIS data via the excellent PostGIS extension.

That is not to say PostgreSQL or PostGIS are flawed; they’re an exemplar of what open source geospatial tech can be and have had a massive impact on GIS in government. They are the perfect solution for relational geodata and multi-writer transaction processing over a network, neither of which are relevant to Protomaps.

Many companies use PostGIS as a convenient SQL frontend to its underlying libraries, GDAL/OGR and GEOS, when the problem could be solved in a simpler fashion by using those directly. By introducing a client-server database, automation becomes much more complex, involving ports, user authorization and process managers, instead of being self-contained in a single file like SQLite or LMDB. Protomaps’ problem of scale again means that fetching data over the wire is just too much overhead, and efficient data structures for cartographic generalization, like planar straight-line graphs, cannot be hammered into the relational model.

The pervasiveness of relational databases like PostGIS as a solution to all geo-shaped problems is, again, a cultural artifact. In this case, it’s the status quo of GIS as an academic discipline and industry: one dominated by a single company (Esri) and its workflows and assumptions, where “doing GIS” means manipulating a relational database, and software development is abstracted away to a vendor.

Other Nopes

Despite years wrangling Ruby on Rails in my Pivotal Labs days, I usually reach for Django for its standardized auth and admin. Server-side JavaScript is avoided; this means re-implementing logic in both JS and Go/Python, which I don’t mind. Java has a mature geospatial ecosystem with libraries like JTS and Osmosis, but moving onto Java is all-or-nothing, given the packging conventions and JVM cold start times.

What Next?

The emerging projects I’m most excited about are “cloud-optimized” formats for geodata, including FlatGeobuf, GeoPngDB, and COPC as complements to the format I’m developing, PMTiles.

If you’d like to keep up with Protomaps, or are interested in building and deploying custom, interactive maps of the entire world, you can reach me at brandon@protomaps.com or find me on Bluesky and Mastodon.

Dynamic Maps, Static Storage

Thu, 06 May 2021 00:00:00 +0000

Update: This post describes the internals of the PMTiles v2 format. Check out the post on PMTiles V3 for the latest!

Serverless computing supposedly will let developers run sophisticated applications on the web without wrangling virtual machines or databases. New products like Lambda offer simpler deploys for code; emerging standards like WebAssembly let you host SQLite databases on GitHub pages and make queries all from your browser. These examples rely on developers adopting new workflows or programming environments. So what if I told you that you’ve been computing serverlessly without realizing it?

Behold this one megabyte embedded video, a clip from Powers of Ten: A Film Dealing with the Relative Size of Things in the Universe and the Effect of Adding Another Zero:

Your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

There’s a couple things going on here:

You can seek to timestamps using the embedded controls to see the Chicago lakefront at different scales; consider this an act of querying a read-only database via an index.
The video is a single file, hostable anywhere, and playable by all modern browsers.
If you open your browser’s development tools, you’ll see requests for specific byte offsets in the file with Range: headers.

This simple example highlights the ease of use of video; A video is thought of as a single unit of content, like a text file or photograph, and not a kind of trendy serverless tech.

Codecs make Video Easy

A remarkable aspect of the above example is that it’s a single megabyte, despite being made of hundreds of individual frames. Assuming each frame is a 200 kilobyte still, storing the eight second video as a directory of images would take fifty times the space!

We can expose the secrets of video codecs using ffmpeg:

ffmpeg -flags2 +export_mvs -i in.mp4 -vf codecview=mv=pf+bf+bb out.mp4

White arrows depict motion vectors for blocks of pixels: the codec exploits redundancy across data to efficiently encode a single archive. This is possible because most videos are not random noise; the contents of a single frame are highly predictive of the next frames.

A Map is just a Video

At first blush, a dynamic map on the web isn’t much like a video and more like a web service; something database-shaped and probably server-ish, despite being a read-only experience. The Web Map Tile Service standard defines a HTTP server API for serving up maps; map tiles on openstreetmap.org are accessed by Web Mercator Z/X/Y URLs.

Given the above encoding example though, let’s imagine if dynamic maps, like videos, were just archives of images, but with frames that traverse space instead of time.

Armed with this insight, we can invent a codec for map data with these goals:

It should be a single file. There are existing workflows to host maps on S3 by syncing individual tiles and directories, but this is inconvenient in the same way syncing tens of thousands of still video frames is inefficient.
It should work the same way for all scales, from a small area map of the Chicago lakefront to the entire Earth.
It should be as simple as possible for the Z/X/Y use case, and hold arbitrary raster or vector tiles, which rules out backwards-compatible formats like Cloud Optimized GeoTIFFs.

A necessary step in making this efficient is, like video encoding, to exploit redundancy across data. Any global-scale map system must acknowledge the earth is 70% ocean. Map image APIs already take advantage of this by special-casing solid blue PNG images.

PMTiles: Pyramids of Map Tiles

PMTiles is a single-file archive format for pyramids of map tiles that aims to be the simplest possible implementation for the above requirements. It has a short and sweet specification that I’ve made available in the public domain, along with BSD-licensed Python and JavaScript reference implementations.

Here’s a visual for how a simple PMTiles archive is organized:

The process for fetching one tile like z:8 x:65 y:95:

Fetch the first 512 kilobytes and parse the directory into a lookup table
Look up by key of the tile you want, in this case 8_65_95
It’s a match! You’ll have a byte range offset: 785366 length: 21400
Fetch those bytes using Range:bytes=785366-806765 and interpret the data as an image

Here’s an interactive example using the sub-200-line pmtiles.js decoder library in the browser, with Creative Commons satellite imagery from EOX’s S2 Cloudless product, containing some tiles covering our happy picnic on the Chicago lakefront:

pmtiles.getZxy(coord.z,coord.x,coord.y).then(result => {
 let range = 'bytes=' + result[0] + '-' + (result[0]+result[1]-1)}
 fetch('PowersOfTwo.pmtiles',{headers:{Range:range}).then(resp => {
 return resp.arrayBuffer()
 }).then(buf => {
 var blob = new Blob( [buf], { type: "image/jpg" } )
 var imageUrl = window.URL.createObjectURL(blob)
 img.current.src = imageUrl
 })
})

The initial fetch for the first 512 kilobytes is performed once and cached in the browser, so subsequent requests don’t incur the same overhead as the first tile. This again exploits the common panning+zooming access pattern, like what’s happening in this integration with Leaflet:

A consequence of the above design is that multiple entries can point to the same offset. A map of an island surrounded by lots of ocean will store our solid blue image once, with any number of tile entries referencing it. Multiple requests for this same offset will return our Blue Friend cached in the browser:

Adding more Zeros

The above example has a clear limitation, which is that the blind initial fetch is for only 512 kilobytes. A reasonable planet-scale map can have fifteen zoom levels; the sum count of tiles then is

1 + 4 + 16 + 64 + 256 + 1,024 + 4,096 + 16,384 + 65,536 + 262,144 + 1,048,576 + 4,194,304 + 1,6777,216 + 67,108,864 + 268,435,456

or 357,913,941 tiles. Given that PMTiles needs 17 bytes per entry, a full directory pyramid with 15 zoom levels is six gigabytes - impractical for any browser to fetch at once, especially if your map will only ever display a couple tiles!

We can solve this by adding a layer of indirection via leaf directories. A single directory in PMTiles has a hard cap of 21,845 entries, or a complete pyramid for eight zoom levels. At the maximum level of the root directory, the top bit of Z is flipped to describe the bytes at the referenced offset being not tile data but another directory with up to 21,845 entries.

Example of how looking up z:14 x:4204 y:6090 works with leaf directories:

Fetch the first 512 kilobytes, parse the root directory.
Check the root directory for 14_4204_6090. It doesn’t exist, because the root directory only contains levels 0-7.
Check the root directory for a leaf directory entry of the parent tile. The z7 parent tile of z:14 x:4204 y:6090 is z:7 x:32 y:47.
Fetch the bytes for the leaf directory and parse it into a lookup table.
Index into the leaf directory with 14_4204_6090 for the byte offset and length of the tile data.

As long as clients cache recently used lookup tables, and zooming/panning remains within z:7 x:32 y:47, no directory fetches are required after the first tile. The ideal ordering of tile data and leaf directories may be on space-filling curves to minimize hardware page faults, but this is an optimization and outside the spec.

An astute reader will notice another seeming flaw in the above design: what if your pyramid is exactly a full nine levels? In this case, the root directory has a full 21,845 entries, and points to 16,384 leaf directories, each containing only five (one Z7 + four Z8) entries! Panning the map at Z7 would then incur a directory fetch for every single tile.

Inefficiently packed leaf directories can be solved the exact same way as duplicated ocean tiles. One leaf directory should contain multiple sibling sub-pyramids, up to 21,845 entries in total; the root directory then holds multiple leaf pointers to the same offset. As long as clients cache directories by offset and not ZXY, panning at Z7 will cost few extra fetches.

Here’s the limits implied by the 17 byte keys in PMTiles:

3 bytes for X and Y: maximum zoom of 23
6 bytes for offsets: maximum archive size of 281 terabytes
4 bytes for lengths: maximum individual tile size of 4.3 gigabytes

go-pmtiles: a PMTiles proxy

If you’re thinking, Yes Brandon, very clever design! We want to use this but all of our existing map systems read Z/X/Y tiles from a server… I’ve written a caching proxy that decodes your PMTiles on S3 via a traditional Z/X/Y endpoint. While no longer purely serverless, this is a two-megabyte executable that serves an entire bucket of PMTiles on a micro instance like this:

# to serve my-bucket/EXAMPLE_1.pmtiles and my-bucket/EXAMPLE-2.pmtiles...
go-pmtiles https://my-bucket.s3-us-east-1.amazonaws.com
> Serving https://my-bucket.s3-us-east-1.amazonaws.com on HTTP port: 8077
# fetch the tiles...
curl http://localhost:8077/EXAMPLE_1/0/0/0.jpg
curl http://localhost:8077/EXAMPLE_2/1/0/0.jpg

The details:

Inherits all the simplicity of PMTiles: a short, concurrent, evented Go program with zero runtime dependencies
Caches directories on the server to minimize latency on the client
Implements GZIP compression for uncompressed formats like vector Protobufs, since byte serving and Content-Encoding don’t play nicely together

You can download binary releases and source code at github.com/protomaps/go-pmtiles.

More Stuff

PMTiles is just one small part of Protomaps, a one-man project to rethink web cartography, and reflects many of the system’s core principles:

Less computation: “Serverless” patterns are here to stay: vendors achieve economies of scale; developers pay less; mapmaking projects outside of dedicated technology firms — by museums, cultural institutions, or journalists — can be preserved across time.
Dependency-free: Favor minimalist formats and statically-linked binaries over more features or backwards compatibility.
Scale-free: Tools and process knowledge for making an airport-sized map work the exact same way for a city, a country, and the entire Earth with no jumps in complexity.

If this system sounds interesting to you, you can reach me at brandon@protomaps.com or find me on Bluesky and Mastodon.

A new way to make maps with OpenStreetMap

Thu, 22 Apr 2021 00:00:00 +0000

If you develop for the web, you’ve probably heard of OpenStreetMap, an open data alternative to Google Maps. Despite the wide name recognition of OSM, you may have questions like how do I make a web map with OSM data? and how can I host maps myself?

There are popular guides on how to work with planet files, PostGIS databases, and run tileservers for the OSM “slippy map” style. Even if you figure those out, you may be curious about vector maps for high-DPI sharpness and client-side customization of appearance and labels.

Protomaps is a new basemap system which is an end-to-end rethinking of this entire stack, oriented around the idea that custom mapmaking should be simple.

In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide.

Building Blocks

Web maps are a “full-stack” challenge: Aesthetic goals, such as a sensible arrangement of map labels, need to be supported not only by JavaScript rendering but also the underlying data backend. Protomaps is designed as an end-to-end system for this reason — see What is Protomaps? — although each part is interoperable with other tools.

As of now (April 2021), each component is viable and being used in production; I’ll describe how each fills a specific gap in the map ecosystem, as well as the direction I am developing it towards.

protomaps.js: a vector map renderer and labeler for the web

protomaps.js on GitHub

Where it fits

There’s powerful WebGL-based map renderers like MapboxGL JS, Tangram and harp.gl for vector map tiles. Despite this, the Leaflet library remains extremely popular because of its simple, DOM-driven design. Protomaps.js brings high-DPI vector rendering and map customization to the Leaflet map experience, with some deliberate compromises: see what protomaps.js is not.
Protomaps.js is also a system for code-driven map symbology. While the above WebGL libraries and image-based renderers like Mapnik expose a limited API for map labels and markers, protomaps.js is extensible at runtime by implementing custom Symbolizers; it’s an immediate-mode system for direct access to Canvas.

What’s next

protomaps.js is alpha stage and needs a lot of work — it’s being used to power all the maps on this site, but the internal API is still unstable. I am open sourcing it now to gauge interest in a pure 2D webmap renderer: protomaps.js on GitHub
Labels are currently, well, bad. Linear feature labels are mostly broken; multiple candidate positions should be considered for point labels like cities and POIs. Some features, such as country-specific highway shields, will need to be supported by the vector data from the Downloads service and web map API.
Support SVG icons via the protosprites library and port popular open source map styles such as Stamen Design’s Toner.
All features need to be designed with performance as a primary goal: while a pure Canvas-based experience will never be as slick as a WebGL one, protomaps.js aims to be just as good as “slippy” image maps.

PMTiles: a serverless format for map tiles

Spec and Reference Python/JS implementations on GitHub

Where it fits

Vector map data in the form of Z/X/Y tiles is typically made available as a commercial API. If you don’t need the entire planet, it should be possible to host a piece of the world yourself.
Existing formats like MBTiles are based on SQLite. This limits the ability to self-host maps to those confident running a server in production.
PMTiles is a format based on HTTP Byte Serving that can be hosted on S3 or any other cloud provider. It can be read directly by the protomaps.js renderer for a totally serverless map application. It’s a perfect match for projects where a long-term subscription to a hosted service is undesirable.

What’s next

Try the PMTiles specification and reference implementations for your own projects.
More implementations of decoders. In the works: a concurrent, caching decoder for Go.

Map Downloads & Web API

Downloads | Web API Documentation

Where it fits

Eliminates all complexity from getting map data: The current status quo is to use PostgreSQL as an intermediate store for OSM data, and to render tiles from PostgreSQL on-demand. Instead, download pre-rendered PMTiles from the Downloads site, or use the global Web API.
Map Downloads (but not web API) snapshot OSM every 60 seconds: this turns OpenStreetMap into a truly live, editable project. This resolves the mismatch between the idea that the map can be improved by anyone, and the reality that map services are weeks or months behind the live data.
For those needing a global hosted service, you can sign up for an API key.

What’s next

Scaling up: Map downloads are currently limited to 500,000 nodes to help isolate bugs - for example, sometimes the ocean “leaks” into land. If you find problems or have questions, please email me.
Map downloads and the Web API should support high-quality label generalizations, highway shields, and beautiful graph-based simplification of road networks.
The Infinite Map: PMTiles downloads should be customizable to include any object or tag in OpenStreetMap, via the OSM Express database.
OSM as a live project also comes with risks related to vandalism or just broken data. Monthly releases of Facebook’s Daylight Map Distribution should be an option for Map Downloads and the Web API.

OSM Express: a database for OpenStreetMap

Documentation | Minutely Extracts

Where it fits

Relational databases are the most popular way to work with OSM, but demand a lossy transformation of the data model. OSMX is a embedded database for nodes, ways and relations to make building applications on fresh OSM data simple. Check it out on GitHub.
All Protomaps services are built on top of OSMX; it’s orders of magnitude faster than alternatives. It’s being used by other companies for tasks like generating diffs. Minutely Extracts provides fresh PBF format data for use in other applications like routing and geocoding.

What’s next

More OSMX reader libraries in other languages.
Applications for visualizing OSM activity in near-realtime built on OSMX.

What’s next for Protomaps

Aside from the existing parts above, there’s other features on the roadmap:

Terrain: protomaps.js should render client-side hillshades and contour lines, and Terrarium-format tiles should be downloadable as PMTiles archives, based on SRTM, ASTER or a mosaic dataset.
Satellite: A cloudless Landsat or Sentinel-2 imagery layer is a great match for canvas-rendered, custom map labels.

I want to end by clarifying that Protomaps is a self-funded, single-person product, executing on a distinct vision of what web cartography can be. If your company or project has a mapping challenge I might help you solve, email me at brandon@protomaps.com or find me on Bluesky and Mastodon, and I’ll of course be publishing on GitHub.

You’ll see more on this blog soon!

🗺️

Zonal Statistics for OpenStreetMap in the browser

Mon, 16 Mar 2020 00:00:00 +0000

A common user interaction in GIS applications is selecting a bounding box or polygon:

Your application may then use this polygon to do things like:

Render a high resolution, printable map image.
Generate a report of the surface area of buildings.
Calculate an isochrone map of distance to transit.

Users of your service expect that any action will complete in a reasonable amount of time. It’s smart to build in a limit for the amount of data processed on the server. One simple way to do this is an area limit, for example: 100 square kilometers.

Using total area doesn’t work well for datasets of man-made objects like OpenStreetMap. The density of features has huge variance in different parts of the world. In the image above, the area on the left is only part of Paris and contains about 10 million nodes. The area on the right is most of the U.S. state of Minnesota, and also has about 10 million nodes. A system limit designed for the densest areas would be too restrictive.

Density

Instead of having an area limit, we can first test the area for the count of features, and then provide feedback in the UI if the limit is exceeded. Common ways to count are:

For data in PostgreSQL, query over a spatial index using a function like ST_Intersects, returning a row count.
For OpenStreetMap, use Overpass API to count nodes or features matching a tag, returning a count output.

This information can be exposed to a web browser via API.

Faster, Simpler

The previous approach has the major drawback of needing a remote, blocking server call. We can instead design a quicker way to estimate the number of features without a dynamic query. This allows for near-instant feedback in the UI, and less moving parts, at the cost of being a coarse approximation.

The GeoTIFF format is a natural fit for this problem. It’s readable by all GIS libraries, supports compression, and embeds a geographic projection.

Each pixel in this image is one level 12 Web Mercator Tile, and each sample is a 16 bit unsigned OpenStreetMap node count (in thousands). This is too dark to be visible to the human eye, but normalizing the color range results in this image:

The resulting compressed GeoTIFF is only 212 KB, which is reasonable to deliver to web browsers. The source code for generating this GeoTIFF is available at github.com/protomaps/OSMEstimator.

Calculating Estimates

Any zonal statistics library can be used to compute the sum total of an area of interest over the GeoTIFF’s samples. The GeoBlaze library implements this in JavaScript. The Rasterio library implements this in Python. Example of computing a sum in Python:

import pyproj
from rasterio import mask
from shapely.geometry import shape

RASTER = rasterio.open('osm_nodes.tif')
PROJECT = partial(pyproj.transform,pyproj.Proj(init='epsg:4326'),pyproj.Proj(init='epsg:3857'))
transformed = transform(PROJECT,shape(geom)) #geom is a GeoJSON-like dict
masked = mask.mask(RASTER,[transformed],all_touched=False)
estimate = masked[0].sum() * 1000

Limitations

The choice of zoom level 12 and the accuracy of the estimation rely on query areas being significantly larger than the pixels of the GeoTIFF. Different applications may want to modify this code for:

Specific features: Modify the program to only consider ways tagged building=yes, instead of all OpenStreetMap nodes.
If the relevant area is only part of the world, the GeoTIFF size can be reduced by cropping the raster.
Trade-offs between estimation accuracy and file size can be controlled by using 8, 16 or 32 bit samples.

Protomaps Blog

New CLI feature: pmtiles merge

Requirements

Disjoint inputs

Recommendations

Future work

Mapterhorn - Terrain for Web Mapping

Project Inspiration

Key project differences

Project Future

A Tour of the New pmtiles.io

The Map view

The Tile View

The Archive Inspector View

Covering the Planet with Compressed Bitmaps

Polygon Rasterization

Data Structure

Algorithm

Results

Code

What is a basemap, anyway?

Basemap Origins

Mapzen

Moving Web Maps Forward

Protomaps Basemap

Project Status

Slicing OpenStreetMap Just Got Fresher

Protomaps Extracts

New project: SliceOSM

Project Structure

Project Alternatives

PMTiles on More Platforms

Style Editor

OpenLayers

AWS CloudFormation

Google Cloud and Azure

Server

Caddy

You Might Not Want PMTiles

GeoJSON

PostGIS

PostGIS on Supabase

GeoParquet

Lonboard

Protomaps receives NLnet open source grant

Motivation

Maps for the Public Sector

Localization

Security & Privacy

Collaborators

Read More

Transitioning Protomaps from Open Core to Open Source

What is Open Core?

FOSS vs. SaaS

The Open Core Experiment

Leaving Open Core

Sponsor-supported SaaS

New Commercial Distribution

Protomaps Access

See Also

What is a clustered PMTiles archive?

Clustered

De-duplication

Locality

Directory delta encoding

Determinism

pmtiles verify

Recommendations

Serverless Maps - Now Open Source

Accelerating Maps

Used in Production

Cost Comparison

Free Datasets

Vector Tiles (zooms 0-10)

Terrain RGB

Copy these datasets!

What's new in PMTiles V3

Spec version 3

File Structure

JavaScript