A user on the OSMand group asked about my process for generating my own tiled maps, and for importing parks and preserves.
The topic, I thought, deserved a decent writeup, so I decided to work my reply into a post here.
Workflow of getting tiles into Backcountry Navigator
I've got a reasonably capable (quad Core i7, 32 GB memory, half a terabyte of SSD, 4 terabytes of RAID 0+1) Linux system at home, where I host a PostGIS database. By far the biggest part of that database is taken up with the OSM North America export from geofabrik.de, which I initially loaded with osm2pgsql, and synchronize with GeoFabrik's nightly diffs using osmosis. I retain the 'slim' tables because I have scripts that need them.
Starting from Lars Ahlzen's TopOSM, I've set up a Mapnik rendering pipeline that uses OSM, the National Elevation Dataset, the USFWS national wetlands inventory, the National Hydrography Dataset, the National Landcover Dataset, and a bunch of state and local databases to assemble an American-style topo map that shows much of the information that I want to see. I've set up a previewer for that map at
https://kbk.is-a-geek.net/catskills/test4.html
(Feel free to pan and zoom.) What I'm set up at present to render is the US east of a line from roughly Atlanta to the Mackinac Strait, and north of Atlanta. (That's because I set it up partly for some correspondents of mine who are interested in maintaining information for approach trails to the Appalachian Trail, and the area I render is roughly the bounding box of that trail.
The map is augmented with information that I got from a number of state and local GIS departments. For example, this view shows a number of trails in magenta. Those are trails that I got from NY State Department of Environmental Conservation's GIS department. I do NOT import those because
- There are license incompatibilities
- The data are stale, and were originally digitized from inappropriately small scale maps. In some places they're quite good indeed, but in other places they're way off.
I find them to be a useful indication of "there is a trail somewhere near here", and a "to do" list for trail mapping with GPS.
For the most part, those auxiliary data came in the form of shapefiles, and got imported into more tables in the PostGIS database. The Mapnik source for the map as you see it has dozens and dozens of layers.
Since I've served the map up, I can tell BackCountry Navigator to use it as a web map, much as it would use the US topos from ArcGIS, or OpenStreetMap tiles, or Bing aerial imagery. The URL for the map is
https://kbk.is-a-geek.net/catskills/tiles/{Z}/{X}/{Y}.jpg
(Please don't incorporate into apps to re-share, I have limited bandwidth and even more limited time to support the thing. Also, ''please'' don't try to bulk-download all the tiles! If you need large amounts of map, email me and we'll work something out.)
Since BackCountry Navigator supports downloading the tiles for an area in advance of a trip, I download from my home Wi-Fi before I go, and run happily without cell service in the woods.
Importing parks and preserves
The imports of parks and preserves have been several projects, each with its own workflow.
New York City watershed recreation
I did an import of New York City Watershed Recreation Lands. For that, the city made available PostScript maps of each of its facilities (note that these are located outside the city, protecting the watershed lands in the Catskill Mountains that provide New York with its water). It turns out that these PostScript files were already georeferenced, and that the names of layers in them were predictable, so I was able to set up a script that downloads them one at a time, scrapes out of the file just the boundary of the facility, and pushes the facility boundary into PostGIS.
The script got rather complicated, because it had to check that it wasn't overwriting data that are already in OSM, repair topology of the polygons, simplify the ways, shrink the polygons back a short distance to avoid collisions, and similar tidying operations. I also developed a mapping between the descriptive attributes in the shapefile and OSM tagging.
All the scripting was done in Tcl/Tk, for no better reason than that I'm familiar with it through having used it for about 25 years.
I proposed the import on the OSM Wiki and went through the usual storm and fury on the 'imports' mailing list.
The eventual import was done by taking the data, one parcel at a time, and using the JOSM remote control interface to push the polygons into JOSM. I did a final eyeball check for each, and committed them to OSM.
I've since revisited the import once, picking up 10 new purchases, 25 boundary changes, and six modified sets of access restrictions.
New York State Department of Environmental Conservation lands
Emboldened by this experience, I took on reworking the seven-year-old import of the NYS Department of Environmental Conservation Lands shapefile. It was a similar workflow: pour the data into PostGIS, tidy up the geometry and topology, map tags, and so on - but on a much larger scale, and starting from a single shapefile rather than several hundred PostScript maps.
Once again, since I had automatable data, I did this one as a formal (re)import proposal
This was again a parcel-by-parcel effort in JOSM, but this time, there was much more manual work, since there were existing versions of the parcels that had to be conflated. It was a pretty hellish job, completed in off-and-on evening work between May and September of 2016. The hardest areas to handle were ones where complicated shorelines formed the boundaries of reserves; Saranac Lakes Wild Forest and Lake George Islands were ones that I recall as being particularly tricky.
Conflation was also tricky if the parcels shared ways with adjacent landuse or landcover polygons. In the worst cases, I simply left the original polygons in place, but removed the tagging identifying the land as state forest, and then overlaid with the protected area.
Once again, now that I keep after it every year or so, the modifications are more straightforward. I reimported again a couple of months ago and managed to do it in a couple of evenings.
New York State Parks
I then moved on to New York's State Parks. Note that the Adirondack Park and the Catskill Park are parks owned by the state, but they are not State Parks; instead they are entities unto themselves, enshrined in the state constitution.
Each of the state parks has a georeferenced PDF trail map available from New York State Office of Parks, Recreation and Historic Preservation. Unlike the New York City PDF's, there were no vector layers for me to scrape. Moreover, the license status of the state park maps is unclear, and I live in the one Federal Circuit where government entities can claim copyright to data such as these. Instead, I treated the PDF's as a 'to do' list of parks that needed to be mapped.
For each of these, I did the following:
Converted the PDF to a GeoTIFF for efficiency, and loaded the PDF into Quantum GIS. (QGIS can read GeoPDF, but becomes unusably slow when it does.)
As a separate layer, loaded up a shapefile of tax parcels owned by New York State. This shapefile has license terms compatible with ODBL - the public has the right to use the data for any lawful purpose.
Selected all the tax parcels that were coterminous with the park. This could be as few as one or as many as several hundred.
Conflated the parcels and repaired the topology. (This was a fair amount of manual patchwork.)
Exported the tidied parcel from QGIS as a shapefile.
Opened the shapefile in JOSM and downloaded the OSM data.
Added tagging. For this, I wound up developing a couple of JOSM presets for 'New York State Park' and 'New York State Historic Site', and did a bunch of copy-and-paste of things like park names, web sites, and telephone numbers from parks.ny.gov.
Conflated with what was already in OSM. A lot of state parks were already there, with somewhat whimsical boundaries. If the boundaries were from TIGER, I had no qualms about overwriting them.
If the boundaries were actually provided by a local mapper, I tried to get in touch with the mapper in question and find out how they were obtained. The mappers were very cooperative, indeed, and got back to me promptly. In virtually all cases, they had traced approximate boundaries from Bing and were happy to have the ones from the tax rolls.
Again, there were adjacent-parcel issues, and again, I sometimes resorted to overlaying the protected area and leaving existing landcover polygons (and adjacent landuse polygons) alone.
I didn't call this one an 'import'. I was comfortable with not doing so. There was far too much manual work involved for it to fall under the definition of 'automated edits.' Everything that went in had been touched with eyeball and mouse. Nobody complained. It is more blessed to beg forgiveness than to ask permission.
Other land areas
I used the same technique, with different source datasets, to fill in a number of county and municipal parks, and some private preserves. This is a work in progress, there's always more to be done. The most recent ones that I brought in were just this past weekend (2017-07-03), with a few more parcels belonging to the nonprofit Mohawk Hudson Land Conservancy.
(I still need to get out to these and GPS the trails!)
That's also how I sorted out the unholy mess of overlapping polygons for West Point, four state parks (Bear Mountain, Harriman, Sterling Forsest, Schunnemunk, Storm King), the Federal corridor for the Appalachian Trail, a private, open-to-the-public preserve (Black Rock Forest), the villages of Harriman, Woodbury, Fort Montgomery and Stony Point, the Woodbury golf course, and the Hudson River riverbank. What a tangle that was!
TL;DR
The one-line summary: "It's never easy, is it?"
No comments:
Post a Comment