Saturday, January 7, 2012

Making maps in New York - hydrography

We left off last time with a map that has the street grid (well, sort of, it still needs to be styled!) and the contour lines (so that if I'm looking at a street or trail, I can tell how steep it is).

The next key piece of information that I need for a walking map is whether I'll get my feet wet: where is the water on the surface? So, let's look at some data sources for hydrography.

The hydrologic features are, we already saw, in the digital raster quadrangles, but (I repeat yet again) we don't want to use them, because they don't scale, and because they're cut off at the margins. So let's get vector data instead.

Fortunately, for New York, surface water features are "one-stop shopping." New York State's Office of Cybersecurity has two shapefiles, one with line features and one with area features, that cover the entire state and are suitable for rendering at 1:24000.

After downloading and unpacking them, I put them in the PostGIS database and spatially indexed them:

$ /usr/lib/postgresql/9.1/bin/shp2pgsql -I -c AreaHydrography.shp nys_area_hydrography \
    | psql -d gis > /dev/null
$ /usr/lib/postgresql/9.1/bin/shp2pgsql -I -c LinearHydrography.shp nys_linear_hydrography \
    | psql -d gis > /dev/null


I wound up loading two layers for the linear features. One of them had the query condition:

name IS NOT NULL AND name != 'Stream'

while the other had:

name IS NULL OR name = 'Stream'

The reason for this was so that I could assign labels only to those streams not named 'Stream': that name appears on more than half the streams in the database!

I also did some styled lines for canals, intermittent streams, and so on. I'll put the QML files at the end of this post.

So now there is open water in my map. I still want to do some more hydrography, because my feet can get just as wet in mud! Swamps are important to indicate, too! Since the area that I'm immediately interested in mapping has no salt water, and since it's outside the Adirondack Park, the New York State Department of Environmental Conservation's list of protected freshwater wetlands should give me quite a reasonable starting point.

Those files are available by county. Downloading my county and several of its neighbours, I tried a quick import into QGIS and into PostGIS. For the E00 file format that was used, both imports failed silently, leaving a set of malformed polygons for the swamp areas. Fortunately, I was able to find a workaround. I used the program 'avcimport', part of the AVCE00 package, to convert the .E00 files into ARC/GIS binary coverages, and then used 'ogr2ogr' to import them:

$ for f in `find . -name \*.e0? -print` ; do  ~/avce00-2.0.0/avcimport $f `basename $f .e00`-imp ; done
$ ogr2ogr -f "PostgreSQL" -s_srs EPSG:26918  "PG:dbname=gis" 001fwa-imp/  -nln nys_wetlands_arc arc
$ ogr2ogr -f "PostgreSQL" -s_srs EPSG:26918  "PG:dbname=gis" 001fwa-imp/  -nln nys_wetlands_pal pal
$ ogr2ogr -f "PostgreSQL" -s_srs EPSG:26918 -append "PG:dbname=gis" 057fwa-imp/  -nln nys_wetlands_arc arc
$ ogr2ogr -f "PostgreSQL" -s_srs EPSG:26918 -append "PG:dbname=gis" 057fwa-imp/  -nln nys_wetlands_pal pal
... continue for several more counties ...

Yet another task that I should have scripted!

Anyway, adding the resulting 'nys_wetlands_pal' and 'nys_wetlands_arc' in QGIS and doing a bit more styling gives quite an attractive result. I extracted an SVG of a swamp fill pattern and used that as the fill pattern for the polygons.

So now there is a lot of hydrography on the basemap. Next step is to start adding some cultural and administrative layers.
Read more...
Posted by Another Kevin at 10:02 PM 0 comments
Labels: , ,

Making maps in New York - topography

When we left off, we had a map project that had OpenStreetMap data in a PostGIS database, If the symbols were properly styled, having this data set alone could produce rather a pretty map. But it's far from ideal as a walking map - for one thing, I want to know when I'm walking whether I should expect to be going up and down steep hills. Of course, the customary way for a map to present this information about topography is to mark up the map with a series of contour lines - lines joining points at the same elevation.

We saw before that we could get contours from the 'topo' layer of the NYS DOT quadrangles, but we also saw that doing so had problems, because (1) the quads appear to be trimmed off at the margins, and (2) the resulting images scale poorly because they are raster images. So let's attempt a different data source.

The US Geological Survey publishes a Digital Elevation Model (DEM) of the United States, meshing it into a fine grid (the point spacing is 1/3 of a second of arc) with the elevation reported at each point. The CUGIR server makes the data available by quadrangle. So we begin by downloading the model for the quadrangles of interest, and unpacking the resulting files. (They wind up placing a ZIP within a ZIP, so have to be unpacked twice.) The result is that we have a bunch of files with the extension '.dem' that have the required data.

If the Python-GDAL tools are installed, then QGIS can take these data and produce contour lines in vector format. These lines will scale gracefully with changes in map scale. The way to get them is to open up a fresh QGIS project (or it's all right temporarily to add layers to an existing one), and load in a DEM file as a raster layer. When it's loaded, right-click on it in the 'Layers' display and choose 'Set Project CRS from Layer' (if you're working in an existing project, it will have to be set back afterward.) This will change the co-ordinate reference of the project to EPSG:26718 (NAD27 Universal Transverse Mercator Zone 18N), which we need because the DEM files use the old 1927 North American Datum coordinate system.

Once you've loaded the layer and adjusted the coordinate frame, choose 'Zoom to Layer' from the view menu. You'll see a gray rectangle that shows the part of the ground that the model covers.
You can color this area according to elevation, if you like, by right-clicking on the layer, choosing 'Properties' from the menu, selecting the 'Style' tab and changing the 'Color Map' to
'Pseudocolor'.
Selecting pseudocolor

Now you'll see the same area in false color: reds are high elevations, and blues are low.
Pseudocolor elevations

Since I'm an American, I tend to prefer maps with contours in feet rather than the more sensible metres. The DEM files have metric elevation, so to produce foot contours, we need to rescale them. Choose 'Raster Calculator' from the 'Raster' menu, and carry out the steps in the picture:
Changing elevations to feet

  • Enter the name of a GeoTIFF file to be created that will contain the elevation data in feet

  • Click 'Current Layer Extent' to make the new file have the same dimensions as the old

  • Set the output format to 'GeoTIFF' and select 'Add result to project' so that you can preview the conversion

  • Use the left-hand list to choose the layer to convert, and complete the expression (layer / 12.0 / 0.0254), which converts feet to metres

  • Click 'OK'


    • You'll get another gray rectangle that you can convert to pseudocolor to determine that the metres-to-feet conversion was successful.

    Now you're ready to produce the actual contour lines. Carry out this task by pulling 'Raster->Extract->Contour' from the menu bar. You'll get a dialog box:
    Making files with the contour lines

  • Select the feet-based raster that you just created as the input file

  • Choose the location of a new 'shp' file (together with the other files that go with it) as the output file. The file must not exist, but the containing directory must exist.

  • Enter an appropriate contour interval. For the rolling hills of upstate New York, at scales 1:24000 and finer, 20 feet is pretty good.

  • Check 'Load into canvas when finished' to be able to review the work

  • Click 'OK.' Don't close the window yet.



    • You should now see contour lines overlaid on top of the color image - uncheck both image layers in the list to make them more visible. If they look good, then we will also want index contours - highlighting of every fifth contour line to make them easier to read. Go back to the contour dialog, change both '20's to '100's, and click 'OK' again. You'll get a second set of contours overlaying the first.

    These steps have to be done for every DEM quadrangle that you download. Given the title of this blog, and my penchant for scripting languages, I really ought to script this workflow. But I haven't done it yet (partly because I'm not positive that I have it completely right).

    Once you have all the contours, it's advantageous to put them in the database. (If you have a large number of contours that don't appear in a particular view, QGIS can use database queries to filter them, and render much faster.) For this, we go back to the command line, and do:

    For the first file:

    $ /usr/lib/postgresql/9.1/bin/shp2pgsql -c r46elu_100ft.shp kbk_100ft_contours \
         | psql -d gis > /dev/null

    (The -c asks to create a database table afresh, and the remaining arguments are the first .shp file and the name of the table.)

    For each subsequent file except the last:

    $ /usr/lib/postgresql/9.1/bin/shp2pgsql -a r46elu_100ft.shp kbk_100ft_contours \
         | psql -d gis > /dev/null

    (The -a asks to append to an existing table in the database.)

    And for the last file:

    $ /usr/lib/postgresql/9.1/bin/shp2pgsql -a -I r46elu_100ft.shp kbk_100ft_contours \
         | psql -d gis > /dev/null

    (The -I option asks to create a spatial index over the layer after loading it.)

    These steps must be carried out for both 20- and 100-foot contours. Once again, I ought to script it. But I haven't done that yet.

    So now, start with a clean slate again in QGIS, and do 'Layer->Add PostGIS layer', selecting the two layers we just built. Remember to use EPSG:26718 as the layer coordinate reference system! These contour lines will be the ones we'll have on the finished map, so we might as well style them prettily. Bring up the 'Properties' on the 20 foot contours, change the color to brown and the line width to 0.18 mm (1/2 point). Then bring up the 'Properties' on the 100 foot contours, and change the color to brown and the line width to 0.35 mm (1 point). Finally, select the 100 foot contour layer, and pull 'Layer->Labeling'. Settings that I find attractive are:


    • Check 'Label this Layer' and select the 'elev' field

    • Choose the font 'Liberation Sans' 8-point italic, in the same shade of brown as the contours.

    • Use a 1-mm buffer

    • In the 'Advanced' tab: Placement parallel, on the line, at a distance of zero, oriented to the line. Medium priority. Suppress labeling for features smaller than 10 mm. Features don't act as obstacles.

    • In the 'Engine Settings' dialog, 'Popmusic Tabu' seems to give a trifle better placement for the contour labels than the default 'Chain'.



    So now we have a serviceable set of contour lines.
    Contour lines

    Don't throw out the original raster images, we may be reusing them to do hill shading later.
    In the meantime, we'll move onward to hydrography in the next installment.
    Read more...
    Posted by Another Kevin at 7:45 PM 0 comments
    Labels: , , , , ,

    Making maps in New York - the street map

    To put together any sort of map that tells a story, you need two kinds of mapping data. The first is a basemap: a general picture of the land, and the second is the thematic data: the stuff you add to the basemap to tell your story. I'm not quite ready to tell the story of Niskayuna's nature preserves yet, but at least I know where they are, and I've found sources for basemap data. Let's have a look at some workflows for putting the basemap together.

    Andy Arthur, in his tutorial on QGIS, gives one approach, which starts from the 7.5-minute quadrangle maps published by the New York State Department of Transportation. While these maps are aligned to the same grid as the better-known USGS topographic maps, they are not the same maps; they are somewhat more up to date than the USGS series, and unlike the scanned USGS maps, they are available in four individual layers, one per ink color. They're available from the New York State GIS Clearinghouse - one of the two major GIS repositories for New York. (The other is the Cornell University Geospatial Information Repository [CUGIR].)

    The maps in question consist of one sheet for each of the nearly one thousand quadrangles that make up New York. Each sheet has four separate GeoTIFF files, one per ink color. On the traditional New York State topos that I used when I worked for a county government in the 1970s, the printing layers were:

    • Plan (black) - Overprinted features in black. These are a mix of everything: cultural features (roads and buildings), political features (boundaries and jurisdiction names), place names, and even some hydrology (linear features like streams, and patterned features like marshes).

    • Hydrology (blue) - Polygons giving surface water cover - rivers and lakes.

    • Topography (brown) - Contour lines giving elevations.

    • Built-up areas (yellow on NYS maps, pink on USGS ones) - A shaded overprint marking the area where habitation is so dense that individual buildings are not shown in the plan.



    Andy Arthur's tutorial on QGIS gives quite a comprehensive description for dealing with these layers in QGIS, so I won't repeat it here.

    Combining these four layers for the Niskayuna quadrangle gives a reasonably attractive map:
    Screenshot of the New York State basemap
    It's certainly a technique that I'll keep in my back pocket. But it's not the way that I've decided to go, for several reasons:

    • The maps are raster graphics, which means that they won't scale gracefully. They will pick up pixel artifacts when printed at any other scale than 1:24000, 508 pixels to the inch. (Why 508? Not sure!)

    • The maps are old data: the one that I show in the screenshot was produced in 1980. A lot has changed in 32 years.

    • The maps as scanned are missing stuff along the margins of the region. If I take two quads and lay them together, they don't quite join. It appears that the scans have been cropped a bit too aggressively.


    Gaps between quadrangles

    The USGS digital raster graphics do not have the problem of failing to join up, but they are, for the most part, even older. (Also, they have the problem that they are in color and can't be separated readily.) The newer USTOPO maps are not really intended for GIS use (and don't play all that well with GIS software): it's better to go back to the GIS sources that make them up, which are for the most part publicly available.

    The first thing that maps usually need to show is context: "where is this map relative to the street where I live?" is the first question to answer. So most maps that I produce will need a road map. Most road maps in the US eventually draw on the TIGER maps maintained by the Census Bureau. One public source that draws heavily on TIGER, but then crowdsources additional data, is the OpenStreetMap project. It's become quite good; if all you need is a street map, it's probably serviceable as a basemap. In any case, it should serve well as a starting point for an overlay of cultural features.

    The data in GIS form for just the state of New York are available from http://downloads.cloudmade.com/americas/northern_america/united_states/new_york. They are encoded in an OpenStreetMap-specific format, but QGIS (at least with plugins) can deal with it.

    But there's an issue: The OSM data set is big. The New York data alone are about three gigabytes after decompression. QGIS becomes unacceptably slow and unstable when trying to work with data of this size. Here, PostGIS (which I mentioned in the last post) comes to the rescue. It allows the map data to be saved in a PostgreSQL database, and allows QGIS to load only the data it needs to render a particular map.

    So, after installing all the tools that PostGIS needs, we follow the instructions to create a spatially enabled database. Next, we use the osm2pgsql utility to load up the OpenStreetMap data into the database from the command line:


    osm2pgsql -S /usr/share/osm2pgsql/default.style -E 32618 -d gis -k -s -p new_york_osm new_york.osm


    Go to lunch. it takes a while.

    OK, once it's done, "Add PostGIS Layer" will allow selection from the database and adding the layer to QGIS. It brings up a dialog that looks as shown below, and I select all the layers that I just put in using 'osm2pgsql', and add them.
    Screenshot: Adding PostGIS layers

    After it chugs for a little while, it shows me the graphics.
    Right after loading OSM data into QGIS

    Yes, they look like hell. We'll get around to that. The point is that the streets and boundaries are there. But right now, we need more data. The map I want is more than a road map.
    The next post will discuss topography.
    Read more...
    Posted by Another Kevin at 11:37 AM 0 comments
    Labels:

    Making maps from open-access data in New York - Software

    Once in a while, I have the urge to draw a map of something. For instance, I wanted in my last post to include a map of the trails we took, to show where some of the interesting bits were. Because I do the odd map from time to time, I've been looking into the availability of software and data for doing them. It used to be that map-making was either done laboriously by hand on a light table, or else involved shelling out massive sums of money for software, for data and for Geographic Information Systems (GIS) consultants. But the world has improved; here I'm making notes about what I'm finding out, partly as an aid to myself because I don't do it often enough to remember where all the pieces are.

    I'm a well-known cheapskate: I hate paying a lot for software and data that I'm going to use only intermittently. But I also do respect copyright. In particular, I'm surely not going to scan in a NY/NJ Trail Conference map: producing their maps is a lot of work, and selling them is one of the few sources of revenue that the Conference has to maintain the trails (short of actual donations).

    Moreover, I'm working on a few things for which good maps simply don't exist. One example is that my community has several rather nice nature preserves. One in particular, the former Schenectady Museum Nature Preserve, transferred to New York State a few years back. It's now an unnamed and unsigned parcel of parkland, that the state really doesn't quite know what to do with. But the footpaths in it still exist and are regularly used - and a few people are even maintaining them.

    But the available maps of the preserve don't quite agree with one another, and they vary in readability and completeness. The three that I've found on line are here, here and here. I don't think any of them is entirely satisfactory as it is. To begin with, the only one that actually shows the land boundary and the blaze colors fails to show any context and lacks contour lines. Surely I can do better.

    So, let's see what's out there. I got inspired by stumbling upon a post by Andy Arthur introducing Quantum GIS for making trail maps. His end result was reasonably attractive, and the program looked useful, so I decided to give it a shot.

    Like many things in the open-source world, the program's distribution is a bit chaotic. There are a lot of pieces with "some assembly required," at least under Debian and related systems such as Ubuntu. Material that I know was needed included QGIS itself, GDAL and QGIS's GDAL Python tools, PostGIS (we'll come to why I wanted that in a later post), which in turn requires the Postgres 'contrib' repository and a package called 'hstore', and Postgres itself, and a number of other minor bits. Many of these are components that are run-time loaded into QGIS, and need compatible versions of client libraries (libtiff 3 versus libtiff 4, PostgreSQL 8.4 vs 9.1, and so on). I managed to get something patched together just before giving up and trying to build the whole shebang from source. We'll see whether the next run of the update manager brings everything crashing down around me.

    Once everything installs, the program looks pretty good. The UI is professional, and the functionality all seems to be there. Now for getting some maps! Fortunately, in New York, data availability is quite good. In the next post, I start trolling for data.
    Read more...
    Posted by Another Kevin at 10:34 AM 0 comments
    Labels: ,

    Thursday, December 29, 2011

    Trip report: Slide Mountain, 2011-12-28

    Yesterday, my daughter and I decided to do something a little different, and hiked Slide Mountain - at 4180 feet (1274 m), the highest peak in the Catskills. (It's one of the four that the Catskill 3500 Club requires aspirants to hike in the winter.)

    Map of Slide Mountain hike, 2011-12-28
    Because of last minute preparations, departure from Niskayuna was delayed until well after 0700, and it was about 0945 when we finally got on the trail. We were bringing our Kahtoola MicroSpikes, a first trial of them for both of us. After signing in at the register, we were good to go.

    Register kiosk at Slide Mountain trailhead

    Because of some recent rain/sleet, the Neversink branch at the trailhead was a bit higher than normal, and many of the steppingstones were awash. The remaining ones were quite icy, but the spikes performed admirably. Indeed, they did so for the whole trip: preliminary assessment is that they beat the pants off of YakTrax. (And at the price, they should!)

    We walked in for 1.65 miles on the Phoenicia-East Branch trail, a low-altitude connecting route that for the most part follows what was once a carriage road. The trail was quite wet, since it crosses many small tributaries of both the Neversink and the Esopus. (Trivia: The Phoenicia-East Branch connector and the Burroughs Range trail nearly follow the Catskill divide; some of the streams wind up in the Hudson, while others eventually flow into the Delaware.) Only one of the streams is bridged, and it's not even the most dangerous of them. (The Neversink has claimed the life of a hiker as recently as 2000, when Timothy O'Lear was swept under debris and drowned attempting the ford at high water.) The three-log bridge was iced over, but with spikes we were sure-footed on it.

    The one footbridge we encountered all day

    Ascent was by the Curtis-Ormsbee trail, named for the two hikers who laid out the route.

    Curtis Ormsbee trail junction

    Curtis and Ormsbee perished in an a blizzard in the White Mountains of New Hampshire in June of 1900; a small cenotaph in their memory marks the trailhead on the old carriage road.

    Curtis Ormsbee memorial

    At the turnoff, we overtook a group of some twenty Chinese hikers who had paused to don crampons, and all of us started up the mountain. (We met up with the Chinese several more times that day.)

    Kevin starts up Curtis-Ormsbee Trail

    The trail is reported to have some quite nice overlooks in better weather, but the snow and blowing snow meant that we could see nothing off them. So we kept on trudging upward into progressively colder and snowier weather. The Curtis-Ormsbee route ascends some rock scrambles, and these definitely required extra traction, since they were also iced over. Cathy attempted pictures of a few, but it's really hard to get the scale right in them. Here's looking down from the edge of one of the ledges:

    Ice scrambles

    Kevin got anxious about time on the way up, owing to the late start and the fact that the markers for the 3500 foot elevation were off the trees, making him think that we'd not come nearly as far as we had. (In retrospect, we saw a couple of birches with neat rectangular areas of damaged bark, which must have been where the markers once had been.) At one point, he said to Catherine, "we'll push on for a while longer, and then turn back." Shortly thereafter, Catherine paused to read her GPS while Kevin was trying to stretch out a leg cramp, and got the happy news that we had already reached 3700 feet and were nearing the junction with the Burroughs Range trail.

    The Burroughs Range trail makes a considerably more gradual (but rocky and icy) ascent, through a balsam forest of a sort normally found only at much higher latitudes.

    Kevin approaches summit

    We made the summit in plenty of time, and snapped the obligatory pictures by the Burroughs memorial just beyond:

    Cathy at summitKevin at summit

    We had told Mary Ann that we might bring her back a bottle of water from the spring that is about 0.2 miles beyond the summit, but the trail on the east side of the mountain is considerably tougher, and Kevin was daunted by it. Cathy decided to push on solo to get the water while Kevin looked for a bit of shelter from the gale to cook lunch. But when Catherine hadn't returned for an unreasonably long time, he decided to follow her down, fortunately meeting her on the way back up triumphantly waving a bottle of spring water and saying, "Oh, Daddy, I'm fine." (She later confided that the trail was considerably worse than she imagined, and there were a few points where she seriously feared a fatal fall, and that she should have listened to the old man's more conservative advice. Live and learn.)

    Conditions at the summit were mild for a Catskill winter: temperature about -13 °C, howling gale from the west, light snow falling, boot-deep snow on top of an inch or two of ice. Nothing requiring snowshoes or ice axe, which are often needed at Catskill summits in winter. A few brief breaks in the whiteout allowed us to glimpse some of the views (all but one of the other 34 Catskill high peaks are visible from the top), but none were long enough to photograph them. They also allowed a fairly legible picture of the Burroughs plaque.

    Slide Mountain summit plaque

    We decided that we no longer had time for a hot lunch, abandoned that plan, grabbed some more snacks, and raced the sun back down the west side of the mountain, following the Burroughs Range trail the whole way. Frozen energy bars, frozen raisins, crunchy slush in the water bottles. Delicious. At least we had the foresight to pack the water bottles upside down, so that we could crack the ice at the mouth of the bottle, rather than needing to reach in an implement to break it.

    The Neversink ford was more difficult on the way out: either the river had risen a bit higher or we were more tired. Kevin wandered up and down a bit looking for safe steppingstones, while Cathy got a bootful of icy water when one of her steps (a submerged log) collapsed under her. No problem, a car was only a few steps away.

    We met only three parties all day: the Chinese group, a threesome (an old man and two young men about 17 and 20), and a woman who was coming out as we went in, who we suspect had just soloed the entire Burroughs Range. Everyone was properly equipped (the woman had snowshoes and ice axe lashed to her pack, and wore a balaclava-mask with goggles pushed up on her forehead); everyone wore crampons and proper layers of clothing; everyone had a bulging pack, so likely brought all essentials; and everyone gave the air of knowing what they were about. A welcome change from these trails in the summer, which are frequently crowded with the clueless.

    Summary: Mission accomplished, a first Catskill high peak for both of us. (The highest one, and in winter, for style points!) About 7.5 miles (12.1 km), 1850 ft (560 m) of elevation gain, plus another 450 ft (150 m) getting back to the summit from the spring.

    Good gear:

    Assisted traction was indeed necessary, and the Kahtoola micro-crampons performed admirably.

    Clothing worked well, we stayed toasty even though we were both soaking wet from the snow, despite the conspicuous absence of names like North Face and Patagonia. Base layer was cheap polypro thermal underwear from Ocean State Job Lot . Midlayer was generic polyester fleece sweatpants (Ocean State) and fleece jackets. (Kevin's jacket was an Eddie Bauer, a present from a project lead at work; Catherine's was from a clearance sale at Eastern Mountain Sports.) Both wore Kevin's nylon dress socks under Job Lot polyester-wool blend boot socks. Windshells were Wally World polyester jackets (both of us), REI convertible nylon pants (Kevin), generic no-name ski pants (Catherine, I'd have to ask where she got them, it was either Dick's Sporting Goods or a thrift store). Kevin wore a Carhartt tuque that had a Thinsulate core, and Catherine a knit wool one with ear lappets, from Claire's Boutique of all places! Both wore well-broken-in Timberland leather boots. Dry socks and puffy layer were stuffed in the day packs; we needed the down only for stops. Kevin carried his Tough Traveler 25 litre day pack, and Catherine her Kelty 25 litre technical pack, both lined with plastic trash bags in the compartments that needed to be kept dry.

    Gear not evaluated:

    No hot meal, so winter performance of the cooking system (GSI pot, homemade alcohol stove) is still not assessed.

    Bad gear:

    Nothing, really. Everything worked pretty well, although Catherine would like to have a daypack with a proper hip belt.

    Good technique:

    Catherine was pleased with her technique doing seated glissades. (Catherine just corrected me to "butt slides.")

    Bad technique:

    Kevin needs to be more assertive about safety. Catherine will be better able to judge trail conditions next time.
    Read more...
    Posted by Another Kevin at 8:29 PM 2 comments

    Sunday, June 27, 2010

    Macondo Prospect surface ship status 2010-08-04 00:00 UTC


    The BP oil spill response is a complicated affair involving a fleet of ships operating in a very confined area. According to a Reuters story, BP maintains a Simultaneous Operations (SIMOPS) room that functions much the same as an air traffic control center to coördinate the operations of dozens of ships and as many as sixteen ROV's working at the same time.


    (Check marks ☑ mark vessels within about a mile of the wellhead; unchecked boxes ☐ mark vessels farther from the wellhead at the time of this posting.)


    Ships on long-term station

    Several ships are holding position or at anchor more or less for the duration of the crisis response. These ships become familiar to the readers here because their names come up often.

    • ☑ Deepwater Discoverer Enterprise (538002215)
      Drillship conducting the LMRP Cap oil recovery. Also attempted the failed “top kill” operations.
    • ☐ Deepwater Discoverer Inspiration (538002878)
      Sister ship to the Discoverer Enterprise, reported to be bringing the new containment cap to replace the LMRP Platform cap. Arrived at Macondo Prospect 10 July.
    • ☑ Helix Producer I (311062000)
      Production vessel expected to take over oil recovery efforts and provide increased containment/production capacity.
    • ☐ Global Olympic Challenger (259755000)
      347 foot multi-service vessel carrying two UHD ROV's whose video is available.
    • ☑ Helix Q4000 (369550000)
      Drillship, stationed to the east of the gusher, recovering and flaring oil via the kill lines of the failed blowout preventer
    • ☑ Helix Express (576813000)
      520 foot pipeline layer, obviously a major player in the recovery operations. Little press coverage. Sister ship to the Q4000.
    • ☑ Deepwater Development Driller II (576971000)
      Drillship, stationed southwest of the gusher, sinking the first relief well.
    • ☑ Deepwater Development Driller III (576281000)
      Drillship, stationed southeast of the gusher, sinking the second relief well.
    • ☑ Skandi Neptune (258142000)
      ROV tender conducting dispersant operations.
    • ☑ Ocean Intervention 3 (258473000)
      ROV tender conducting construction operations for oil recovery.
    • ☑ Boa Deep C (224262000)
      ROV tender conducting construction operations for oil recovery.
    • ☑ Holiday (368135000)
      Shown in a BP technical briefing as an ROV tender expected to be attending the Helix Producer. No video from this vessel's ROV's appears to be publicly available
    • ☑ HOS Achiever (579299000)
      432 foot multi-service vessel. Shown in a BP technical briefing as an ROV tender expected to be attending the Helix Producer. Known to carry two ROV's and a 150-tonne crane, and observed to maintain permanent station in the cluster of vessels at MC252. No video from this vessel's ROV's appears to be publicly available.
    • ☐ HOS Iron Horse (235072115)
      432 foot multi-service vessel. Known to carry two ROV's and a 150-tonne crane, and observed to maintain permanent station in the cluster of vessels at MC252. No video from this vessel's ROV's appears to be publicly available.
    • ☑ Loch Rannoch (232829000)
      Heavy tanker expected to assist in offloading recovered oil.

    Two specialized “stimulation ships” are working the scene. These are vessels specifically adapted to pumping fluids into the well at high pressure (and, it is to be hoped, dealing with the consequences!). Normally, a stimulation ship is used to enhance the production of a well when additional fracturing (“fracking”) of the rock or “propping” of the formation is needed to keep the pores from closing and the flow from stopping. In this accident, the stimulation ships have been brought on the scene for the Top Kill and Static Kill attempts.

    • ☑ Galliano Marine Service Bj Blue Dolphin (369552000)
      300 foot stimulation vessel. Carries a crew of 45. Can support three umbilical lines and flow 80 barrels of fluids per minute with her 23,000 horsepower pumping system.
    • ☑ Halliburton Stim Star 3 (369050000)
      260 foot stimulation vessel.

    Fire vessels

    An oil platform has vast quantities of flammable liquids about; an oil spill is many times worse. Firefighting vessels are required at all times.

    Oil Skimmers

    Many vessels are skimming oil from the water's surface in the immediate vicinity of the disaster. Included among these are a fleet belonging to the Marine Spill Response Corporation, which has deployed at least ten Responder-class vessels, three barges, six Fast Response Vehicles, four KC-130 tanker aircraft and two King Air 90 spotter aircraft to the spill. MSRC makes a summary of its Atlantic and Gulf Coast vessels available on the net.
    National Response Corporation also conducts oil recovery operations on the site.

    • ☐ A Whale (636014465)
      Supertanker modified to be an enormous oil skimmer. It didn't work.
    • ☐ Harvey Thunder (369579000)
      13,500 hp, 135 foot ocean towing vessel.

      Deduced to be in skimming operations from the ship's track.

    • ☐ MSRC California Responder (366608000)
    • ☐ MSRC Delaware Responder (366593000)
    • ☐ MSRC Florida Responder (366603000)
    • ☐ MSRC Gulf Coast Responder (366598000)
    • ☐ MSRC Louisiana Responder (366592000)
    • ☐ MSRC Maine Responder (366599000)
      A wire service story reports that the Maine Responder sailed for the Gulf from Portland, ME on 2010-05-02 and is expected to remain in the Gulf for the duration of the emergency.
    • ☐ MSRC Mississippi Responder (366601000)
    • ☐ MSRC New Jersey Responder (366594000)
    • ☐ MSRC Pacific Responder (366605000)
    • ☐ MSRC Southern Responder (366596000)
    • ☐ MSRC Texas Responder (366595000)
    • ☐ MSRC Virginia Responder (366602000)
    • ☐ Resolve Salvage & Fire Lana Rose (367137210)
      100-foot salvage tug engaged in oil skimming

    Dispersant Vessels

    Several vessels, according to a diagram from BP, conduct dispersant operations on the water's surface near the rigs.

    Support vessels

    Every platform needs a small armada of tankers, supply ships, tugs, barges and crewboats to keep it supplied and haul its waste and oil away. Contractors for supplying the BP disaster recovery operations include Edison Chouest Offshore, Global Industries, Hornbeck Offshore, Moran Towing Corporation, , SEACOR Marine and United Tugs.

    SEACOR Marine has one fleet of vessels specifically adapted to handle towing and anchor handling for the moorings of deepwater drilling rigs.

    Most of the multi-service vessels are equipped as deepwater ROV tenders, but to the best of my knowledge, video from their ROV's has not been published.

    Government and research vessels

    NOAA and other institutions have vessels studying the effects of the oil spill. NOAA, in particular, has issued a press release describing its operations in the Gulf disaster.

    • ☐ Beau Rivage
      Charter boat under contract to NOAA, conducting bottom longline fishing in the prohibited area for seafood safety samples
    • ☐ Brooks McCall (338257000)
      Research vessel conducting water column sampling.
    • ☐ Rv Endeavor (303471000)
      185-foot research vessel owned by the National Science Foundation and operated by the University of Rhode Island
    • ☐ Jack Fitz
      Private research vessel and ROV carrier reported in the Huffington Post to be investigating the subsea oil plume.
    • ☐ Ocean Veritas (367116090)
      Research vessel conducting water column sampling.
    • ☐ EGS Ridley Thomas (538002534)
      Survey vessel
    • ☐ NOAA Delaware Ii
      155 foot research vessel
    • ☐ NOAA Caretta
      Mississippi research vessel conducting a trawling and plankton survey in the oiled area off Mississippi.
    • ☐ NOAA Gandy
      Mississippi research vessel operating vertical line survey in the eastern Gulf for seafood safety samples.
    • ☐ NOAA Gordon Gunter
      224 foot research vessel, studying marine mammals
    • ☑ Noaa Henry Bigelow (369991000)
      209-foot research vessel that is nearly identical to the Pisces and has apparently substituted for her in sonar monitoring of the shut-in well.
    • ☐ NOAA Nancy Foster
      187 foot resesaarch vessel, mapping platform and ROV tender
    • ☐ NOAA Oregon Ii
      170 foot research vessel equipped with trawls and longlines for study of sea life
    • ☐ NOAA Pisces (369970145)
      Not to be confused with the Toisa Pisces, the NOAA Pisces is a 209-foot research vessel conducting acoustic surveys of the disaster and monitoring water quality.
    • ☐ NOAA Thomas Jefferson (369958000)
      204 foot survey vessel
    • ☑ Tidewater War Admiral (368164000)
      Chartered by BP to monitor current patterns in the Gulf.
    • ☑ US Coast Guard C G Decisive (367298000)
      211 foot medium endurance cutter
    • ☐ US Coast Guard Resolute
      211 foot medium endurance cutter
    • ☐ US Coast Guard USCGC Walnut (366953000)
      225 foot seagoing buoy tender; home port is Honolulu.

    Expected vessels

    Some vessels had previously been in attendance on the blowout and are expected to return. Others have been reported in the media as expected to participate in the response but have not yet been observed.

    • ☐ Clear Leader (538002877)
    • ☐ Toisa Pisces (636011860)
      Production vessel expected to take over oil recovery efforts and provide increased containment/production capacity.
    • ☐ Viking Poseidon (259873000)
      ROV tender conducting subsea construction. Formerly provided two video feeds. As of 2010-06-25, moored at Gulf Copper Dry Dock and Rig Repair in Galveston, TX. Unknown whether BP intends for this vessel to return to the disaster site.
    • ☐ West Sirius (357575000)
      A press release reported that the semisubmersible deepwater drilling platform West Sirius had been chartered by BP for the Gulf. The vessel was seen at the site on 2010-06-25, but left that day, steaming in a southwesterly direction.

    Seismic survey vessels

    BP, in the course of conducting its well integrity testing on July 13-14, has announced that it is conducting continuing seismic surveys of the site to characterize the conditions below the seafloor. The vessels that appear to be particicipating in this survey all belong to WesternGeco a business unit of Schlumberger. Seismic vessels are a very specialized ship; there are only a few dozen worldwide.

    • ☐ Geco Searcher (352473000)
    • ☐ Geco Topaz (352923000)
      Mentioned in a news briefing as having needed to sail through the cluster of ships at Macondo Prospect, requiring complex support from the simultaneous operations center to move them out of the way.
    • ☐ Geco Triton (355468000)
    • ☐ Gilavar (371810000)
    • ☐ Western Trident (357269000)

    Other Vessels

    These vessels have been seen assisting at the disaster, but information on their role is sparse. Further research by the community would be appreciated.



    Read more...
    Posted by Another Kevin at 5:37 PM 0 comments

    Saturday, May 8, 2010

    Tcl operations, and many different zeroes

    One reason for the title of this blog that I've explored fairly little up to now is that I'm one of the gnomes who make the Tcl programming language go. For this reason, many of my computer applications, when they start up, depart from a script - a Tcl script.

    Last night, a colleague pointed me to a thread on linuxquestions.org that begins with a misunderstanding of how data are interpreted in Tcl, and diverts into a screed that Tcl is "objectively bad."

    Fundamentally, the original misunderstanding appears to be one of notation. The original poster complains how Tcl is 'clumsy in dealing with binary data.' He has a string comprising a single character, for example 'B', and he expects to be able to write, [expr {$char & 0x80}]. Needless to say, that expression fails:

    can't use non-numeric string as operand of "&"

    Why is this? Well, fundamentally, in Tcl, Everything Is A String - it's one of the language's guiding principles. That principle means that there is no difference between the string "12" and the number 12: they comprise the same characters, hence they are the same entity. A side effect of this rule is that there is no difference between the character 0 and the number 0: again, they are the same string of characters. This unity is a tremendous convenience when you are using Tcl as a shell (or for writing scripts that serve the same purpose as shell scripts): you don't have to add a lot of excess quotes just to distinguish the types of things.

    It gets in the way only when you are trying to do low-level stuff like parsing, such as is done in C:

    for (i = 0; (c = str[i]) != '\0'; ++i) {
        /* do something with the character c */
    }


    Tcl allows similar processing that (in my opinion) is just as convenient:

    foreach c [split $str] {
        # do something with the character c
    }

    But inside the loop, the character c is a one-character string in Tcl; in C, it's an integer whose value is the representation of the character in ASCII (or some other encoding).

    If you do want, for some reason, to iterate over the bytes of a string as integers, it's easily done:

    binary scan $str c* chars
    foreach c $chars {
        # do something with the integer c
    }

    If there were demand for it - and as far as I'm aware, nobody's ever asked - Tcl's expression engine could be modified to accept single-quoted characters as integer constants. It appears that to Tcl's users, the existing facilities to manipulate bytes are Good Enough - partly because of Tcl's culture of extensibility. If you need high-performance, low-level programming, you do it in C and provide a Tcl API.

    And so the original poster's question could be easily dismissed as "not understanding the language" - were it not for the somewhat vitriolic posts that followed. The one that rankled in particular said,

    Well, TCL is an objectively bad language because it is (used to be ?) a true source code interpreter. So, syntax errors are not found until containing them code is executed.

    For example, in VLSI synthesis may take several days, and it's quite a pity if the whole process ultimately fails because of a silly syntax error at the script end.

    That's why static TCL checkers have been invented ...

    "Objectively bad?" Oh, my.

    Given that static Tcl checkers like Nagelfar do exist, most of that poster's screed can be reduced to a complaint that Tcl doesn't force you to use them. Surely, if you're going to invest days in a VLSI synthesis, you'll want to ensure that you've bulletproofed your program in every way possible: perhaps those who are dealing with lighter-weight tasks where you can fix a script and rerun in literally seconds have different sets of tradeoffs? Does "objectively bad" reduce to "unfit for this purpose without some extra help?"

    Admittedly, I am aware of studies claiming that programmers presented with strongly-typed languages debug their programs faster. But most of them are old, and compare the strong typing of a language like Pascal against the untyped pointers of PL\1 : anyone else remember the disaster that those were? I'm not aware of any studies that both are conducted with experienced programmers (beyond the level, say, of first- or second-year undergraduates) and are evaluating modern languages. The folklore says, "the more checking up front, the better," but the actual experience with up-front program checking always seems to reach a point of diminishing, nay, negative returns. After that point, adding additional strictness simply makes the programmer jump through hoops arguing to the compiler that the program is correct before the system will deign to run it. When that point is reached seems to vary, so there's certainly still room for subjectivity in evaluating it; in my opinion "objectively bad" is far too strong a statement.

    And the "strongly typed" languages fall prey to the other problem: runtime exceptions (SIGSEGV in C, NullPointerException in Java, etc.). Tcl essentially never gets those: if a Tcl script manages to cause one without help from extension code in another language, it's "all hands on deck" among the maintainers to fix the bug. Even if the script is obviously erroneous, Tcl's maintainers pride themselves on the ability to catch and report all runtime errors.

    But Tcl is unquestionably not fit for all purposes. No language is. If you find that you have an application where it does poorly, I probably have similar applications somewhere, and I'd be happy to recommend other languages to you. For a good many "high-level" tasks, it seems to fit the way I think. Argue if you will that I'm not "right-thinking," but don't assert that your right thinking is objectively better without data to back it up!

    (Hmm, is this attitude Tcl's marketing problem? The Tcl community seems to be remarkably free of zealots that assert that it is objectively better than its competitors. If that is a problem, it's a problem that I enjoy having; I tend to struggle to deal with zealots of any stripe.)
    Read more...
    Posted by Another Kevin at 12:06 PM 0 comments
    Labels: ,
    Subscribe to: Posts (Atom)