OpenStreetMap, Open Data, Scripting What, Why, How?

Manuela Butuc
Map analyst @ Telenav

Bogdan Petrea
Map analyst @ Telenav

PROGRAMMING

OpenStreetMap (OSM) is a collaborative project, created with the purpose of offering a complete map, permanently updated by the community, being also free for any kind of uses: websites, mobile apps and GPS systems.

Telenav intends to use OpenStreetMap for many internal projects, especially for embedded navigation systems.

In order to have a quality map, it is mandatory for the data to be revised and constantly updated, reason why the company has a dedicated department for this task: improving OSM data. The Map-Analysts Team improves and updates OSM by using free data and several internal and open-source tools.

Open data - what it is, why and how to get it

The main idea of open data is to offer a variety of data types, for any kind of user, for free: demographical or geographical data. To be considered “open”, those datasheets must meet criteria like availability, access, reuse and universal share.

In order to be used for upgrading and updating OSM, this kind of data must be geospatial in nature, and must have certain attributes, attributes that navigation systems use, such as road name, flow direction, speed limit or turn restrictions.

This kind of datasets are usually accessible and free to download from the data.gov portal of each county, or from the official city websites, especially in the United States and Canada. For example, out of the 50 Canadian cities we work on, almost 80% have open data, without taking into consideration Canvec and Geobase data, which are governmental data with transportation network, covering 100% of the Canadian territory.

Open Data Inception – open-source data portal

Purpose of the project

Once we have the open data available, we must use it in the most efficient way possible. Extracting roads without names from the OSM Database, in order to overlap and compare them with the open data manually was too slow and required a lot of attention. So, we had an idea: “We already have an internal tool that compares geometries and creates a change file with the missing roads. Why don’t we extend this idea to compare attributes as well?”. The attributes (also called tags) are key-value data. They describe the road category, the maximum legal speed, the road name, the number of lanes, and many others. If we could compare OSM road attributes with the attributes from local data, the problem is solved.

No sooner said than done! With the help of the Java department, the Cygnus tool became Cygnus+, and it is now able to compare OSM attributes with the attributes from the local data. With this tool, the road mapping becomes a semiautomatic process, way more effective, more qualitative and quantitative nevertheless.

How it works

The basic concept of the Cygnus+ tool consists in the comparison of geospatial data with the OSM data, using the attributes of these features and a buffer created around road segments within the area of interest.

As expected, the local data does not use the ‘OSM language’, meaning their attributes don’t have the exact same labeling. For example, for the direction of traffic, the Canvec attribute is named “traffic flow direction”, and has the following values: same direction, opposite direction or both, while in OSM, the roads with only one direction of traffic have the attribute “oneway”=”yes”.

In order to ‘translate’ the local data into OSM language, we need to create a translation file (written in Python) that fits the characteristics and attributes of that particular data. Below you can find an example of a translation file that fits the local data from Canvec.

Local data attributes

def filterTags(attrs):
    with open("log.txt", "a") as log_file:
        log_file.write("called")
    if not attrs:
        with open("log.txt", "a") as log_file:
            log_file.write("no attributes")
        return
    tags = {}

    if "STREET_NAM" in attrs:
        tags["name"] = attrs["STREET_NAM"]

    if "REGIONAL_R" in attrs:
        tags["alt_name"] = attrs["REGIONAL_R"]

    if "LANES" in attrs:
        tags["lanes"] = attrs["LANES"]

    if 'SHOULDER' in attrs:
        if attrs['SHOULDER'].strip() == 'PAVED':
            tags['surface'] = 'paved'
        if attrs['SHOULDER'].strip() == 'GRAVEL':
            tags['surface'] = 'gravel'

    if "FLOW_DIREC" in attrs:
        if attrs["FLOW_DIREC"].strip() == "FromTo":
            tags["oneway"] = "yes"
        elif attrs["FLOW_DIREC"].strip() == "ToFrom":
            tags["oneway"] = "-1"

    if 'CARTO_CLAS' in attrs:
        if attrs['CARTO_CLAS'].strip() == 'Expressway / Highway':
            tags['highway'] = 'motorway'
        elif attrs['CARTO_CLAS'].strip() == 'Freeway':
            tags['highway'] = 'motorway'
        elif attrs['CARTO_CLAS'].strip() == 'Arterial':
            tags['highway'] = 'secondary'
        elif attrs['CARTO_CLAS'].strip() == 'Alleyway / Lane':
            tags['highway'] = 'service'
        elif attrs['CARTO_CLAS'].strip() == 'Collector':
            tags['highway'] = 'tertiary'
        elif attrs['CARTO_CLAS'].strip() == 'Local Street':
            tags['highway'] = 'residential'
        elif attrs['CARTO_CLAS'].strip() == 'Cul-de-Sac':
            tags['highway'] = 'residential'
            tags['noexit'] = 'yes'
        elif attrs['CARTO_CLAS'].strip() == 'Ramp':
            tags['highway'] = 'tertiary_link'
        elif attrs['CARTO_CLAS'].strip() == 'Private':
            tags['highway'] = 'service'
            tags['access'] = 'private'
        elif attrs['CARTO_CLAS'].strip() == 'Roundabout':
            tags['highway'] = 'residential'
            tags['junction'] = 'roundabout'

    return tags

Translation file

This translation file will be used together with the open-source script ogr2osm.py, which converts the local data (usually found in shapefile format) into OSM XML files. These files can then be converted to osm.pbf using osmconvert (also an open-source script).

The tag comparison tool needs graph files as input, so it is necessary to convert the osm.pbf files into graph files. We do this by using an internal tool named graph-converter.

Once we obtained the graph file with local data, it can be used as input for the Tag comparison tool. The output will be an XML file that contains only the road segments which have different attributes as compared to the ones in the OpenStreetMap database.

Workflow for processing and comparison of the data

If, for a given road segment, an attribute already exists in OSM, but has a different value as compared to local data, the resulting file will contain that specific road segment with the following tag: <tag k="telenav:cygnus-plus" v="CHANGED"/>. If the attribute is completely missing from the OSM database, but it is present in the local data, the road segment will have the following tag in the resulting file: <tag k="telenav:cygnus-plus" v="NEW"/>. That way, we can prioritize certain editing steps.

After obtaining the output from the Tag Comparison Tool, we open it with JOSM (Java OpenStreetMap Editor) and we use it to add missing attributes or to correct existing ones.