In the previous post, I presented an approach to generalize big trajectory datasets by extracting flows between cells of a data-driven irregular grid. This generalization provides a much better overview of the flow and directionality than a simple plot of the original raw trajectory data can. The paper introducing this method also contains more advanced visualizations that show cell statistics, such as the overall count of trajectories or the generalization quality. Another bit of information that is often of interest when exploring movement data, is the time of the movement. For example, at LBS2016 last week, M. Jahnke presented an application that allows users to explore the number of taxi pickups and dropoffs at certain locations:

Jahnke, M., Ding, L., Karja, K., & Wang, S. (2017). Identifying Origin/Destination Hotspots in Floating Car Data for Visual Analysis of Traveling Behavior. In Progress in Location-Based Services 2016 (pp. 253-269). Springer International Publishing.

By adopting this approach for the generalized flow maps, we can, for example, explore which parts of the research area are busy at which time of the day. Here I have divided the day into four quarters: night from 0 to 6 (light blue), morning from 6 to 12 (orange), afternoon from 12 to 18 (red), and evening from 18 to 24 (dark blue).

Aggregated trajectories with time-of-day markers at flow network nodes (data credits: GeoLife project, map tiles: Carto, map data: OSM)

The resulting visualization shows that overall, there is less movement during the night hours from midnight to 6 in the morning (light blue quarter). Sounds reasonable!

One implementation detail worth considering is which timestamp should be used for counting the number of movements. Should it be the time of the first trajectory point entering a cell, or the time when the trajectory leaves the cell, or some average value? In the current implementation, I have opted for the entry time. This means that if the tracked person spends a long time within a cell (e.g. at the work location) the trip home only adds to the evening trip count of the neighboring cell along the trajectory.

Since the time information stored in a PostGIS LinestringM feature’s m-value does not contain any time zone information, we also have to pay attention to handle any necessary offsets. For example, the GeoLife documentation states that all timestamps are provided in GMT while Beijing is in the GMT+8 time zone. This offset has to be accounted for in the analysis script, otherwise the counts per time of day will be all over the place.

Using the same approach, we could also investigate other variations, e.g. over different days of the week, seasonal variations, or the development over multiple years.

This post is part of a series. Read more about movement data in GIS.

Movement data in GIS #3: visualizing massive trajectory datasets

By underdark

2016-11-07

Movement data in GIS, QGIS, spatio-temporal data, Visualization

12 Comments

In the fist two parts of the Movement Data in GIS series, I discussed modeling trajectories as LinestringM features in PostGIS to overcome some common issues of movement data in GIS and presented a way to efficiently render speed changes along a trajectory in QGIS without having to split the trajectory into shorter segments.

While visualizing individual trajectories is important, the real challenge is trying to visualize massive trajectory datasets in a way that enables further analysis. The out-of-the-box functionality of GIS is painfully limited. Except for some transparency and heatmap approaches, there is not much that can be done to help interpret “hairballs” of trajectories. Luckily researchers in visual analytics have already put considerable effort into finding solutions for this visualization challenge. The approach I want to talk about today is by Andrienko, N., & Andrienko, G. (2011). Spatial generalization and aggregation of massive movement data. IEEE Transactions on visualization and computer graphics, 17(2), 205-219. and consists of the following main steps:

Extracting characteristic points from the trajectories
Grouping the extracted points by spatial proximity
Computing group centroids and corresponding Voronoi cells
Dividing trajectories into segments according to the Voronoi cells
Counting transitions from one cell to another

The authors do a great job at describing the concepts and algorithms, which made it relatively straightforward to implement them in QGIS Processing. So far, I’ve implemented the basic logic but the paper contains further suggestions for improvements. This was also my first pyQGIS project that makes use of the measurement value support in the new geometry engine. The time information stored in the m-values is used to detect stop points, which – together with start, end, and turning points – make up the characteristic points of a trajectory.

The following animation illustrates the current state of the implementation: First the “hairball” of trajectories is rendered. Then we extract the characteristic points and group them by proximity. The big black dots are the resulting group centroids. From there, I skipped the Voronoi cells and directly counted transitions from “nearest to centroid A” to “nearest to centroid B”.

From thousands of individual trajectories to a generalized representation of overall movement patterns (data credits: GeoLife project, map tiles: Stamen, map data: OSM)

The resulting visualization makes it possible to analyze flow strength as well as directionality. I have deliberately excluded all connections with a count below 10 transitions to reduce visual clutter. The cell size / distance between point groups – and therefore the level-of-detail – is one of the input parameters. In my example, I used a target cell size of approximately 2km. This setting results in connections which follow the major roads outside the city center very well. In the city center, where the road grid is tighter, trajectories on different roads mix and the connections are less clear.

Since trajectories in this dataset are not limited to car trips, it is expected to find additional movement that is not restricted to the road network. This is particularly noticeable in the dense area in the west where many slow trajectories – most likely from walking trips – are located. The paper also covers how to ensure that connections are limited to neighboring cells by densifying the trajectories before computing step 4.

Running the scripts for over 18,000 trajectories requires patience. It would be worth evaluating if the first three steps can be run with only a subsample of the data without impacting the results in a negative way.

One thing I’m not satisfied with yet is the way to specify the target cell size. While it’s possible to measure ellipsoidal distances in meters using QgsDistanceArea (irrespective of the trajectory layer’s CRS), the initial regular grid used in step 2 in order to group the extracted points has to be specified in the trajectory layer’s CRS units – quite likely degrees. Instead, it may be best to transform everything into an equidistant projection before running any calculations.

It’s good to see that PyQGIS enables us to use the information encoded in PostGIS LinestringM features to perform spatio-temporal analysis. However, working with m or z values involves a lot of v2 geometry classes which work slightly differently than their v1 counterparts. It certainly takes some getting used to. This situation might get cleaned up as part of the QGIS 3 API refactoring effort. If you can, please support work on QGIS 3. Now is the time to shape the PyQGIS API for the following years!

The source code for this experiment is available on GitHub.

This post is part of a series. Read more about movement data in GIS.

More icons & symbols for QGIS

By underdark

2016-10-23

QGIS

1 Comment

An update on this topic has been posted on 2020-10-17.

The possibility to easily share plugins with other users and discover plugins written by other community members has been a powerful feature of QGIS for many years.

The QGIS Resources Sharing plugin is meant to enable the same sharing for map design resources. It allows you to share collections of resources, including but not limited to SVGs, symbols, styles, color ramps, and processing scripts.

Using the Resource Sharing plugin is like using the Plugin Manager. Once installed, you are presented with a list of available resource collections for download. You will find that there are already some really nice collections, including nautical symbols, Mapbox Maki Icons, and my Google-like OSM road style.

By pressing Install, the resource collection is downloaded and you can have a look at the content using the Open folder button. In case of the Mapbox Maki Icon collection, it contains a folder of SVGs:

resourcesharing_folder

Using the new icons is as simple as opening the layer styling settings and selecting the Mapbox Maki Icons collection in the SVG group list:

Similarly, if you download the OSM Spatialite Googlemaps collection, its road line symbols are added to your existing list of available line symbols:

By pressing the Open Library button, you get to the Style Manager where you can browse through all installed symbols and delete, rename, or categorize them.

The Resource Sharing plugin was developed by Akbar Gumbira during this year’s Google Summer of Code. The full documentation, including instructions for how to share your own symbols with the community, is available at www.akbargumbira.com/qgis_resources_sharing.

Movement data in GIS #2: visualization

By underdark

2016-10-09

Movement data in GIS, QGIS, spatio-temporal data

1 Comment

In the first part of the Movement Data in GIS series, I discussed some of the common issues of modeling movement data in GIS, followed by a recommendation to model trajectories as LinestringM features in PostGIS to simplify analyses and improve query performance.

Of course, we don’t only want to analyse movement data within the database. We also want to visualize it to gain a better understanding of the data or communicate analysis results. For example, take one trajectory:

(data credits: GeoLife project)

Visualizing movement direction is easy: just slap an arrow head on the end of the line and done. What about movement speed? Sure! Mean speed, max speed, which should it be?

Speed along the trajectory, a value for each segment between consecutive positions.

With the usual GIS data model, we are back to square one. A line usually has one color and width. Of course we can create doted and dashed lines but that’s not getting us anywhere here. To visualize speed variations along the trajectory, we therefore split the original trajectory into its segments, 1429 in this case. Then we can calculate speed for each segment and use a graduated or data defined renderer to show the results:

Speed along trajectory: red = slow to blue = fast

Very unsatisfactory! We had to increase the number of features 1429 times just to show speed variations along the trajectory, even though the original single trajectory feature already contained all the necessary information and QGIS does support geometries with measurement values.

Starting from QGIS 2.14, we have an alternative way to deal with this issue. We can stick to the original single trajectory feature and render it using the new geometry generator symbol layer. (This functionality is also used under the hood of the 2.5D renderer.) Using the segments_to_lines() function, the geometry generator basically creates individual segment lines on the fly:

Segments_to_lines( $geometry) returns a multi line geometry consisting of a line for every segment in the input geometry

Once this is set up, we can style the segments with a data-defined expression that determines the speed on the segment and returns the respective color along a color ramp:

Speed is calculated using the length of the segment and the time between segment start and end point. Then speed values from 0 to 50 km/h are mapped to the red-yellow-blue color ramp:

ramp_color(
  'RdYlBu',
  scale_linear(
    length( 
      transform(
	    geometry_n($geometry,@geometry_part_num),
		'EPSG:4326','ESRI:54027'
		)
    ) / (
      m(end_point(  geometry_n($geometry,@geometry_part_num))) -
      m(start_point(geometry_n($geometry,@geometry_part_num)))
    ) * 3.6,
    0,50,
    0,1
  )
)

Thanks a lot to @nyalldawson for all the help figuring out the details!

While the following map might look just like the previous one in the end, note that we now only deal with the original single line feature:

Similar approaches can be used to label segments or positions along the trajectory without having to break the original feature. Thanks to the geometry generator functionality, we can make direct use of the LinestringM data model for trajectory visualization.

This post is part of a series. Read more about movement data in GIS.

How to visualize bird migration data with QGIS TimeManager

By underdark

2016-09-24

QGIS, spatio-temporal data, Visualization

1 Comment

A common use case of the QGIS TimeManager plugin is visualizing tracking data such as animal migration data. This post illustrates the steps necessary to create an animation from bird migration data. I’m using a dataset published on Movebank:

Fraser KC, Shave A, Savage A, Ritchie A, Bell K, Siegrist J, Ray JD, Applegate K, Pearman M (2016) Data from: Determining fine-scale migratory connectivity and habitat selection for a migratory songbird by using new GPS technology. Movebank Data Repository. doi:10.5441/001/1.5q5gn84d.

It’s a CSV file which can be loaded into QGIS using the Add delimited text layer tool. Once loaded, we can get started:

1. Identify time and ID columns

Especially if you are new to the dataset, have a look at the attribute table and identify the attributes containing timestamps and ID of the moving object. In our sample dataset, time is stored in the aptly named timestamp attribute and uses ISO standard formatting %Y-%m-%d %H:%M:%S.%f. This format is ideal for TimeManager and we can use it without any changes. The object ID attribute is titled individual-local-identifier.

movebank_data

The dataset contains 128 positions of 14 different birds. This means that there are rather long gaps between consecutive observations. In our animation, we’ll want to fill these gaps with interpolated positions to get uninterrupted movement traces.

2. Configuring TimeManager

To set up the animation, go to the TimeManager panel and click Settings | Add Layer. In the following dialog we can specify the time and ID attributes which we identified in the previous step. We also enable linear interpolation. The interpolation option will create an additional point layer in the QGIS project, which contains the interpolated positions.

timemanager_settings

When using the interpolation option, please note that it currently only works if the point layer is styled with a Single symbol renderer. If a different renderer is configured, it will fail to create the interpolation layer.

Once the layer is configured, the minimum and maximum timestamps will be displayed in the TimeManager dock right bellow the time slider. For this dataset, it makes sense to set the Time frame size, that is the time between animation frames, to one day, so we will see one frame per day:

timemanager_dock

Now you can test the animation by pressing the TimeManager’s play button. Feel free to add more data, such as background maps or other layers, to your project. Besides exploring the animated data in QGIS, you can also create a video to share your results.

3. Creating a video

To export the animation, click the Export video button. If you are using Linux, you can export videos directly from QGIS. On Windows, you first need to export the animation frames as individual pictures, which you can then convert to a video (for example using the free Windows Movie Maker application).

These are the basic steps to set up an animation for migration data. There are many potential extensions to this animation, including adding permanent traces of past movements. While this approach serves us well for visualizing bird migration routes, it is easy to imagine that other movement data would require different interpolation approaches. Vehicle data, for example, would profit from network-constrained interpolation between observed positions.

If you find the TimeManager plugin useful, please consider supporting its development or getting involved. Many features, such as interpolation, are weekend projects that are still in a proof-of-concept stage. In addition, we have the huge upcoming challenge of migrating the plugin to Python 3 and Qt5 to support QGIS3 ahead of us. Happy QGISing!

How to fix a broken Processing model with AttributeError: ‘NoneType’ object has no attribute ‘getCopy’

By underdark

2016-09-21

QGIS

2 Comments

Broken Processing models are nasty and this error is particularly unpleasant:

...
File "/home/agraser/.qgis2/python/plugins/processing/modeler/
ModelerAlgorithm.py", line 110, in algorithm
self._algInstance = ModelerUtils.getAlgorithm(self.consoleName).getCopy()
AttributeError: 'NoneType' object has no attribute 'getCopy'

It shows up if you are trying to open a model in the model editor that contains an algorithm which Processing cannot find.

For example, when I upgraded to Ubuntu 16.04, installing a fresh QGIS version did not automatically install SAGA. Therefore, any model with a dependency on SAGA was broken with the above error message. Installing SAGA and restarting QGIS solves the issue.

Movement data in GIS: issues & ideas

By underdark

2016-09-18

GIS, Movement data in GIS, PostGIS, spatio-temporal data

8 Comments

Since I’ve started working, transport and movement data have been at the core of many of my projects. The spatial nature of movement data makes it interesting for GIScience but typical GIS tools are not a particularly good match.

Dealing with the temporal dynamics of geographic processes is one of the grand challenges for Geographic Information Science. Geographic Information Systems (GIS) and related spatial analysis methods are quite adept at handling spatial dimensions of patterns and processes, but the temporal and coupled space-time attributes of phenomena are difficult to represent and examine with contemporary GIS. (Dr. Paul M. Torrens, Center for Urban Science + Progress, New York University)

It’s still a hot topic right now, as the variety of related publications and events illustrates. For example, just this month, there is an Animove two-week professional training course (18–30 September 2016, Max-Planck Institute for Ornithology, Lake Konstanz) as well as the GIScience 2016 Workshop on Analysis of Movement Data (27 September 2016, Montreal, Canada).

Space-time cubes and animations are classics when it comes to visualizing movement data in GIS. They can be used for some visual analysis but have their limitations, particularly when it comes to working with and trying to understand lots of data. Visualization and analysis of spatio-temporal data in GIS is further complicated by the fact that the temporal information is not standardized in most GIS data formats. (Some notable exceptions of formats that do support time by design are GPX and NetCDF but those aren’t really first-class citizens in current desktop GIS.)

Most commonly, movement data is modeled as points (x,y, and optionally z) with a timestamp, object or tracker id, and potential additional info, such as speed, status, heading, and so on. With this data model, even simple questions like “Find all tracks that start in area A and end in area B” can become a real pain in “vanilla” desktop GIS. Even if the points come with a sequence number, which makes it easy to identify the start point, getting the end point is tricky without some custom code or queries. That’s why I have been storing the points in databases in order to at least have the powers of SQL to deal with the data. Even so, most queries were still painfully complex and performance unsatisfactory.

So I reached out to the Twitterverse asking for pointers towards moving objects database extensions for PostGIS and @bitnerd, @pwramsey, @hruske, and others replied. Amongst other useful tips, they pointed me towards the new temporal support, which ships with PostGIS 2.2. It includes the following neat functions:

ST_IsValidTrajectory — Returns true if the geometry is a valid trajectory.
ST_ClosestPointOfApproach — Returns the measure at which points interpolated along two lines are closest.
ST_DistanceCPA — Returns the distance between closest points of approach in two trajectories.
ST_CPAWithin — Returns true if the trajectories’ closest points of approach are within the specified distance.

Instead of points, these functions expect trajectories that are stored as LinestringM (or LinestringZM) where M is the time dimension. This approach makes many analyses considerably easier to handle. For example, clustering trajectory start and end locations and identifying the most common connections:

(data credits: GeoLife project)

Overall, it’s an interesting and promising approach but there are still some open questions I’ll have to look into, such as: Is there an efficient way to store additional info for each location along the trajectory (e.g. instantaneous speed or other status)? How well do desktop GIS play with LinestringM data and what’s the overhead of dealing with it?

This post is part of a series. Read more about movement data in GIS.

How to use Print Composer templates

By underdark

2016-09-17

QGIS

In the previous post, Mickael shared a great map design. The download includes a print composer template, that you can use to recreate the design in a few simple steps:

1. Create a new composition based on a template

Open the Composer manager and configure it to use a specific template. Then you can select the .qpt template file and press the Add button to create a new composition based on the template.

2. Update image item paths

If the template uses images, the paths to the images most likely need to be fixed since the .qpt file stores absolute file paths instead of relative ones.

With these steps, you’re now ready to use the design for your own maps. Happy QGISing!

Reporting back from Bonn & Oslo

By underdark

2016-09-03

QGIS

1 Comment

Over the last two weeks, I had the pleasure to attend both the international FOSS4G conference in Bonn, Germany, as well as the regional FOSS4G-NOR in Oslo, Norway. Both events were superbly organized and provided tons of possibilities to share experiences and find new inspiration.

Talks at both conferences have been recorded and can be watched online: Bonn / Oslo

I enjoyed having the opportunity to give two very different talks. In Bonn, I presented work on pedestrian routing and navigation, which was developed within the PERRON project:

It was particularly nice that we had plenty of time for Q&A after this presentation since only two talks were scheduled for this session rather than the usual three. I’d also like to thank everyone for the great feedback – both in person and on Twitter!

In Oslo, I had the honor to give the opening keynote on OpenSource in general and the QGIS project in particular:

2 – Anita Graser – QGIS – A Community-powered GIS Project from krokskogstrollet on Vimeo.

Both conferences were packed with great sessions and talks. If I had to pick favorites from last week’s presentations, I would have to opt for Iván Sánchez presenting his latest projects, including what3fucks and geohaiku:

6 – Iván Sánchez Ortega, Mazemap – Addressing NSFW Geodesic Grids from krokskogstrollet on Vimeo.

Followed closely by the impressive project presentations of the student organizers of FOSS4G-NOR:

10 – Program Committee – What are the results when students use Open Source? from krokskogstrollet on Vimeo.

All three projects: OPPTUR, GISTYLE, and the flexible traffic web viewer were great demos of what can be achieved with open source tools. Mathilde’s GISTYLE project is also available on Github.

An inspiring GISummer comes to an end, but with so many videos to watch and workshop materials to explore, I’m convinced that the autumn will be no less exciting.

OSGeo Code Sprint in Bonn

By underdark

2016-08-27

QGIS

It’s been a great week in Bonn! I joined the other members of the QGIS project at the pre-FOSS4G code sprint at the Basecamp, the weirdest location we’ve had for a developer meeting so far. We used this opportunity to have a face-to-face meeting of the QGIS PSC with special guests Matthias Kuhn (on QGIS 3.0 and bug tracker discussions) and Lene Fischer (on community team issues) – notes here.

QGIS PSC meeting in action (from left to right: Otto Dassau, Paolo Cavallini, Anita Graser, Andreas Neumann, Jürgen E. Fischer), picture by Tim Sutton

I also finally took the time to compile a blog post on the results of the QGIS user survey 2015.

The code sprint was also a great opportunity to present the results of Akbar Gumbira’s Google Summer of Code project: the QGIS Resource Sharing plugin. This plugin makes it possible to easily share resources (such as SVG icons, symbol definitions, QGIS styles, and Processing scripts) with other QGIS users through an interface that closely resembles the well-known QGIS Plugin Manager. Akbar has also prepared great presentation with background info and screen casts showcasing his project.

QGIS Resource Sharing presentation, picture by @foss4g

The plugin is now available in the Plugin Repository and we have created the official QGIS Resources repository on Github. If you have symbols or styles that you want to share with the community, please create a resource collection and send us a pull request to add it to the official list.

Thanks to all the organizers who worked hard to make this one of the most well-organized and enjoyable code sprints I’ve ever been to. You are awesome!