Tweets to QGIS

The idea behind this post was to create a video of twitter activity using Time Manager. You can watch the results of my first test run here:

And this is how it’s done:

First, you have to collect some tweets with location information. The following command will collect tweets within a certain geographic region from the Twitter Stream API using curl. You need a Twitter user account to use the API. (Curl comes readily available with OSGeo4W install.)

curl -k -d @locations.txt https://stream.twitter.com/1/statuses/filter.json -uuser:password > tweets.json

The contents of locations.txt is the geographic extent you are interested in, e.g. for Austria:

locations=9,45,17,50

After collecting some data, you can load the tweets into QGIS. Executing the following lines in Python Console will add an in-memory point layer to the map. (I am only extracting coordinates and time stamp from the tweets, but you can access more information through the JSON object.)

import simplejson
from PyQt4.QtCore import *
from datetime import *

f=open('C:/temp/tweets.json','r')

# create layer
vl = QgsVectorLayer("Point", "tweets", "memory")
vl.startEditing()
pr = vl.dataProvider()

# add fields
pr.addAttributes( [ QgsField("t", QVariant.String) ] )

# create features
for line in f:
   try:
      j=simplejson.loads(line)
      fet=QgsFeature()
      fet.setGeometry(QgsGeometry.fromPoint(QgsPoint(j['geo']['coordinates'][1],j['geo']['coordinates'][0])))
      fet.setAttributeMap({0:QVariant(str(datetime.strptime(j['created_at'],'%a %b %d %H:%M:%S +0000 %Y')))})
      pr.addFeatures([fet])
   except:
      pass

vl.commitChanges()
vl.updateExtents()

QgsMapLayerRegistry.instance().addMapLayer(vl)

To use the result in Time Manager, you have to export the layer to e.g. Shapefile because it’s not possible to add query strings to in-memory layers.

If you are interested in learning more about PyQGIS, you can find a lot of useful material in the PyQGIS Cookbook.

Advertisement
9 comments
  1. Jonathan Critchley said:

    Great tutorial! I was wondering if there is any way to use python to extract tweets containing key words in the text field of the .json? For example, only map tweets that contain “happy” in the text field. Thanks!

    • underdark said:

      Sure. The JSON object also contains the tweet text which you can analyze however you want.

      • Jonthan Critchley said:

        That’s just the thing, was wondering if you have suggestions on what python commands to use and where it would be used.
        I figured that using QVariant(j[‘text’]) will include a text field with the tweet content, however I’m more interested to know if there’s any way in filtering the final displayed tweets.
        Chapter 9 in the cookbook mentioned filtering by LIKE, but I’m not sure if that is what I want and where it would be used?

      • underdark said:

        So you want to filter in QGIS GUI instead of already in the Python script?

      • Jonthan Critchley said:

        Well I just wanted to see if it is possible to filter using the python script? I just have very little experience using python so don’t know how to write the script to do so. Using my very little understanding I was able to see how I would include the text field to the attribute table, but no idea where to start filtering to only show tweets that contain certain words.
        However, if you have a solution to filter using the Qgis GUI then I’m open to any options!

  2. Keith said:

    Man, this is SOOOO much easier then using any arcGIS type of thing. Thank you SO much!

  3. Stathis Arapostathis said:

    Hi,

    At first many many thanks for this tutorial.
    I’m not familiar with Python script, however I’d like to ask a probably silly question, but I’m stacked :)

    I’d like to add some more attributes from the json file I have created. I would like to add the text tweets. So, I’ve edited the code on line 13:

    pr.addAttributes( [ QgsField(“t”, QVariant.String), QgsField(“text”, QVariant.String) ] )

    and I think I have to make an edit on line 21?
    Could you help me on what I have to add there in order to be able to add some more fields on my own?

    Many thanks again and regards
    SA

%d bloggers like this: