Tweets to QGIS
The idea behind this post was to create a video of twitter activity using Time Manager. You can watch the results of my first test run here:
And this is how it’s done:
First, you have to collect some tweets with location information. The following command will collect tweets within a certain geographic region from the Twitter Stream API using curl. You need a Twitter user account to use the API. (Curl comes readily available with OSGeo4W install.)
curl -k -d @locations.txt https://stream.twitter.com/1/statuses/filter.json -uuser:password > tweets.json
The contents of locations.txt is the geographic extent you are interested in, e.g. for Austria:
locations=9,45,17,50
After collecting some data, you can load the tweets into QGIS. Executing the following lines in Python Console will add an in-memory point layer to the map. (I am only extracting coordinates and time stamp from the tweets, but you can access more information through the JSON object.)
import simplejson from PyQt4.QtCore import * from datetime import * f=open('C:/temp/tweets.json','r') # create layer vl = QgsVectorLayer("Point", "tweets", "memory") vl.startEditing() pr = vl.dataProvider() # add fields pr.addAttributes( [ QgsField("t", QVariant.String) ] ) # create features for line in f: try: j=simplejson.loads(line) fet=QgsFeature() fet.setGeometry(QgsGeometry.fromPoint(QgsPoint(j['geo']['coordinates'][1],j['geo']['coordinates'][0]))) fet.setAttributeMap({0:QVariant(str(datetime.strptime(j['created_at'],'%a %b %d %H:%M:%S +0000 %Y')))}) pr.addFeatures([fet]) except: pass vl.commitChanges() vl.updateExtents() QgsMapLayerRegistry.instance().addMapLayer(vl)
To use the result in Time Manager, you have to export the layer to e.g. Shapefile because it’s not possible to add query strings to in-memory layers.
If you are interested in learning more about PyQGIS, you can find a lot of useful material in the PyQGIS Cookbook.
Pingback: Nice Animations with Time Manager’s Offset Feature | Free and Open Source GIS Ramblings
Great tutorial! I was wondering if there is any way to use python to extract tweets containing key words in the text field of the .json? For example, only map tweets that contain “happy” in the text field. Thanks!
Sure. The JSON object also contains the tweet text which you can analyze however you want.
That’s just the thing, was wondering if you have suggestions on what python commands to use and where it would be used.
I figured that using QVariant(j[‘text’]) will include a text field with the tweet content, however I’m more interested to know if there’s any way in filtering the final displayed tweets.
Chapter 9 in the cookbook mentioned filtering by LIKE, but I’m not sure if that is what I want and where it would be used?
So you want to filter in QGIS GUI instead of already in the Python script?
Well I just wanted to see if it is possible to filter using the python script? I just have very little experience using python so don’t know how to write the script to do so. Using my very little understanding I was able to see how I would include the text field to the attribute table, but no idea where to start filtering to only show tweets that contain certain words.
However, if you have a solution to filter using the Qgis GUI then I’m open to any options!
Man, this is SOOOO much easier then using any arcGIS type of thing. Thank you SO much!
Hi,
At first many many thanks for this tutorial.
I’m not familiar with Python script, however I’d like to ask a probably silly question, but I’m stacked :)
I’d like to add some more attributes from the json file I have created. I would like to add the text tweets. So, I’ve edited the code on line 13:
pr.addAttributes( [ QgsField(“t”, QVariant.String), QgsField(“text”, QVariant.String) ] )
and I think I have to make an edit on line 21?
Could you help me on what I have to add there in order to be able to add some more fields on my own?
Many thanks again and regards
SA
Hi Stathis,
Glad you found this small tutorial useful. The pyQGIS Cookbook should answer your question with useful code snippets: http://www.qgis.org/pyqgis-cookbook/vector.html#writing-vector-layers