Get Gmail search queries as terminal output or RSS feeds

Since i’ve implemented GTD in my Gmail account i always find myself in need to see a specific query result, outside gmail interface. Such functionality was one of the main reasons of using Remember The Milk, where queries was really powerful.  IMAP wasn’t an answer, since it was able to export only one label.

Meanwhile i’ve found out about libgmail, a Gmail API for Python. After couple of minutes i’ve got a custom search results inside my terminal :) This could allow to embend it almost everywhere and mainly in conky. After that i’ve decided to build an RSS feeds, scheduled in cron which send them to external server. Problem was that i didn’t want to store the password inside the script nor type it in each time.

As i’m (un)happy user of KDE4 and i know that it comes with quite wide range of language binding so why not try them out. That was PyKDE4 and PyQt4 modules. This one took some more time, since for example every app communicating with KWallet needs to have a window instance, even if it’s windowless. But there were also good sides. I’ve already had my username and password inside KWallet – i’ve configured kopete before. So basically, script doesn’t need any account information. After some struggling i’ve ended up with something like this (gmail-search.py):

#!/usr/bin/python
import sys
import getopt
from time import gmtime, strftime, strptime, time

opts, args = getopt.gnu_getopt(sys.argv[1:], 'j')

from PyKDE4.kdeui import *
from PyQt4 import *
import libgmail
from platform import python_version
import re
if (python_version() >= '2.6'):
	import json
else:
	import simplejson as json

class App(QtGui.QApplication):
	def __init__(self):
		QtGui.QApplication.__init__(self, [])
		self.ui = QtGui.QMainWindow()

def remove_html_tags(data):
	p = re.compile(r'')
	return p.sub('', data)

# KWALLET
app = App()
mywallet = KWallet.Wallet(0, 'kdewallet')
mywallet = mywallet.openWallet('kdewallet', app.ui.winId())
#print mywallet.folderList()
mywallet.setFolder('Kopete')
mypass = QtCore.QString()
account = '';
for entry in mywallet.entryList():
	match = re.search('^Account_[^_]+_(.+?)@gmail.com$', unicode(entry))
	if (match):
		account = match.group(1)
		mywallet.readPassword(entry, mypass)
		break
if (not account):
	print "No kopete gmail account found!n"
	exit(1);

# TERMINAL
#account = 'username@gmail.com';
#from getpass import getpass
#mypass = getpass("Password please: ")

# PLAIN FILE
#account = 'username@gmail.com';
#mypass = open('/file/path').readline();

ga = libgmail.GmailAccount(account, mypass)
try:
	ga.login()
except libgmail.GmailLoginFailure:
	print "nLogin failed. (Wrong username/password?)"
else:
	#print "nLog in successful."
	result = ga.getMessagesByQuery(args[0])
	rss = {
		'channel': {
			'title': 'GMAIL: '+args[0],
			'description': args[0],
			'pubDate': strftime("%a, %d %b %Y %H:%M:%S", gmtime(time()))
		},
		'items': []
	};
	for t in result:
		# COMMEND LINE OUTPUT
		if (not ('-j', '') in opts):
			print ' * '+remove_html_tags(t.subject)
		else:
			# JSON OUTPUT
			date = remove_html_tags(t.date)
			# 11:45
			if (re.search('d?d:dd', date)):
				date = strftime("%a, %d %b %Y ${time}:%S", gmtime(time())).replace('${time}', date)
			# Mar 7
			elif (re.search('w{3} d+', date)):
				date = strptime(date+strftime(" %Y", gmtime(time())), "%b %d %Y")
				date = strftime("%a, %d %b %Y %H:%M:%S", date)
			# 31/12/2008
			else:
				date = strptime(date, "%d/%B/%Y")
				date = strftime("%a, %d %b %Y %H:%M:%S", date)
			# TODO more date formats
	#		rawMsg = ga.getRawMessage(t.matching_msgid)
			rss['items'].append({
				'title': remove_html_tags(t.subject),
				'description': t.snippet,
				'pubDate': date,
	#			'guid': re.search("Message-ID: ]+)>", 	rawMsg).group(1)
				'guid': t.matching_msgid
			});
	if (('-j', '') in opts):
		print json.dumps(rss);

You can switch KWallet authentification off to typed into terminal if you want. Script above doesn’t create feed yet, by default it print wiki-like list of message subjects. When you add -j switch it will return JSON data, which i use as input for RSS template compiled with QueryTemplates. Source is of course simple (generate-template.php):

parse("rss.xml")->
	find('channel')->
		varsToSelector('data.channel', array('title', 'description', 'pubDate'), '%k')->
	end()->
	find('item')->
		varsToLoop('data.items', 'item')->
		varsToSelector('item', array('title', 'description', 'pubDate', 'guid'), '%k')->
	end()->
	prependPHP('
$data = "";
while (($l = fgets(STDIN)) !== false)
	$data .= $l;
$data = json_decode($data);
	')->
	save()
;

But you don’t need the source, just the final template (rss-template.php):

    {'channel'}->{'title'})) print $data->{'channel'}->{'title'};  ?>
    {'channel'}->{'description'})) print $data->{'channel'}->{'description'};  ?>
{'channel'}->{'pubDate'})) print $data->{'channel'}->{'pubDate'};  ?>
		{'items'})) $__a630a = $data->{'items'}; if (isset($__a630a) && (is_array($__a630a) || is_object($__a630a))) { foreach($__a630a as $item):  ?>
			{'title'})) print $item->{'title'};  ?>
{'pubDate'})) print $item->{'pubDate'};  ?>
			{'guid'})) print $item->{'guid'};  ?>

But if you like to use the source, here is markup for it taken from wikipedia (rss.xml):

    Lift Off News
    Liftoff to Space Exploration.
Tue, 10 Jun 2003 04:00:00 GMT

			Star City
Tue, 03 Jun 2003 09:39:21 GMT
			http://liftoff.ma.gov/2003/06/03.html#item573

To create the feed simply combine output via pipe:

gmail-search.py -j “label:S-Next-Action -label:S-finished” | php rss-template.php

Whole process takes about 2-3 seconds, depending on traffic load. As for dependencies, on debian system you can install the packages like so:

apt-get install python-kde4 python-simplejson python-libgmail

Besides creating feeds libgmail can be used as powerful message filter, what could be very usefull in automating GTD processes.