Get Gmail search queries as terminal output or RSS feeds

Since i’ve implemented GTD in my Gmail account i always find myself in need to see a specific query result, outside gmail interface. Such functionality was one of the main reasons of using Remember The Milk, where queries was really powerful.  IMAP wasn’t an answer, since it was able to export only one label.

Meanwhile i’ve found out about libgmail, a Gmail API for Python. After couple of minutes i’ve got a custom search results inside my terminal :) This could allow to embend it almost everywhere and mainly in conky. After that i’ve decided to build an RSS feeds, scheduled in cron which send them to external server. Problem was that i didn’t want to store the password inside the script nor type it in each time.

As i’m (un)happy user of KDE4 and i know that it comes with quite wide range of language binding so why not try them out. That was PyKDE4 and PyQt4 modules. This one took some more time, since for example every app communicating with KWallet needs to have a window instance, even if it’s windowless. But there were also good sides. I’ve already had my username and password inside KWallet – i’ve configured kopete before. So basically, script doesn’t need any account information. After some struggling i’ve ended up with something like this (gmail-search.py):

QueryTemplates development progress

QueryTemplates 1.0 Beta3 is out since a week or so and next release is around the corner, so i decided to do some wrap-up of latest facts in library development:

• Extended data targets are no supported by all methods. This means each method can be re-targeted with jQuery insert methods like for example varsToSelectorBefore or even varsToSelectorAttr.
• New codeTo method type. It simply injects code groups. Idea evolved from methodTo approach, which would inject method calls. I find codeTo amuch more flexible and useful, but it can be less readable.
• New valuesToLoop method. In opposite to varsToLoop aka loop(), it will iterate existing data over DOM structure and commit it right away. This has less purpose on server side (but can be useful), but it’s essential on the client-side when you don’t want to make a server request.
• New conditional chains methods – onlyJS() and onlyPHP(). They allow to highly customize template for certain language. For example, everything between onlyPHP() and endOnly() will be applied only to PHP template, even when the same chain is used to create JS template.
• Light version for JavaScript is available. It contains all valuesTo methods which includes valuesToSelector, valuesToStack and valuesToLoop. Form methods and mutation events support will be next. It’s a jQuery plugin and demo is available.
• formFromVars will generate full form templates from structure and variable informations. Such form supports error messages and default values. Of course input-wrapper template is customizable. As for today there are couple of things to be done in this area and formFromValues will also be available soon.
• Syntax reference is complete. Each method has it’s own wiki page with 6-part example: markup, data, formula aka code, template, tree dump before and tree dump after. Also, new JavaScript Light version is included in this reference. User comment are allowed. Personally i think it’s the best part of project’s page ;)
• Some API refactorings had taken place. Old names are still accessible, but marked as deprecated.
  • varsFrom type was created from such methods as saveTextAsVar, saveAsVar, valuesToVars.
  • loop, loopOne and loopSeparate now becomes varsToLoop, varsToLoopFirst and varsToLoopSeparate and are grouped in ToLoop type.
• dumpTree is now more verbose and can be used both via HTML and plain text. Just do ->dumpTree() anywhere in the chain to see readable representation of selected structure. It’s used for syntax reference examples.

As you can see project is quite close to 1.0 final. Code seems stable and each method has pseudo-unit-test in form of example available in Syntax reference, which is generated as a real template. I just can’t wait to start implementing CSS interface i wrote before. Additionaly, as JavaScript version is out, there can be some use for jQueryServer.

Can CSS stylesheets be applied on the server-side ?

The question is: can CSS stylesheet be applied in the server-side ? And if yes, would would be the purpose ? I should say yes, they can and i would explain how and why. So what for are stylesheets ? “Applying cascading styles onto the DOM” may be the answer. But i would focus more on that why actually stylesheets are used for such purpose?

• Because they are clean ?
• Because CSS selectors do so much job, that besides them only really needed things are simple properties with obligatory number of arguments ?
• Because cascading inheritance is just right for the DOM ?
• CSS selector are simply benefiting from DOM structure’s nature

So first thing which needs to be implemented on the server is the DOM. When we already got it, so we can query it thou selector, that’s the first half. Now the properties. What could be the property on the server ? border-color ? Rather not. Do we need to style the DOM on the server ? Is it shown anywhere ? Not right now. On the server, page’s DOM is being build. Typically by various ways of string concatenation.

So if we need to build a DOM and same time we can query it with selectors, then the style’s properties should build / modify it’s structure. For example, show some articles or blog posts on the page. So here is pure CSS parser-validable code with some proposed server properties.

.section.articles ul > li {
  /* loop "articles" variable as "article" using "num" as index
  after that remove all other-than-first LIs */
  loop-one: articles num article;

  /* apply filter method on next variable output */
  vars-filter: htmlentities;

  /* use "article" fields and populate them inside nodes
  matched by ".a-%k" where %k is field name, eg .a-title */
  vars-to-selector: articles var(articleFields) null ".a-%k";

  /* example custom method to adding class for 1st loop node using index "num" */
  add-class-to-first: num custom-class;
}
/* show .comments node if variable articleComments is true */
.comments {
  if-var: articleComments;
}
/* bubbled event */
.section.articles:inserted {
  /* another custom method */
  /* null means 0 arguments */
  strip-tables: null;
}

So what have we done here ?

1. We simply build articles loop which prints it’s contents into proper child nodes automagically. It could have 3 fields or 345, but line in CSS is only one.
2. We’ve also used a custom method do add class basing on the loop position.
3. Comments presentance is conditional.
4. One Mutation Event is used. Hidden in :insert pseudoclass, really named DOMNodeInserted will apply on all new nodes which matches prior selector and were just inserted into the DOM structure.
5. Each unspecified event defaults to :load.

All properties should be simply methods which applies something on all matched nodes. Sound familiar ? That’s exactly what many of jQuery methods are doing. Take a look:

.section.articles {
  remove-class: section;
  attr: rel section;
  /* all css properties can be used
  but will result in inline "style" attributes */
  css: padding 2px;
}
/* this rule applies to all load-time nodes
and the new one too */
.change-me,
.change-me:inserted {
  text: "changed innerText";
  prepend: "
<div>changed</div>
";
  remove-class: change-me;
  /* reallocation, interesting... */
  insert-into: ".content";
  /* passing some data to forward ? */
  data: some-flag "some value for developer";
}

All from above properties are jQuery methods. Possibilities are huge. But let’s stick with simple things right now. How would it benefit ? Who will write those server-css-sheets ? I should say…

Events in CSS selector-driven markup templates make them highly reusable

One of the biggest adventage of DOM oriented template engines such as QueryTemplates is events support. Using Event Delegation and some Mutation Events you can set global handlers for all templates. Purpose of such handlers may be varient. I’m presenting 6 of them as examples.

Let’s start from events table

Each event can easily have many handlers. One of them is a mutation event – DOMNodeInserted.

$globalHandlers = array(
  array('unload', 'everyTableHaveTbody'),
  array('load', 'insertNavigation'),
  array('DOMNodeInserted', 'newListsAlwaysOrdered'),
  array('unload', 'noAds'),
  array('unload', 'clearEmptyClasses'),
  array('load', 'stripTables'),
);

Here’s our global handlers

// global handlers
// can be functions, methods, create_function and closures (in PHP 5.3)

function everyTableHaveTbody($e) {
  pq($e->target->ownerDocument)
    ->find('table')
      // nested closure should be here
      ->each('everyTableHaveTbodyCallback');
}
function everyTableHaveTbodyCallback($table) {
  if (pq('> tbody', $table)->length == 0)
    pq($table)->contents()->wrapAll('
<tbody>');
}
function noAds($e) {
  pq($e->target->ownerDocument)
    ->find('.ads')->remove();
}
function newListsAlwaysOrdered($e) {
  if (pq($e->target)->is('ul'))
    pq($e->target)->replaceWith(
      pq('
<ol>')->append(
        pq($e->target)->contents()
      )
    );
}
function insertNavigation($e) {
  $nav = pq($e->target->ownerDocument)
    ->find('.navigation')
      ->empty()
      // nested closure should be here
      ->each('insertNavigationCallback');
}
function insertNavigationCallback($nav) {
  pq($nav)->append(file_get_contents('navigation.html'));
}
function stripTables($e) {
  pq($e->target)->find('table.strip-me')
    ->removeClass('strip-me')
    ->find('> tr:odd, > tbody > tr:odd')
      ->addClass('odd');
}
function clearEmptyClasses($e) {
  $nav = pq($e->target->ownerDocument)
    ->find('*[class=""]')
      ->removeAttr('class');
}

QueryTemplates – template’s logic and markup in one file

Using latest phpQuery revisions you can easily keep template’s markup and logic in one place with 2 extra lines of code. This heavily breaks the general concept, but it may be useful in some cases. Idea is based on support for callbacks as template sources. First you start dumping markup with 1 line.

<?php ob_start(); ?>

After that the markup goes

<html>
  <body>
<div>Hello world</div>
</body>
</html>

And at the end there is whole logic. $markup is dumped using new CallbackReturnValue, which takes a variable as parameter. When callback is called it just returns it.

<?php
require('../src/QueryTemplates.php');
$markup = new CallbackReturnValue(ob_get_clean());
require template('test')->parse($markup)
	->find('div')
		->text('Hello template!')
;

This approach could also be used for getting markup with requestAction in CakePHP views.

toReference() inside chains

There is one extremely useful and not-so widely known toReference() method. It saves actually matched elements inside variable, by reference (as the name says). It can be used to target such cases as:

1. Preserving chain

// declare our variables (this is VERY important)
$deepClass = $section = null;
$template['div.main div.otherclass .deep-class:first']
  ->toReference($deepClass)
//  ->...  // do something
  ->find('.section')
    ->toReference($section)
//  ->...  // do something
  ->end()
  ->next()
    // highlighter but; only one empty()
    ->empty()
    ->append($deepClass->contents())
    ->eq(0)->add($section)->eq(1)
//      ->...  // do something, stack is [$section]
    ->end()->end()->end()
//      ->...  // stack is same as after ->next()
;

2. Splitting chains

// declare our variables (this is VERY important)
$row = $titleBody = null;
$template
  ->find('ul:first > li')
    ->loopOne('posts', 'postNum', 'r')
      // add dynamic class
      ->addClassPHP('if (! $postNum) print "first"')
      ->find('> .title, > .body')
        ->varsToStack('r["Post"]', $postFields)
        // save our $titleBody variable
        ->toReference($titleBody)
      ->end()
      // save our $row variable
      ->toReference($row)
//    ->...  // lots of code ;)
;
// just continue your work
$row->find('h3:first, .comments')
// ->...
;
// anywhere...
doSomethingOnFields($titleBody);

In SVN version of QueryTemplates there is a fix for cache issue. The workaround for beta2 is to do “if ($ref)” before, or turning off the cache with “$cacheTimeout = -1″.

Extending QueryTemplates with closures

Lastest changes in phpQuery allows us to extend it much more easily. One of the ways to do this is directly same as in jQuery, using phpQuery::extend() method. When you combine this with upcoming closures in PHP 5.3, you will feel like-almost-in-browser. Take a look at this:

$source = dirname(__FILE__).'/table.html';
$rows = array(
	array(
		'field-1' => 'foo1',
		'field-2' => 'foo2',
	),
	array(
		'field-1' => 'bar1',
		'field-2' => 'bar2',
	),
);
// this will work in PHP 5.3
// this code will evaluate on onLoad event, thus is cache firendly
$onload = function() {
  $tableFirstRowClass = function($self, $var, $classname = 'first') {
    return $self->addClassPHP("
      if ($$var == 0)
        print '$classname';
    ");
  };
  // here we pass new closure into extend method, just like in jQuery
  // compact() creates array for us
  phpQuery::extend('phpQueryObject', compact('tableFirstRowClass'));
};
require template('table')->parse($source)
  // just bind it
  ->bind('onload', $onload)
  // just fire it up ;)
  ->trigger('onload')
  ->find('table > tr, table > * > tr')
    ->loopOne('rows', 'i', 'row')
      ->varsToSelector('row', $rows[0])
      // now you can use this just-like-this
      ->tableFirstRowClass('i')
;

But for now to extend phpQuery on-the-fly we have to use create_function or delegate existing one.

We can also use new Scripts plugin feature, which does almost the same. Just like all Scripts’ plugin script files, newly attached function will have all available variables. Example below works in PHP < 5.3:

function onload() {
  // in PHP 5.3, this can be done simply like this
//  phpQuery::script(...
  // but for now we have to use fake phpQuery::$plugins namespace
  phpQuery::$plugins->script('tableFirstRowClass', create_function(
    '$self, $params, &$return, $config', '
    $className = ! $params[1] ? "first" : $params[1];
    $self->addClassPHP("
      if ($$params[0] == 0) print '$className';
    ");
  '));
};
$onload = 'onload';

One major difference between extending Scripts and phpQuery itself is way of using new functions. With our new script, we have to small change to main code.

// using phpQuery::extend()
//->tableFirstRowClass('i')
// using Scripts extend
->script('tableFirstRowClass', 'i')

QueryTemplates finally released

I’ve finally managed to release QueryTemplates, a pure HTML templating engine i’ve been working past months. There’re extensive examples which should allow anyone to easily understand the idea. Previously posted Pure HTML templates theory sums up some thoughts about this templating pattern. You can read more about new library on the wiki and there’s also an official blog. Feel free to post feedback.