Tobiasz Cudnik

Archive for 2009|Yearly archive page

Get Gmail search queries as terminal output or RSS feeds

In QueryTemplates, Snippets, Workflow on 19.03.2009 at 2:53

Since i’ve implemented GTD in my Gmail account i always find myself in need to see a specific query result, outside gmail interface. Such functionality was one of the main reasons of using Remember The Milk, where queries was really powerful.  IMAP wasn’t an answer, since it was able to export only one label.

Meanwhile i’ve found out about libgmail, a Gmail API for Python. After couple of minutes i’ve got a custom search results inside my terminal :) This could allow to embend it almost everywhere and mainly in conky. After that i’ve decided to build an RSS feeds, scheduled in cron which send them to external server. Problem was that i didn’t want to store the password inside the script nor type it in each time.

As i’m (un)happy user of KDE4 and i know that it comes with quite wide range of language binding so why not try them out. That was PyKDE4 and PyQt4 modules. This one took some more time, since for example every app communicating with KWallet needs to have a window instance, even if it’s windowless. But there were also good sides. I’ve already had my username and password inside KWallet – i’ve configured kopete before. So basically, script doesn’t need any account information. After some struggling i’ve ended up with something like this (gmail-search.py):

Read the rest of this entry »

Personal torrent stream (part 1)

In phpQuery, Web Scraping on 19.03.2009 at 2:52

I really don’t like reading movie reviews before actually seeing them. For this reason, i found a new nanocrowd.com service quite useful. It claims to provide very specific movie recommendations based on opinions of anonymous masses. Although it’s accounts are unfortunately still in private beta, we can test recommendations based on one movie.

Long time i’ve been thinking about downloading movies, without actually downloading them. Finding out nanocrowd convinced me to do this. Of course i’ve used phpQuery‘s WebBrowser plugin. Scenario is as follows:

  1. Connect to torrentz.net
  2. Fill search box and submit
  3. Filter results
  4. Print results and ask which to download
  5. Get result sites and find supported one
  6. Download to local filesystem
  7. Optionally, send via SSH to another machine

To be practically usefull, script supports following filters:

–min-size
–max-size
–min-peers

So to get a 1st matching movie in most common cases would be

echo 1 | torrent-download.php “Movie Name” –min-peers=50 –max-size=1500 –min-size=500

Now almost every decent torrent downloading app supports autoloding torrent files found in specified directory. As for today, only thing you need to do is to type movie name. Like i’ve mentioned before, nanocrowd is still not open to registration. When it will be, i will post an update (or maybe someone have a free invitation ?). Anyway, don’t use this code to download illegal movies…

Read the rest of this entry »

QueryTemplates development progress

In QueryTemplates on 01.03.2009 at 1:06

QueryTemplates 1.0 Beta3 is out since a week or so and next release is around the corner, so i decided to do some wrap-up of latest facts in library development:

  • Extended data targets are no supported by all methods. This means each method can be re-targeted with jQuery insert methods like for example varsToSelectorBefore or even varsToSelectorAttr.
  • New codeTo method type. It simply injects code groups. Idea evolved from methodTo approach, which would inject method calls. I find codeTo amuch more flexible and useful, but it can be less readable.
  • New valuesToLoop method. In opposite to varsToLoop aka loop(), it will iterate existing data over DOM structure and commit it right away. This has less purpose on server side (but can be useful), but it’s essential on the client-side when you don’t want to make a server request.
  • New conditional chains methods – onlyJS() and onlyPHP(). They allow to highly customize template for certain language. For example, everything between onlyPHP() and endOnly() will be applied only to PHP template, even when the same chain is used to create JS template.
  • Light version for JavaScript is available. It contains all valuesTo methods which includes valuesToSelector, valuesToStack and valuesToLoop. Form methods and mutation events support will be next. It’s a jQuery plugin and demo is available.
  • formFromVars will generate full form templates from structure and variable informations. Such form supports error messages and default values. Of course input-wrapper template is customizable. As for today there are couple of things to be done in this area and formFromValues will also be available soon.
  • Syntax reference is complete. Each method has it’s own wiki page with 6-part example: markup, data, formula aka code, template, tree dump before and tree dump after. Also, new JavaScript Light version is included in this reference. User comment are allowed. Personally i think it’s the best part of project’s page ;)
  • Some API refactorings had taken place. Old names are still accessible, but marked as deprecated.
    • varsFrom type was created from such methods as saveTextAsVar, saveAsVar, valuesToVars.
    • loop, loopOne and loopSeparate now becomes varsToLoop, varsToLoopFirst and varsToLoopSeparate and are grouped in ToLoop type.
  • dumpTree is now more verbose and can be used both via HTML and plain text. Just do ->dumpTree() anywhere in the chain to see readable representation of selected structure. It’s used for syntax reference examples.

As you can see project is quite close to 1.0 final. Code seems stable and each method has pseudo-unit-test in form of example available in Syntax reference, which is generated as a real template. I just can’t wait to start implementing CSS interface i wrote before. Additionaly, as JavaScript version is out, there can be some use for jQueryServer.

Can CSS stylesheets be applied on the server-side ?

In Ideas, QueryTemplates, Workflow on 27.01.2009 at 3:23

The question is: can CSS stylesheet be applied in the server-side ? And if yes, would would be the purpose ? I should say yes, they can and i would explain how and why. So what for are stylesheets ? “Applying cascading styles onto the DOM” may be the answer. But i would focus more on that why actually stylesheets are used for such purpose?

  • Because they are clean ?
  • Because CSS selectors do so much job, that besides them only really needed things are simple properties with obligatory number of arguments ?
  • Because cascading inheritance is just right for the DOM ?
  • CSS selector are simply benefiting from DOM structure’s nature

So first thing which needs to be implemented on the server is the DOM. When we already got it, so we can query it thou selector, that’s the first half. Now the properties. What could be the property on the server ? border-color ? Rather not. Do we need to style the DOM on the server ? Is it shown anywhere ? Not right now. On the server, page’s DOM is being build. Typically by various ways of string concatenation.

So if we need to build a DOM and same time we can query it with selectors, then the style’s properties should build / modify it’s structure. For example, show some articles or blog posts on the page. So here is pure CSS parser-validable code with some proposed server properties.

.section.articles ul > li {
  /* loop "articles" variable as "article" using "num" as index
  after that remove all other-than-first LIs */
  loop-one: articles num article;

  /* apply filter method on next variable output */
  vars-filter: htmlentities;

  /* use "article" fields and populate them inside nodes
  matched by ".a-%k" where %k is field name, eg .a-title */
  vars-to-selector: articles var(articleFields) null ".a-%k";

  /* example custom method to adding class for 1st loop node using index "num" */
  add-class-to-first: num custom-class;
}
/* show .comments node if variable articleComments is true */
.comments {
  if-var: articleComments;
}
/* bubbled event */
.section.articles:inserted {
  /* another custom method */
  /* null means 0 arguments */
  strip-tables: null;
}

So what have we done here ?

  1. We simply build articles loop which prints it’s contents into proper child nodes automagically. It could have 3 fields or 345, but line in CSS is only one.
  2. We’ve also used a custom method do add class basing on the loop position.
  3. Comments presentance is conditional.
  4. One Mutation Event is used. Hidden in :insert pseudoclass, really named DOMNodeInserted will apply on all new nodes which matches prior selector and were just inserted into the DOM structure.
  5. Each unspecified event defaults to :load.

All properties should be simply methods which applies something on all matched nodes. Sound familiar ? That’s exactly what many of jQuery methods are doing. Take a look:

.section.articles {
  remove-class: section;
  attr: rel section;
  /* all css properties can be used
  but will result in inline "style" attributes */
  css: padding 2px;
}
/* this rule applies to all load-time nodes
and the new one too */
.change-me,
.change-me:inserted {
  text: "changed innerText";
  prepend: "
<div>changed</div>
";
  remove-class: change-me;
  /* reallocation, interesting... */
  insert-into: ".content";
  /* passing some data to forward ? */
  data: some-flag "some value for developer";
}

All from above properties are jQuery methods. Possibilities are huge. But let’s stick with simple things right now. How would it benefit ? Who will write those server-css-sheets ? I should say…

Read the rest of this entry »

Events in CSS selector-driven markup templates make them highly reusable

In Ideas, QueryTemplates on 25.01.2009 at 16:36

One of the biggest adventage of DOM oriented template engines such as QueryTemplates is events support. Using Event Delegation and some Mutation Events you can set global handlers for all templates. Purpose of such handlers may be varient. I’m presenting 6 of them as examples.

Let’s start from events table

Each event can easily have many handlers. One of them is a mutation event – DOMNodeInserted.

$globalHandlers = array(
  array('unload', 'everyTableHaveTbody'),
  array('load', 'insertNavigation'),
  array('DOMNodeInserted', 'newListsAlwaysOrdered'),
  array('unload', 'noAds'),
  array('unload', 'clearEmptyClasses'),
  array('load', 'stripTables'),
);

Here’s our global handlers

// global handlers
// can be functions, methods, create_function and closures (in PHP 5.3)

function everyTableHaveTbody($e) {
  pq($e->target->ownerDocument)
    ->find('table')
      // nested closure should be here
      ->each('everyTableHaveTbodyCallback');
}
function everyTableHaveTbodyCallback($table) {
  if (pq('> tbody', $table)->length == 0)
    pq($table)->contents()->wrapAll('
<tbody>');
}
function noAds($e) {
  pq($e->target->ownerDocument)
    ->find('.ads')->remove();
}
function newListsAlwaysOrdered($e) {
  if (pq($e->target)->is('ul'))
    pq($e->target)->replaceWith(
      pq('
<ol>')->append(
        pq($e->target)->contents()
      )
    );
}
function insertNavigation($e) {
  $nav = pq($e->target->ownerDocument)
    ->find('.navigation')
      ->empty()
      // nested closure should be here
      ->each('insertNavigationCallback');
}
function insertNavigationCallback($nav) {
  pq($nav)->append(file_get_contents('navigation.html'));
}
function stripTables($e) {
  pq($e->target)->find('table.strip-me')
    ->removeClass('strip-me')
    ->find('> tr:odd, > tbody > tr:odd')
      ->addClass('odd');
}
function clearEmptyClasses($e) {
  $nav = pq($e->target->ownerDocument)
    ->find('*[class=""]')
      ->removeAttr('class');
}

Read the rest of this entry »

QueryTemplates – template’s logic and markup in one file

In phpQuery, QueryTemplates on 25.01.2009 at 15:41

Using latest phpQuery revisions you can easily keep template’s markup and logic in one place with 2 extra lines of code. This heavily breaks the general concept, but it may be useful in some cases. Idea is based on support for callbacks as template sources. First you start dumping markup with 1 line.

<?php ob_start(); ?>

After that the markup goes

<html>
  <body>
<div>Hello world</div>
</body>
</html>

And at the end there is whole logic. $markup is dumped using new CallbackReturnValue, which takes a variable as parameter. When callback is called it just returns it.

<?php
require('../src/QueryTemplates.php');
$markup = new CallbackReturnValue(ob_get_clean());
require template('test')->parse($markup)
	->find('div')
		->text('Hello template!')
;

This approach could also be used for getting markup with requestAction in CakePHP views.

toReference() inside chains

In phpQuery, QueryTemplates on 25.01.2009 at 15:26

There is one extremely useful and not-so widely known toReference() method. It saves actually matched elements inside variable, by reference (as the name says). It can be used to target such cases as:

1. Preserving chain

// declare our variables (this is VERY important)
$deepClass = $section = null;
$template['div.main div.otherclass .deep-class:first']
  ->toReference($deepClass)
//  ->...  // do something
  ->find('.section')
    ->toReference($section)
//  ->...  // do something
  ->end()
  ->next()
    // highlighter but; only one empty()
    ->empty()
    ->append($deepClass->contents())
    ->eq(0)->add($section)->eq(1)
//      ->...  // do something, stack is [$section]
    ->end()->end()->end()
//      ->...  // stack is same as after ->next()
;

2. Splitting chains

// declare our variables (this is VERY important)
$row = $titleBody = null;
$template
  ->find('ul:first > li')
    ->loopOne('posts', 'postNum', 'r')
      // add dynamic class
      ->addClassPHP('if (! $postNum) print "first"')
      ->find('> .title, > .body')
        ->varsToStack('r["Post"]', $postFields)
        // save our $titleBody variable
        ->toReference($titleBody)
      ->end()
      // save our $row variable
      ->toReference($row)
//    ->...  // lots of code ;)
;
// just continue your work
$row->find('h3:first, .comments')
// ->...
;
// anywhere...
doSomethingOnFields($titleBody);

In SVN version of QueryTemplates there is a fix for cache issue. The workaround for beta2 is to do “if ($ref)” before, or turning off the cache with “$cacheTimeout = -1″.

Extending QueryTemplates with closures

In phpQuery, QueryTemplates on 25.01.2009 at 14:24

Lastest changes in phpQuery allows us to extend it much more easily. One of the ways to do this is directly same as in jQuery, using phpQuery::extend() method. When you combine this with upcoming closures in PHP 5.3, you will feel like-almost-in-browser. Take a look at this:

$source = dirname(__FILE__).'/table.html';
$rows = array(
	array(
		'field-1' => 'foo1',
		'field-2' => 'foo2',
	),
	array(
		'field-1' => 'bar1',
		'field-2' => 'bar2',
	),
);
// this will work in PHP 5.3
// this code will evaluate on onLoad event, thus is cache firendly
$onload = function() {
  $tableFirstRowClass = function($self, $var, $classname = 'first') {
    return $self->addClassPHP("
      if ($$var == 0)
        print '$classname';
    ");
  };
  // here we pass new closure into extend method, just like in jQuery
  // compact() creates array for us
  phpQuery::extend('phpQueryObject', compact('tableFirstRowClass'));
};
require template('table')->parse($source)
  // just bind it
  ->bind('onload', $onload)
  // just fire it up ;)
  ->trigger('onload')
  ->find('table > tr, table > * > tr')
    ->loopOne('rows', 'i', 'row')
      ->varsToSelector('row', $rows[0])
      // now you can use this just-like-this
      ->tableFirstRowClass('i')
;

But for now to extend phpQuery on-the-fly we have to use create_function or delegate existing one.

We can also use new Scripts plugin feature, which does almost the same. Just like all Scripts’ plugin script files, newly attached function will have all available variables. Example below works in PHP < 5.3:

function onload() {
  // in PHP 5.3, this can be done simply like this
//  phpQuery::script(...
  // but for now we have to use fake phpQuery::$plugins namespace
  phpQuery::$plugins->script('tableFirstRowClass', create_function(
    '$self, $params, &$return, $config', '
    $className = ! $params[1] ? "first" : $params[1];
    $self->addClassPHP("
      if ($$params[0] == 0) print '$className';
    ");
  '));
};
$onload = 'onload';

One major difference between extending Scripts and phpQuery itself is way of using new functions. With our new script, we have to small change to main code.

// using phpQuery::extend()
//->tableFirstRowClass('i')
// using Scripts extend
->script('tableFirstRowClass', 'i')

Utilizing Mutation Events for automatic and persistent event attaching

In Ideas, Snippets on 19.01.2009 at 19:43

Most of you probably heard about Event Delegation, a technique where you bind to event of interest on parent node and when triggered event bubbles from child node, action in taken inside the parent. This pattern is widely popularized by reglib and Live Query (plugin for jQuery). Newly released jQuery 1.3 has it’s own implementation thou $.live() method.

This patten has couple of advantages over classic attaching model. First, it’s faster for BIG number of (same) nodes. Secondly, which is most important i think, it allows easy content exchange via AJAX. But with all that it has also disadvantages and main is that you doesn’t really attach to a node, only to it’s parent. That means when you relocate node in the DOM it looses it’s behavior.

That’s why i’ve came with the idea of using Mutation Events to take best parts from both patterns. Mutation Events aren’t supported in IE, but i’ve rolled out some compatibility layer to target this problem. Works also with document fragment appends introduced in jQuery 1.3.

You can go straight to the demo, which was tested and works with:

  • Firefox 3 and 2 linux
  • IE6 SP2 win32
  • Opera 9.63 linux
  • Safari 3.6 win32
  • Chrome 1.0 win32

First lets see main code:

function attachEvents(node) {
	// node is an optional callback for filter()
	// that applies rules only for new nodes
	node = node || '*';
	$('.container1 .trigger1').filter(node).click(function(e) {
		alert('mutation events work');
		e.preventDefault();
	});
}

This is simple, common function that applies behaviors to the document. The only major difference from other such functions is node parameter, used as callback for filter() method, which limits found nodes to interesting ones. Now the actual event bindings:

// node inserted into DOM
$(document).bind('DOMNodeInserted', function(e) {
	attachEvents(function(node){
		return !e.target.skipEventAttaching &&
			this == e.target
	});
});
// node removed from DOM
$(document).bind('DOMNodeRemoved', function(e) {
	// add information property, that node has been detached
	e.target.skipEventAttaching = true;
});

Here the magic happens ;) First handler fires up our event attaching function with callback for filter as a parameter. Callback will check if found node is target of the mutation event or is it reinserted. Second handler adds flag about reinserting.

You can see demo which compares this technique with jQuery 1.3 and reglib. It consists of 3 different implementation:

  • jQuery 1.3 using Mutation Events
  • jQuery 1.3 using $.live
  • reglib

Test it like this: click add node (couple of times), click .trigger (first one, last one) then do move node and again click moved node. Each time when clicking on .trigger, one (and only one) alert should pop out saying that certain technique works.

Additionally there’s a IE compatibility for mutation events in conditional comment. Great help writing it was from ieproto example.

Of course this is proof-of-concept and souldn’t be used in production until fully tested and fixed.

Test cases for websites using phpQuery and SimpleTest

In Ideas, phpQuery on 13.01.2009 at 23:09

Using phpQuery and some UnitTest framework (SimpleTest in this example) you can automatically test web page for presentence of specific part and it’s position. Set of such tests can save a lot of time during website development and after that even more.

Not only you can do simple tests like “are articles visible” but using WebBrowser plugin you can test whole process, let’s say a user registration. Example of such test i would like to present. This test will include following steps:

  1. Enter main page and follow the registration link
  2. Fill registration form and submit it
  3. Check if result is expected

Like i sad before, SimpleTest is framework of choice, but it doesn’t matter so much. What’s important:

  • WebBrowser needs callbacks
  • Callbacks should be declared as functions (for PHP < 5.3)
  • Inside callbacks, $this variable is unavailable
  • First step of the test should check if last step has succeeded

Full code below:

require('simpletest/autorun.php');
require('phpQuery/phpQuery.php');

class CustomerTest extends UnitTestCase {
	public static $_this;
	public static $registration = array(
		'username' => null,
		'success' => false
	);
	function testRegistration() {
		self::$_this = $this;
		phpQuery::browserGet('http://localhost/tested-site/',
			array('CustomerTest', '_testRegistrationLink')
		);
		$this->assertTrue(
			self::$registration['success'], "Registration unsuccessful"
		);
	}
	function _testRegistrationLink($browser) {
		$registrationLink = null;
		$browser->find('a:contains(rejestracja)')
			->WebBrowser(array('CustomerTest', '_testRegistrationForm'))
			->toReference($registrationLink)
				// jump to _testRegistrationForm
				->click();
		self::$_this->assertTrue(
			$registrationLink->length, "Registration link missing"
		);
	}
	function _testRegistrationForm($browser) {
		$registrationForm = null;
		$username = md5(microtime());
		$browser['.customers.form form']
			->toReference($registrationForm)
			->WebBrowser(array('CustomerTest', '_testRegistrationResult'))
			->find('input[name*=login]')->val($username)->end()
			->find('input[name*=email]')->val($username.'@test.com')->end()
			// jump to _testRegistrationResult
			->submit();
		self::$_this->assertTrue(
			$registrationForm->length, "Registration form missing"
		);
		self::$registration['username'] = $username;
	}
	function _testRegistrationResult($browser) {
		$loginForm = $browser->find('h2:text(Logowanie)');
		self::$_this->assertTrue($loginForm->length, "Login form missing");
		if ($loginForm->length)
			self::$registration['success'] = true;
	}
}

WebBrowser doesn’t support AJAX, so not all sites can be tested like this (although you can do it with AHAH after some work), but cookies and HTTP authentication should satisfy most needs.

Of course that’s noting new, projects accomplishing similar goal exist quite time now, eg jWebUnit, but neither of them have jQuery under the hood  ;)

Follow

Get every new post delivered to your Inbox.