Archive for the 'language features' Category

10
Aug
09

some groovy metamagic in practice


When writing scripts, you’re not always interested in having a large number of classes. I know I want to get things done quick.

Java’s standard library already contains most of the stuff you’d need to use. But … it’s too verbose!
This reads a file line by line and prints it to stdout :


import java.io.*;
import java.net.*;

public class xr {
    public static void main(String[] args) {
        try {
            BufferedReader br = new BufferedReader(new FileReader(args[0]));
            while(true) {
                String line = br.readLine();
                if(line == null)
                    break;
                System.out.println(line);
            }
            br.close();
        } catch(Exception e) {
            e.printStackTrace();
        }
    }
}

This is the Groovy equivalent :


if(args[0]) {
   new File(args[0]).readLines().each {
      println it
   }
}

That was only a fraction of what Groovy is capable of doing.

In a dynamic language, you skip the type declaration, and sometimes bad objects get sent to some functions.
For example, today at work I kept receiving this error :

Caught: groovy.lang.GroovyRuntimeException: Could not find matching constructor for: java.io.File(java.io.File)
	at xr.run(xr.groovy:1)

Some of my methods were expecting a String parameter, which they would convert to a File object. But … they received File objects, and everything blew up. One way of solving this is to make sure you’re only sending Strings, or if the parameter’s a File, you can convert it to a String, and then the method will create a File out of it.

Bad logic, right?

It turns out there’s another alternative to this, a groovier way. What if we add another constructor to the File class? If this were Java, we would have had to open the source file, add the constructor and recompile.
Luckily, Groovy lets us add behavior at runtime.


File.metaClass.constructor << { File param ->
   return new File(param.absolutePath)
}

This solved my problem. Even though it ( probably ) wasn’t the greatest solution to my problem, for me it was a great fix.

Another script I wrote was executing another binary, whose output it captured and processed. The string needed to execute the binary was build at runtime, but somewhere along the way, I kept receiving exceptions. So … I thought about adding some traces. Because I didn’t want to add println statements only so that I could remove them when things started to work, I turned to another metaprogramming solution: method interception.
Basically, this allows you to do something to a method. For example, you could modify the arguments before passing them to the method, or you could add traces ( like in my case ). Here’s the code you I used with the String class:


String.metaClass.invokeMethod = { method,args ->
    def metaMethod = delegate.metaClass.getMetaMethod(method,args)
    println "executing $method with args $args on $delegate"
    return metaMethod.invoke(delegate,args)
}

and that would show you a message when any of String’s methods get called. It can be of great help, trust me 🙂

Another thing you can override is how the properties are being accesed. For example this two snippets of code are equivalent:

first

def a = new File("file.txt")
println a.getAbsolutePath()

second

def a = new File("file.txt")
println a.absolutePath

The difference is, the second version won’t show up when you add tracing to the method calls. If you want those to show up as well, you write code like the following :


File.metaClass.getProperty = { prop ->
	println "called prop $prop"
	def metaProp = delegate.metaClass.getMetaProperty(prop)
	return metaProp.getProperty(delegate)
}

I hope you can put this information to good use … I use these all the time 🙂

14
Jul
09

hacking python at runtime:a cool way of modifying your scripts


I recently discovered the _ast module. Using it, one can process Python’s syntax trees.

I’m going to illustrate the use of this module with a simple example. Here’s the code snippet we’re going to work on :


import geo
for file in os.listdir("."):
	print file

Let’s talk a bit about this snippet.

It’s kind of obvious this script has runtime errors. On my Python installation there’s no module named geo … chances are there’s not one on your system either 🙂
Another error would be the call to os.listdir. This is not an error per-se, because if you import the os module, no exception will get thrown at runtime. We can easily notice that the os module has not been imported.

Here’s what we’ll accomplish using the _ast module :

  • remove the import of the module geo
  • add an import to the os module
  • to make things more interesting, we’ll change the “.” argument of the listdir function to the “D:\\” drive

So, how do we get the syntax tree? Pretty simple :


import _ast

source_code = """
  import geo
  for file in os.listdir("."):
    print file
"""
ast = compile(source_code,"<string>","exec",_ast.PyCF_ONLY_AST)

The compile function will build an AST object. The members of the AST can be accessed through the ast.body list. In this example, ast.body[0] will be the import statement, and ast.body[1] will be the for statement.

So, now we have an AST!

The first thing we’ll do is clone the import object. I’m doing this so that I don’t have to create an import object manually. I’m lazy, I know. If you don’t know how to clone a Python object, the following snippet illustrates it :


import copy
an_object_copy = copy.deepcopy(an_object)


With this import clone, I want to import the os module. But, since this clone still has geo as it’s argument, we need to change that. We change that with the following snippet :


os_import = copy.deepcopy(ast.body[0])
os_import.names[0].name = "os"


This object is now equivalent to the following code:


import os


This is nice, but we have to add it to the AST. I’ll take advantage of this to remove the import geo statement:


# remove the import geo statement
ast.body.remove(ast.body[0])
# we insert the import os as the first statement
ast.body.insert(0,os_import)


Right now, the code is runnable. No exceptions will be thrown. Before we run it, let’s accomplish the final task too. Let’s modify the argument of listdir from “.” to “D:\\”.We know that the for object will be the second object in the list:


for_obj = ast.body[1]
# change the argument of listdir to D:\
for_obj.iter.args[0].s = "D:\\"


This changes the argument. In case the attributes I’m setting seem magic, you can find them out using Python’s introspection system ( it’s how I found them too ). You can use this system from ipython, or even your python interpreter, by calling the dir function on any object. This call will list the name of the methods the object has.
Now that we have modified the AST, we need to transform it into runnable code. We do that by calling the compile function :


code = compile(ast,"<string>","exec")


We can run the code with the exec function :


exec code

Here’s the full code of the script:


import _ast
import copy

def fix_source(source_string):
	ast = compile(source_string,"<string>","exec",_ast.PyCF_ONLY_AST)
	# clone the import object, so we can modify it
	os_import = copy.deepcopy(ast.body[0])
	# remove the import geo statement
	ast.body.remove(ast.body[0])
	# change the import argument to os
	os_import.names[0].name = "os"
	# add the import os statement to the ast
	ast.body.insert(0,os_import)
	for_obj = ast.body[1]
	# change the argument of listdir to D:\
	for_obj.iter.args[0].s = "D:\\"
	# transform the AST into something runnable
	return compile(ast,"<string>","exec")
	
if __name__ == "__main__":
	source_code = """
import geo
for file in os.listdir("."):
	print file
"""
	code = fix_source(source_code)
	exec code

I’m sure you can put this trick to use. It’s one of the greatest “hacks” I know.

I’ll try to post a Ruby alternative as soon as time allows me.

09
Mar
09

cool python debugging trick


I’ve been debugging on some code, and I thought it would be great to be able to decorate functions that would perform something when called with a desired argument. I asked about in stackoverflow, and MizardX provided me with a great solution ( which I’ve hacked a bit ). Here it is :

from functools import wraps
import pdb	

def intercept(target=None,**trigger):
    def decorator(func):
        names = getattr(func,'_names',None)
        if names is None:
            code = func.func_code
            names = code.co_varnames[:code.co_argcount]
        @wraps(func)
        def decorated(*args,**kwargs):
            all_args = kwargs.copy()
            for n,v in zip(names,args):
                all_args[n] = v
            for k,v in trigger.iteritems():
                if k in all_args and all_args[k] == v:
					if not target is None:                    
						return target(all_args)
					else:
						pdb.set_trace()
            return func(*args,**kwargs)
        decorated._names = names
        return decorated
    return decorator

Assuming we have the following function:

def whatever(a,b):
	# do something with a and b

We can do some cool hacking on it :

  • by decorating it like this : @intercept(a=20), the python debugger will kick in and stop the function’s execution if a was passed the value 20
  • by decorating it like this : @intercept(function_here,@a=20), the function will be called and the debugger won’t fire up

Cool, right?

09
Mar
09

decorators are pretty useful


Yesterday I wrote about a java indenter I was working on. I thought it worked just fine, but I found a source file where it didn’t perform how it should. Given that it was written in python, and there were pretty many function calls, I decided to use a decorator to do the tracing. One of the reasons I went with this decision was so I could learn more about them. After a small problem I had with the self argument not getting passed on to the function, everything went fine. The decorator first looked like this:


def trace(func):
	def ofunc(*args):
		func_name = func.__name__		
		xargs = args		
		print "entering %s with args %s" % (func_name,xargs)
		ret_val = func(*args)
		print "return value |%s|" % ret_val
		print "exiting %s\n" % (func_name)
		return ret_val
	return ofunc 

… and then it produced too much output, and the print statements were slowing the script a lot. It was to be expected, right? I guess, but still, some speed gains would be appreciated, right?

So, I’ve decided to switch the print statements inside the trace decorator with StringIO:


def trace(func):
	def ofunc(*args):
		global strio		
		func_name = func.__name__		
		xargs = args		
		print>>strio, "entering %s with args %s" % (func_name,xargs)
		ret_val = func(*args)
		print>>strio, "return value |%s|" % ret_val
		print>>strio, "exiting %s\n" % (func_name)
		return ret_val
	return ofunc 


The performance gain was noticeable indeed, but, there wasn’t a way to get the resulting string back. Enter another decorator! I decided the tracing would be over when the object would be garbage-collected, so I decorated the __del__ method with the decorator end_trace:


def end_trace(func):
	def pfunc(*args):
		global strio
		print strio.getvalue()
		strio.close()
		return func(*args)
	return pfunc	

Case solved! I think the source code indenter is working fine now, but, I’ll test it some more before I post the source code.




Blog Stats

  • 218,505 hits