August 31, 2008 at 7:39 pm · tags: python twisted repl
As you all already know TwistedMatrix is great to write asynchronous event-driven network oriented programs: define how your protocol responds in case of events, attach some callbacks if you need them, wrap it in a factory and activate the reactor.
The reactor runs a giant loop in which events are processed in a non-blocking fashion. Sometimes, though, everything a man needs it’s just to make it stop. At least for a while, at least for the sake of getting data from the user.
The most prominent example of a command line client program that needs to stop and wait is a REPL. The following kind of REPL is a bit unorthodox, you’ll see.
The secret sauce (at least the one I found) to write such kind of interactive program in Twisted is twisted.internet.stdio.StandardIO. It connects your protocol to standard input and output:
class Repl(basic.LineReceiver):
delimiter = '\n'
prompt_string = 'cmd> '
def prompt(self):
self.transport.write(self.prompt_string)
def connectionMade(self):
self.sendLine('Welcome to Console')
self.prompt()
def lineReceived(self, line):
# blank line
if not line:
self.prompt()
return
self.issueCommand(line)
def issueCommand(self, command):
# send the command to the server
d = sendCmd("%s%s" % (command, self.delimiter))
d.addCallback(self._checkResponse)
def _checkResponse(self, args):
success, num_lines, data = args
if num_lines > 20:
# use less to display the response
self.lessify(data)
else:
self.sendLine(data)
self.prompt()
def lessify(self, data):
p = subprocess.Popen(["less"], stdin=subprocess.PIPE)
p.communicate(data)
def connectionLost(self, reason):
reactor.stop()When the protocol it is connected to the standard output with
stdio.StandardIO(TaskConsole())
reactor.run()
the program displays the prompt, hence when a line is received from the standard input it is sent to the other protocol attached on the network and a callback is registered for the response. I also decided to use less to display the response if it’s more than some lines but that’s a detail.
The sendCmd function instantiate the networked protocol and its factory:
def sendCmd(cmd):
factory = CFactory(cmd)
reactor.connectTCP('127.0.0.1', 1234, factory)
return factory.deferredThen, when the server replies with some content we check to see if everything went ok and the reconstruct the whole response sending it back the REPL:
class Client(basic.LineReceiver):
delimiter = '\n'
def connectionMade(self):
# send the command received by the cmdline to the server
self.sendLine(self.factory.cmd)
self.buffer = []
self.cmd_success = True
def lineReceived(self, line):
# basic check error/success
if line.startswith('OK'):
return
if line.startswith('ERR'):
self.cmd_success = False
return
if line == 'END':
# join the response at the end of it
self.responseFinished(
len(self.buffer), "\n".join(self.buffer))
self.buffer = []
else:
self.buffer.append(line)
def responseFinished(self, num_lines, data):
# disconnect
self.sendLine('quit')
self.transport.loseConnection()
# send back the response to the REPL
self.factory.deferred.callback((
self.cmd_success, num_lines, data))
class CFactory(protocol.ClientFactory):
protocol = Client
def __init__(self, cmd):
self.cmd = cmd
self.deferred = defer.Deferred()How cool is that? Not really to be honest. It has a big gigantic fault: every time you input a line a connection to the server is opened and closed. That’s bad, really bad.
It didn’t take too long to create a version that use just one connection:
def connectionMade(self):
self.sendLine('Welcome to Console')
self.factory = CFactory()
self.connector = reactor.connectTCP(
'127.0.0.1', 1234, self.factory)
self.prompt()We store the connector and the factory. issueCommand does not open a connection anymore, just:
def issueCommand(self, command):
self.connector.transport.write("%s%s" % (command, self.delimiter))
self.factory.deferred.addCallback(self._checkResponse)We write directly to the transport of the connector (and not the one connected to the stdout) and register the callback on the factory’s deferred.
That’s better in my opinion and a nice start. I know that within twisted.conch.stdio there’s something more evolved. I’ll try to look into it when I have more time.
You can find the first version (bad) and the second version (better) online.
What do you think about this try?
July 13, 2008 at 3:08 pm · tags: joblife python
I am still alive, just don’t have time to blog something meaningful.
By the way, this is what I am doing/I have done recently:
- Patched httplib2 to support MD5-sess (at work)
- Patched soaplib to use httplib2 with the above patch (at work)
- All the above is because we are interfacing with a couple of SOAP servers (in Java and .NET)
- Reading The Ruby Way by Hal Fulton and JavaScript the Good Parts by Douglas Crockford (read) Dreaming In Code by Scott Rosenberg
- Planning my vacation: I’ll be staying in London for 3 weeks, from the 27th of July to the 17th of August. If you want to catch up and drink something contact me at l dot oluyede at gmail dot com
- Bought a Nokia E51 which is UMTS/HDSPA/WiFi ready Symbian phone (and also cheap)
- Still reading a lot of stuff in my backlog
May 19, 2008 at 9:02 am · tags: Pinder Python
In this new release of my Campfire API, Pinder 0.6.5, I fixed some bugs and added the methods ping() and topic() (which matches change_topic()) to the Room objects.
There is by the way an incompatible change: I do not distribute anymore BeautifulSoup and httplib2 with the library.
Go take it!
May 7, 2008 at 2:05 pm · tags: pycon Python
I opened a Twitter account for the PyCon Italy conference. I will try to keep updated as soon as things come up and the conference starts on Friday.
http://twitter.com/pyconit
April 14, 2008 at 1:19 pm · tags: Django googleappengine Python
While I was boring myself to death last weekend and while everybody was talking about it I came up with a sample application.
Now I can talk about Google AppEngine as well 
It is simply a pastebin using the DataStore API, the webapp framework, the Users API, pygments and Django templates.
It is heavily inspired by http://dpaste.com
http://share11.appspot.com
The great thing? It took me less than an afternoon (mostly reading the framework docs) and it is ~200 LOC. Even greater? One shell command to upload and deploy
April 9, 2008 at 9:10 pm · tags: pycon Python
We had to make some schedule updates for the conference.
The savory new flavor of Py2.6 and Py3.0 is the title of the second day’s keynote. Who’s the speaker? Raymond Hettinger
The talk about callbacks and Python patterns have been replaced by a talk about the now utterly famous Google AppEngine.
The speakers of the PyPy talk are Antonio Cuni and Samuele Pedroni.
The title of Stallman’s keynote is Free Software in ethics and in practice.
April 9, 2008 at 10:30 am · tags: pycon Python
UPDATE: Raymond Hettinger will hold the keynote of the second day!
PyCon Due is definitely taking shape.
The conference will take place on a three day span.
The first day venue (free of any kind of charge) will be Palazzo Vecchio, in Florence. The conference will be introduced by the spokeswoman of Arts of the city. The opening keynote will be given by Richard Stallman, a person who needs no introduction in the software world.
After that, which is definitely important, we plan to go out and eat all together at a restaurant. Social life is equally important
The actual conference will begin the next day, the 10th of May, divided in three parallels track: a tutorial track, an introduction one and one based on experiences and real life usage of the language.
We’ll have talks about Python for beginners, various incarnations of Python, Plone, information retrieval, PyQt, nginx and WSGI, Cython, advanced uses of Django, PyMaemo, Unicode, PyPy, Zope 3 and more on that day. There will be also a Skype sponsored talk about Skype4Py.
The second day will be full of talks about profiling and debugging, SQLAlchemy, Django again, callbacks and patterns, map sharing, concurrency, Twisted Matrix, IronPython, Ajax, compilers in Python, the Fedora Unified Network Controller, FlyPDF and component architectures and more.
As you can imagine will be a tough three days conference, one an Italian (Python) developer should not miss in my opinion.
A lot of well known speakers: Alex Martelli (another who needs no introduction), Federico di Gregorio (author of psycopg2), Arkadiusz Wahlig (author of Skype4Py), Manlio Perillo (of nginx’s mod_wsgi fame), Antonio Cuni (from the PyPy team), Giovanni Bajo (mantainer of PyInstaller and of GCC fame), Brian Fitzpatrick (Subversion anyone?), Menno Smits (from the Resolver Systems team), Michele Simionato (of metaclasses, decorators and mro fame) and more.
As part of the board of the conference I’m a little biased by I’m really looking forward for this one 
See you there!
April 4, 2008 at 10:24 pm · tags: pycon Python
UPDATE: I forgot to say that there will be simultaneous translation for non-italian speakers and attendants.
PyCon Due Italy is coming, here the press release (in Italian):
Firenze, 27 Marzo 2008: Sono finalmente aperte le iscrizioni a PyCon Due, la seconda conferenza italiana dedicata al linguaggio di programmazione Python, che si terrà a Firenze il 9, 10 e 11 Maggio 2008.
Dopo il largo successo di pubblico e critica ottenuto da PyCon Uno l’anno scorso, l’Associazione di Promozione Sociale “Python Italia” organizza quest’anno un evento ancora più ambizioso. Sono attesi infatti più di 300 tra professionisti, studenti e ricercatori, per una tre giorni intensa di appuntamenti e interventi imperdibili.
L’evento di apertura si svolgerà Venerdì 9 Maggio alle ore 15:00 nel prestigioso Salone De’ Cinquecento in Palazzo Vecchio dove, dopo il saluto da parte di Lucia De Siervo (Assessore all’Informatizzazione del Comune di Firenze, che ha donato all’evento il suo Patrocinio), Richard Stallman (fondatore del movimento del Free Software, inventore della licenza GPL e ideatore del progetto GNU) terrà un keynote sul tema “Free Software e Free Ethics”. Questo evento di apertura è ad ingresso libero (fino ad esaurimento posti, ma con priorità a chi è registrato al PyCon).
In seguito, nelle giornate di Sabato 10 e Domenica 11, all’Auditorium Al Duomo, si terranno numerose conferenze dedicate al linguaggo Python, su tre track parallele. Parteciperanno speaker internazionali di fama mondiale come Alex Martelli, Samuele Pedroni, Brian Fitzpatrick, e molti altri. Sono previsti anche interventi dedicati a programmatori che si avvicinano al linguaggio per la prima volta.
Inoltre, sono previste sessioni speciali come la sessione di recruiting (dedicate ai programmatori in cerca di nuova occupazione, e alla quale parteciperanno le aziende con posizioni lavorative aperte), momenti di svago e relax serale, e l’estrazione di regali offerti dagli sponsor (che includono aziende di caratura internazionale come Skype e Google).
Per maggiori informazioni, potete visitare il sito Internet ufficiale dedicato all’evento. Registrandosi in anticipo, inoltre, sono previsti forti sconti sul biglietto d’ingresso.
If you are at least midly interested in the Python word, come!
March 24, 2008 at 4:42 pm · tags: HTTP Python REST
Dave Benjamin summarized on his blog 10 reasons why XML-RPC should be better than REST but I think he completely missed the point about REST APIs. Let’s see:
1 - Standard, cross-language, typeful serialization of data
REST is definitely standard and cross-language. Regarding serialization, you can transfer what you want where you want but if your goal is persisting to make a local resemblance of what you have remotely, I think that’s bad. What we, as developers, really care is having the possibility to use a service remotely with an eye on scalability and transparence. Hence, HTTP
2 - User-defined error codes and messages
Nobody forbids anyone to invent your own headers and place whatever meaning you want upon them. With HTTP you can surely do that. Even whole extensions have been written.
3 - “Boxcarring” of requests to reduce overhead
Box carring is not really an advantage, there are other ways to reduce overhead. See also 207 multi status
4 - Serialization of binary content
Who said REST protocols do not support binary data? You can send whatever you want from the client to server, place the right content type and you’re done. See also media resources in the Atom Publishing Protocol.
5 - Serialization of date-time values
Do I really have to answer that?
6 - Standardized parameter passing
And what if I want to pass a parameter that’s not on the list of the allowed types in the specification? Remember, SOAP has born also to address some of the limitations of XML-RPC.
7 - Introspection allowing for straightforward code generation
Let me say one thing once and for all. Code generation for this purposes is bad, and almost useless (if you are used to highly dynamic languages). Anyway, REST through hypertext is implicitly introspection enabled. Also, documentation of APIs is there for us
8 - High-level APIs for just about every language
I think there’s no need to say that each and every language has some sort of HTTP client.
9 - No manual parsing of XML, ever
This point does not really make sense to me. If one XML-RPC method does return XML as content, you have to parse it because is part of the application domain. The same for SOAP or REST. If, as Dave mentioned, the language has some sort of high-level API for the protocol you don’t have to parse what’s on the wire manually: no manual parsing of XML-RPC responses, or SOAP envelopes or HTTP responses.
10 - Only three lines to call a function in Python and several other languages
This is really pointless. Using a REST API from httplib2 or restclient has the same brevity.
I think Dave did not get REST at all, his conclusion speaks for him:
Not that the REST doesn’t have its benefits, but someone ought to be saying this. XML-RPC isn’t complicated like SOAP, it runs just about everywhere, and it lets you get on with your work rather than arguing about semicolons versus slashes or XML versus JSON or countless other things. Besides, when your goal is to support as many languages as possible, you want to minimize the amount of code you write for each language. As far as I’ve seen, nothing else accomplishes literally no-code binding like XML-RPC.
I suggest reading RESTful Web Services, it’s really a good book. I also suggest reading the latest article wrote by Stefan Tilkov addressing the most popular REST doubts.
December 27, 2007 at 4:21 pm · tags: Python Python SVN
Here we are with another update from Python 2.x SVN.
Old disassembly:
0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_CONST 1 (1)
7 ROT_TWO
8 LOAD_CONST 2 ('x')
11 STORE_SUBSCR
12 DUP_TOP
13 LOAD_CONST 3 (2)
16 ROT_TWO
17 LOAD_CONST 4 ('y')
20 STORE_SUBSCR
New disassembly:
0 BUILD_MAP 0
3 LOAD_CONST 1 (1)
6 LOAD_CONST 2 ('x')
9 STORE_MAP
10 LOAD_CONST 3 (2)
13 LOAD_CONST 4 ('y')
16 STORE_MAP
Another optimization has been added: BUILD_MAP now has a meaning. It’s the estimated size of the dictionary. Allows dictionaries to be pre-sized (upto 255 elements) saving time lost to re-sizes with their attendant mallocs and re-insertions. Has zero effect on small dictionaries (5 elements or fewer), a slight benefit for dicts upto 22 elements (because they had to resize once anyway), and more benefit for dicts upto 255 elements (saving multiple resizes during the build-up and reducing the number of collisions on the first insertions). Beyond 255 elements, there is no addional benefit.
Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for compatibility are available.
Added signal.set_wakeup_fd(). There is a correspondent C API as well: PySignal_SetWakeupFd.
Slightly change in __ hash __ behavior and rich comparison: __ hash __ can be inherited when one __ lt __, __ le __, __ gt __, __ ge __ are overridden, as long as __ eq __ and __ ne __ aren’t.
Improved performance of built-in any()/all() by avoiding PyIter_Next().
Next entries »
