Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

 Python -- Scripting language which supports coroutines

This is the forth page of an ongoing series of pages covering scripting language topics for the beginning programmer (others cover Unix shells, TCL and Perl ) 

Newer versions of Python (starting with 2.2) supports co-routines in a platform independent way (via generators concept).  I think that this makes Python in certain area a better scripting language then other members of the "most popular troika" (Perl, PHP and JavaScript). Availability of coroutines favorably distinguish Python from faster, richer (because of CPAN) and better integrated with regular expressions concept (as well as Unix) Perl.

Now Ruby competes with Python in this area and programmers who value coroutines paradigm of software development (and it is really paradigm, not a language feature) can try both languages and compare the level of integration and power provided (my impression is that Ruby is slightly better in thin area). 

Still Python enjoy support of Google and that is still no similar sponsor for Ruby. Look at the history of  JavaScript which survives as an abandoned language, but paid heavy price both in terms of speed of development of the language and popularity :-(.

Some of the main reasons that Python is popular as a first language is that it is more or less forgiving and what is more important is more helpful about syntax errors (due to its complex lexical structure and syntax Perl is as horrible in that as one can get -- real nightmare).

Python also has another innovative aspect (new is a well forgotten past - FORTRAN 4 used an indentation to distinguish between labels and statements ;-):  it uses indentation to determine nesting of statements. Multiline statements are marked like a multiline UNIX command: with a backslash. Multiple close of the blocks become just the matter of appropriate amount of moving nesting to the left. 

Another attractive feature is clean and powerful I/O statements, although Python lacks the power of the elegant Perl open statement).

Finally, Python scales quite well from learning tool to professional instrument. One can learn it at a basic procedural level and then learn co-routines based programming  in a relatively short period of time (only Modula permits the same flexibility).

Although Python is a  scripting language that has application area similar to Perl, similarities seem to end quickly. They're not overly similar in implementation details, nor even remotely similar in syntax. Their extension mechanisms also take entirely different directions.  Perl's motto seems to be TIMTOWTDI with an attitude of "whatever gets the job done", where as Python seems to follow the practice of KISS, preferring simplicity and consistency in design to flexibility. 

We all understand that now not necessary the best language win. Fortunately there are several positive signs for Python:

1. Jython is one of the few scripting languages that integrates well with Java. That makes it ideal for rapid development and for easily maintained programs.
2. Python has more or less clean interface with C and C++. Modules which do need speed can be replaced later by C++
3. Python has extensive library of modules. Among them:

• pikcle - object serialization
• re - regular expressions
• struct - access to C structs
• ConfigParser - parse config files
• getopt - parse command-line options
• socket (also threading. SocketServer, )
• cgi (also CGIHTTPServer, urllib,urlparse - parse a url into (scheme, netloc, path, parameters, query, fragment)
• anydbm - access to DBM-style databases
• httplib - HTTP client
• ftplib - FTP client (also poplib, imaplib, nntplib, smtplib, telnetlib)
• SimpleHTTPserver
• htmllib (also sgmllib and xmllib)
• rfc822
• mimetools
• sha

Indentation style in Python is actually an interesting innovation that very few people understand. But making this revolutionary step -- relegating block brackets to the lexical level of blank space and comments Python failed to make a necessary adjustment and include pretty printer into interpreter (with the possibility of creating pretty printed program from comments). Such a pretty printer actually needs to understand two things: the format of comments that suggest indenting like

//{ <label>

//} <label>

and the current number of spaces in the tab like  pragma tab = 8. The interesting possibility is that in pretty printed program those comments can be removed and after a reading pretty printed program into the editor reinstated automatically. Such feature would be extremely cool and can be implemented in any scriptable editor like THE, Emacs (note: If you like the keybindings and programmability of Emacs, but don't like its size, try JED which also has a pretty good Python mode), etc.

Python actually encourages a programmer to use a decent editor but we knew that already, right? The main benefit I see to syntactic indenting is that is narrows down the possible range of coding styles. If you think about it, most of the (dare I say) splintering of C/C++/Java coding styles is due to the placement of the { and } symbols. This acts against readability for other developers. Since there are no braces, there are no style wars over where to put the braces and that is a very important advantage for any environment with several supersizes ego.

An often overlooked advantage of this Python feature is not only that language saved two important for any scripting language symbols for other uses, but that such solution automatically leads to more compact (as for the number of lines) programs (deletion of curly brackets usually help to lessen the number of lines in C or Perl program by 20% or more). 

My impression is that few people understand that C  solution for blocks ({ } blocks) was pretty weak in comparison with its prototype language (PL/1): it does not permit nice labeled closer of blocks like

A:do ... end A;

in PL/1. IMHO introduction of a pretty printer as a standard feature of both a compiler and a GUI environment is long overdue and here Python can make its mark.

Moreover such an approach might help somewhat compensate for all those OO excesses that lengthen the program two or three time in comparison with a pure procedural language like PL/1, Ada or Modula-2.

Here are some companies that are using Python in commercial apps (borrowed from a Slashdot post, now you need to add Goggle to the list):

• Advanced Management Solutions Inc. AMS provides the AMS REALTIME suite of enterprise software for project management, resource management, cost management and timesheets. The Python language engine is embedded in AMS REALTIME as a means of extending the products, and also as a way of enabling custom behavior and company-specific business rules to be supported.
• CWI CWI, Python's home, has used Python in, among other things, GrINS, a 20,000 line authoring environment for transportable hypermedia presentations, and a 5,000 line multimedia teleconferencing tool, as well as many many smaller programs. See the collection of multi-media project papers.
• Digital Creations A long-time sponsor of the PSA, Digital creations develops with Python - and also makes some of their Python software available for free!
• ILU ILU (it's spelled Inter-Language Unification but it's pronounced eye-loo) is a (very) CORBA-ish multi-language object interface system. It has bindings for Common Lisp, C++, ANSI C, Modula-3 and Python.
• Infoseek Ultraseek Server, Infoseek's commercial search engine product, is implemented as an elaborate multi-threaded Python program with the primitive indexing and search operations performed by a built-in module. Most of the program is written in Python, and both a built-in spider and HTTP server can be customized with additional Python code. The program contains over 11,000 lines of Python code, and the user interface is implemented with over 17,000 lines of Python-scripted HTML templates. Try it out on the Python.Org web search page or download an evaluation copy from Infoseek Software.
• eGroups.com (previously Findmail) A comprehensive public archive of Internet mailing lists, implemented in pure Python. Latest statistics from Scott Hassan: 180,000 lines of Python doing everything from a 100% dynamic website to all email delivery, pumping out 200 messages/second on a single 400 MHz Pentium!
• LLNL A group at the Lawrence Livermore National Laboratories is basing a new numerical engineering environment on Python, replacing a home-grown scripting language of ten-year standing. Paul Dubois is a central figure in that effort.
• NASA Johnson Space Center uses Python in its Integrated Planning System as the standard scripting language. Efforts are underway to develop a modular collection of tools for assisting shuttle pre-mission planning and to replace older tools written in PERL and shell dialects. Python will also be installed in the new Mission Control Center to perform auxiliary processing integrated with a user interface shell. Ongoing developments include an automated grammar based system whereby C++ libraries may be interfaced directly to Python via compiler techniques. This technology can be extended to other languages in the future.
• IV Image Systems AB IV Image Systems uses Python for many projects, including a satellite image production system for the Swedish Meteorological and Hydrological Institute (SMHI) (see next entry). This system receives raw data from several weather satellites, and produces images for many purposes, including the satellite images used for the presentation of the daily weather on Swedish TV 4. For more information, contact Goran Bondeson.
• Swedish Meteorological and Hydrological Institute SMHI is the home of the Swedish civilian weather, hydrological and oceanographic services. It's Python-based remote sensing software for automatic product generation, using NOAA and Meteosat data, provides information to bench forecasters, objective analysis schemes, and commercial interests such as the media. At SMHI's Research & Development Unit, a Python-based "Radar Analysis and Visualization Environment"

Nikolai Bezroukov

Old News ;-)

 [Jul 8, 2008] Python Call Graph 0.5.1  by Gerald Kaszuba

About: pycallgraph is a Python library that creates call graphs for Python programs.

Changes: The "pycg" command line tool was renamed to "pycallgraph" due to naming conflicts with other packages.

[Jun 23, 2008] freshmeat.net Project details for cfv

About:
cfv is a utility to both test and create .sfv (Simple File Verify), .csv, .crc, .md5(sfv style), md5sum, BSD md5, sha1sum, and .torrent checksum verification files. It also includes test-only support for .par and .par2 files. These files are commonly used to ensure the correct retrieval or storage of data.

Release focus: Major bugfixes

Changes:
Help output is printed to stdout under non-error conditions. A mmap file descriptor leak in Python 2.4.2 was worked around. The different module layout of BitTorrent 5.x is supported. A "struct integer overflow masking is deprecated" warning was fixed. The --private_torrent flag was added. A bug was worked around in 64-bit Python version 2.5 and later which causes checksums of files larger than 4GB to be incorrectly calculated when using mmap.

[Jun 20, 2008] BitRock Download Web Stacks

BitRock Web Stacks provide you with the easiest way to install and run the LAMP platform in a variety of Linux distributions. BitRock Web Stacks are free to download and use under the terms of the Apache License 2.0. To learn more about our licensing policies, click here.

You can find up-to-date WAMP, LAMP and MAMP stacks at the BitNami open source website. In addition to those, you will find freely available application stacks for popular open source software such as Joomla!, Drupal, Mediawiki and Roller. Just like BitRock Web Stacks, they include everything you need to run the software and come packaged in a fast, easy to use installer.

BitRock Web Stacks contain several open source tools and libraries. Please be sure that you read and comply with all of the applicable licenses. If you are a MySQL Network subscriber (or would like to purchase a subscription) and want to use a version of LAMPStack that contains the MySQL Certified binaries, please send an email to sales@bitrock.com.

For further information, including supported platforms, component versions, documentation, and support, please visit our solutions section.

[Mar 12, 2008] Terminator - Multiple GNOME terminals in one window

Rewrite of screen in Python?

This is a project to produce an efficient way of filling a large area of screen space with terminals. This is done by splitting the window into a resizeable grid of terminals. As such, you can produce a very flexible arrangements of terminals for different tasks.

Read me

Terminator 0.8.1
by Chris Jones <cmsj@tenshu.net>

This is a little python script to give me lots of terminals in a single window, saving me valuable laptop screen space otherwise wasted on window decorations and not quite being able to fill the screen with terminals.

Right now it will open a single window with one terminal and it will (to some degree) mirror the settings of your default gnome-terminal profile in gconf. Eventually this will be extended and improved to offer profile selection per-terminal, configuration thereof and the ability to alter the number of terminals and save meta-profiles.

You can create more terminals by right clicking on one and choosing to split it vertically or horizontally. You can get rid of a terminal by right clicking on it and choosing Close. ctrl-shift-o and ctrl-shift-e will also effect the splitting.

ctrl-shift-n and ctrl-shift-p will shift focus to the next/previous terminal respectively, and ctrl-shift-w will close the current terminal and ctrl-shift-q the current window

Ask questions at: https://answers.launchpad.net/terminator/
Please report all bugs to https://bugs.launchpad.net/terminator/+filebug

It's quite shamelessly based on code in the vte-demo.py from the vte widget package, and on the gedit terminal plugin (which was fantastically useful).

vte-demo.py is not my code and is copyright its original author. While it does not contain any specific licensing information in it, the VTE package appears to be licenced under LGPL v2.

the gedit terminal plugin is part of the gedit-plugins package, which is licenced under GPL v2 or later.

I am thus licensing Terminator as GPL v2 only.

Cristian Grada provided the icon under the same licence.

Python and the Programmer

Python and the Programmer
A Conversation with Bruce Eckel, Part I
by Bill Venners
Jun 2, 2003
 

Summary
Bruce Eckel talks with Bill Venners about why he feels Python is "about him," how minimizing clutter improves productivity, and the relationship between backwards compatibility and programmer pain.

Bruce Eckel wrote the best-selling books Thinking in C++ and Thinking in Java, but for the past several years he's preferred to think in Python. Two years ago, Eckel gave a keynote address at the 9th International Python Conference entitled "Why I love Python." He presented ten reasons he loves programming in Python in "top ten list" style, starting with ten and ending with one.

In this interview, which is being published in weekly installments, I ask Bruce Eckel about each of these ten points. In this installment, Bruce Eckel explains why he feels Python is "about him," how minimizing clutter improves productivity, and the relationship between backwards compatibility and programmer pain.

Bill Venners: In the introduction to your "Why I Love Python" keynote, you said what you love the most is "Python is about you." How is Python about you?

Bruce Eckel: With every other language I've had to deal with, it's always felt like the designers were saying, "Yes, we're trying to make your life easier with this language, but these other things are more important." With Python, it has always felt like the designers were saying, "We're trying to make your life easier, and that's it. Making your life easier is the thing that we're not compromising on."

For example, the designers of C++ certainly attempted to make the programmer's life easier, but always made compromises for performance and backwards compatibility. If you ever had a complaint about the way C++ worked, the answer was performance and backwards compatibility.

Bill Venners: What compromises do you see in Java? James Gosling did try to make programmers more productive by eliminating memory bugs.

Bruce Eckel: Sure. I also think that Java's consistency of error handling helped programmer productivity. C++ introduced exception handling, but that was just one of many ways to handle errors in C++. At one time, I thought that Java's checked exceptions were helpful, but I've modified my view on that. (See Resources.)

It seems the compromise in Java is marketing. They had to rush Java out to market. If they had taken a little more time and implemented design by contract, or even just assertions, or any number of other features, it would have been better for the programmer. If they had done design and code reviews, they would have found all sorts of silliness. And I suppose the way Java is marketed is probably what rubs me the wrong way about it. We can say, "Oh, but we don't like this feature," and the answer is, "Yes, but, marketing dictates that it be this way."

Maybe the compromises in C++ were for marketing reasons too. Although choosing to be efficient and backwards compatible with C was done to sell C++ to techies, it was still to sell it to somebody.

I feel Python was designed for the person who is actually doing the programming, to maximize their productivity. And that just makes me feel warm and fuzzy all over. I feel nobody is going to be telling me, "Oh yeah, you have to jump through all these hoops for one reason or another." When you have the experience of really being able to be as productive as possible, then you start to get pissed off at other languages. You think, "Gee, I've been wasting my time with these other languages."

Number 10: Reduced Clutter

Bill Venners: In your keynote, you gave ten reasons you love Python. Number ten was reduced clutter. What did you mean by reduced clutter?

Bruce Eckel: They say you can hold seven plus or minus two pieces of information in your mind. I can't remember how to open files in Java. I've written chapters on it. I've done it a bunch of times, but it's too many steps. And when I actually analyze it, I realize these are just silly design decisions that they made. Even if they insisted on using the Decorator pattern in java.io, they should have had a convenience constructor for opening files simply. Because we open files all the time, but nobody can remember how. It is too much information to hold in your mind.

The other issue is the effect of an interruption. If you are really deep into doing something and you have an interruption, it's quite a number of minutes before you can get back into that deeply focused state. With programming, imagine you're flowing along. You're thinking, "I know this, and I know this, and I know this," and you are putting things together. And then all of a sudden you run into something like, "I have to open a file and read in the lines." All the clutter in the code you have to write to do that in Java can interrupt the flow of your work.

Another number that used to be bandied about is that programmers can produce an average of ten working lines of code per day. Say I open up a file and read in all the lines. In Java, I've probably already used up my ten working lines of code for that day. In Python, I can do it in one line. I can say, "for line in file('filename').readlines():," and then I'm ready to process the lines. And I can remember that one liner off the top of my head, so I can just really flow with that.

Python's minimal clutter also helps when I'm reading somebody else's code. I'm not tripping over verbose syntax and idioms. "Oh I see. Opening the file. Reading the lines." I can grok it. It's very similar to the design patterns in that you have a much denser form of communication. Also, because blocks are denoted by indentation in Python, indentation is uniform in Python programs. And indentation is meaningful to us as readers. So because we have consistent code formatting, I can read somebody else's code and I'm not constantly tripping over, "Oh, I see. They're putting their curly braces here or there." I don't have to think about that.

Number 9: Not Backwards Compatible in Exchange for Pain

Bill Venners: In your keynote, your ninth reason for loving Python was, "Not backwards compatible in exchange for pain." Could you speak a bit about that?

Bruce Eckel: That's primarily directed at C++. To some degree you could say it refers to Java because Java was derived primarily from C++. But C++ in particular was backwards compatible with C, and that justified lots of language issues. On one hand, that backwards compatibility was a great benefit, because C programmers could easily migrate to C++. It was a comfortable place for C programmers to go. But on the other hand, all the features that were compromised for backwards compatibility was the great drawback of C++.

Python isn't backwards compatible with anything, except itself. But even so, the Python designers have actually modified some fundamental things in order to fix the language in places they decided were broken. I've always heard from Sun that backwards compatibility is job one. And so even though stuff is broken in Java, they're not going to fix it, because they don't want to risk breaking code. Not breaking code always sounds good, but it also means we're going to be in pain as programmers.

One fundamental change they made in Python, for example, was "type class unification." In earlier versions, some of Python's primitive types were not first class objects with first class characteristics. Numbers, for example, were special cases like they are in Java. But that's been modified so now I can inherit from integer if I want to. Or I can inherit from the modified dictionary class. That couldn't be done before. After a while it began to be clear that it was a mistake, so they fixed it.

Now in C++ or Java, they'd say, "Oh well, too bad." But in Python, they looked at two issues. One, they were not breaking anybody's existing world, because anyone could simply choose to not upgrade. I think that could be an attitude taken by Java as well. And two, it seemed relatively easy to fix the broken code, and the improvement seemed worth the code-fixing work. I find that attitude so refreshing, compared to the languages I'd used before where they said, "Oh, it's broken. We made a mistake, but you'll have to live with it. You'll have to live with our mistakes."

Next Week

Come back Monday, June 9 for Part I of a conversation with Java's creator James Gosling. I am now staggering the publication of several interviews at once, to give the reader variety. The next installment of this interview with Bruce Eckel will appear on Monday, June 23. If you'd like to receive a brief weekly email announcing new articles at Artima.com, please subscribe to the Artima Newsletter.

Talk Back!

Have an opinion about programmer productivity, backwards compatibility, or breaking code versus programmer pain. Discuss this article in the News & Ideas Forum topic, Python and the Programmer.

Resources

Bruce Eckel's Mindview, Inc.:
http://www.mindview.net/

Bruce Eckel's essay on checked exceptions: Does Java Need Checked Exceptions?:
http://www.mindview.net/Etc/Discussions/CheckedExceptions

Bruce Eckel's Public and In-House Seminars:
http://mindview.net/Seminars

Bruce Eckel's Weblog:
http://www.mindview.net/WebLog

Python.org, the Python Language Website:
http://www.python.org/

Introductory Material on Python:
http://www.python.org/doc/Intros.html

Python Tutorial:
http://www.python.org/doc/current/tut/tut.html

Python FAQ Wizard:
http://www.python.org/cgi-bin/faqw.py

[Apr 3, 2007] Charming Python Python elegance and warts, Part 1

Generators as not-quite-sequences

Over several versions, Python has hugely enhanced its "laziness." For several versions, we have had generators defined with the yield statement in a function body. But along the way we also got the itertools modules to combine and create various types of iterators. We have the iter() built-in function to turn many sequence-like objects into iterators. With Python 2.4, we got generator expressions, and with 2.5 we will get enhanced generators that make writing coroutines easier. Moreover, more and more Python objects have become iterators or iterator-like; for example, what used to require the .xreadlines() method or before that the xreadlines module, is now simply the default behavior of open() to read files.

Similarly, looping through a dict lazily used to require the .iterkeys() method; now it is just the default for key in dct behavior. Functions like xrange() are a bit "special" in being generator-like, but neither quite a real iterator (no .next() method), nor a realized list like range() returns. However, enumerate() returns a true generator, and usually does what you had earlier wanted xrange() for. And itertools.count() is another lazy call that does almost the same thing as xrange(), but as a full-fledged iterator.

Python is strongly moving towards lazily constructing sequence-like objects; and overall this is an excellent direction. Lazy pseudo-sequences both save memory space and speed up operations (especially when dealing with very large sequence-like "things").

The problem is that Python still has a schizoaffective condition when it comes to deciding what the differences and similarities between "hard" sequences and iterators are. The troublesome part of this is that it really violates Python's idea of "duck typing": the ability to use a given object for a purpose just as long as it has the right behaviors, but not necessarily any inheritance or type restriction. The various things that are iterators or iterator-like sometimes act sequence-like, but other times do not; conversely, sequences often act iterator-like, but not always. Outside of those steeped in Python arcana, what does what is not obvious.

Divergences

The main point of similarity is that everything that is sequence- or iterator-like lets you loop over it, whether using a for loop, a list comprehension, or a generator comprehension. Past that, divergences occur. The most important of these differences is that sequences can be indexed, and directly sliced, while iterators cannot. In fact, indexing into a sequence is probably the most common thing you ever do with a sequence -- why on earth does it fall down so badly on iterators? For example:

Listing 9. Sequence-like and iterator-like things
 

>>> r = range(10) >>> i = iter(r) >>> x = xrange(10) >>> g = itertools.takewhile(lambda n: n<10, itertools.count()) #...etc...

 

For all of these, you can use for n in thing. In fact, if you "concretize" any of them with list(thing), you wind up with exactly the same result. But if you wish to obtain a specific item -- or a slice of a few items -- you need to start caring about the exact type of thing. For example:

Listing 10. When indexing succeeds and fails
 

>>> r[4] 4 >>> i[4] TypeError: unindexable object

 

With enough contortions, you can get an item for every type of sequence/iterator. One way is to loop until you get there. Another hackish combination might be something like:

Listing 11. Contortions to obtain an index
 

>>> thing, temp = itertools.tee(thing) >>> zip(temp, '.'*5)[-1][0] 4

 

The pre-call to itertools.tee() preserves the original iterator. For a slice, you might use the itertools.islice() function, wrapped up in contortions.

Listing 12. Contortions to obtain a slice
 

>>> r[4:9:2] [4, 6, 8] >>> list(itertools.islice(r,4,9,2)) # works for iterators [4, 6, 8]

 

A class wrapper

You might combine these techniques into a class wrapper for convenience, using some magic methods:

Listing 13. Making iterators indexable
 

>>> class Indexable(object): ... def __init__(self, it): ... self.it = it ... def __getitem__(self, x): ... self.it, temp = itertools.tee(self.it) ... if type(x) is slice: ... return list(itertools.islice(self.it, x.start, x.stop, x.step)) ... else: ... return zip(temp, range(x+1))[-1][0] ... def __iter__(self): ... self.it, temp = itertools.tee(self.it) ... return temp ... >>> integers = Indexable(itertools.count()) >>> integers[4] 4 >>> integers[4:9:2] [4, 6, 8]

 

Share this...     Digg this story Post to del.icio.us Slashdot it!

So with some effort, you can coax an object to behave like both a sequence and an iterator. But this much effort should really not be necessary; indexing and slicing should "just work" whether a concrete sequence or a iterator is involved.

Notice that the Indexable class wrapper is still not as flexible as might be desirable. The main problem is that we create a new copy of the iterator every time. A better approach would be to cache the head of the sequence when we slice it, then use that cached head for future access of elements already examined. Of course, there is a trade-off between memory used and the speed penalty of running through the iterator. Nonetheless, the best thing would be if Python itself would do all of this "behind the scenes" -- the behavior might be fine-tuned somehow by "power users," but average programmers should not have to think about any of this.

In the next installment in this series, I'll discuss accessing methods using attribute syntax.

 

[Oct 26, 2006] ONLamp.com -- What's New in Python 2.5

It's clear that Python is under pressure from Ruby :-)

It's hard to believe Python is more than 15 years old already. While that may seem old for a programming language, in the case of Python it means the language is mature. In spite of its age, the newest versions of Python are powerful, providing everything you would expect from a modern programming language.

This article provides a rundown of the new and important features of Python 2.5. I assume that you're familiar with Python and aren't looking for an introductory tutorial, although in some cases I do introduce some of the material, such as generators.

[Sep 30, 2006] Python 2.5 Release We are pleased to announce the release of Python 2.5 (FINAL), the final, production release of Python 2.5, on September 19th, 2006.

• conditional expressions look pretty ugly.
• Some standard Python objects now support Ruby-style syntax using the 'with' statement. File objects are one example:

with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...

cout << ( a==b ? "first option" : "second option" )

[Sept 20, 2006] Python 101 cheat sheet

[01 Feb 2000] Python columnist Evelyn Mitchell brings you a quick reference and learning tools for newbies who want to get to know the language. Print it, keep it close at hand, and get down to programming!

[Jul 27, 2006] eWeek/Microsoft Ships Python on .Net  Darryl K. Taft

Microsoft has shipped the release candidate for IronPython 1.0 on its CodePlex community source site.

In a July 25 blog post, S. "Soma" Somasegar, corporate vice president of Microsoft's developer division, praised the team for getting to a release candidate for a dynamic language that runs on the Microsoft CLI (Common Language Infrastructure). Microsoft designed the CLI to support a variety of programming languages. Indeed, "one of the great features of the .Net framework is the Common Language Infrastructure," Somasegar said.

"IronPython is a project that implements the dynamic object-oriented Python language on top of the CLI," Somasegar said. IronPython is both well-integrated with the .Net Framework and is a true implementation of the Python language, he said.

And ".Net integration means that this rich programming framework is available to Python developers and that they can interoperate with other .Net languages and tools," Somasegar said. "All of Python's dynamic features like an interactive interpreter, dynamically modifying objects and even metaclasses are available. IronPython also leverages the CLI to achieve good performance, running up to 1.5 times faster than the standard C-based Python implementation on the standard Pystone benchmark."

Click here to read an eWEEK interview with Python creator Guido van Rossum.

Moreover, the download of the release candidate for IronPython 1.0 "includes a tutorial which gives .Net programmers a great way to get started with Python and Python programmers a great way to get started with .Net," Somasegar said.

Somasegar said he finds it "exciting to see that the Visual Studio SDK [software development kit] team has used the IronPython project as a chance to show language developers how they can build support for their language into Visual Studio. They have created a sample, with source, that shows some of the basics required for integrating into the IDE including the project system, debugger, interactive console, IntelliSense and even the Windows forms designer. "

IronPython is the creation of Jim Hugunin, a developer on the Microsoft CLR (Common Language Runtime) team. Hugunin joined Microsoft in 2004.

In a statement written in July 2004, Hugunin said: "My plan was to do a little work and then write a short pithy article called, 'Why .Net is a terrible platform for dynamic languages.' My plans changed when I found the CLR to be an excellent target for the highly dynamic Python language. Since then I've spent much of my spare time working on the development of IronPython."

However, Hugunin said he grew frustrated with the slow pace of progress he could make by working on the project only in his spare time, so he decided to join Microsoft.

IronPython is governed by Microsoft's Shared Source license.

Dig Deep into Python Internals by Gigi Sayfan Part 1 of 2

Python, the open source scripting language, has grown tremendously popular in the last five years—and with good reason. Python boasts a sophisticated object model that wise developers can exploit in ways that Java, C++, and C# developers can only dream of.  

This article is the first in a two-part series that will dig deep to explore the fascinating new-style Python object model, which was introduced in Python 2.2 and improved in 2.3 and 2.4. The object model and type system are very dynamic and allow quite a few interesting tricks. In this article I will describe the object, model, and type system; explore various entities; explain the life cycle of an object; and introduce some of the countless ways to modify and customize almost everything you thought immutable at runtime.

The Python Object Model
Python's objects are basically a bunch of attributes. These attributes include the type of the object, fields, methods, and base classes. Attributes are also objects, accessible through their containing objects.

The built-in dir() function is your best friend when it comes to exploring python objects. It is designed for interactive use and, thereby, returns a list of attributes that the implementers of the dir function thought would be relevant for interactive exploration. This output, however, is just a subset of all the attributes of the object. The code sample below shows the dir function in action. It turns out that the integer 5 has many attributes that seem like mathematical operations on integers.

dir(5)

['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', '__delattr__', '__div__', 
'__divmod__', '__doc__', '__float__', '__floordiv__', '__getattribute__', '__getnewargs__', 
'__hash__', '__hex__', '__init__', '__int__', '__invert__', '__long__', '__lshift__', '__mod__',
'__mul__', '__neg__', '__new__', '__nonzero__', '__oct__', '__or__', '__pos__', '__pow__', '
__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__',
'__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__',
'__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__str__', '__sub__', '__truediv__', '__xor__']

The function foo has many attributes too. The most important one is __call__ which means it is a callable type. You do want to call your functions, don't you?

def foo()
      pass
...
dir(foo)

['__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__get__', '__getattribute__',
'__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 
'func_doc', 'func_globals', 'func_name']

Next I'll define a class called 'A' with two methods, __init__ and dump, and an instance field 'x' and also an instance 'a' of this class. The dir function shows that the class's attributes include the methods and the instance has all the class attributes as well as the instance field.

>>> class A(object):
...     def __init__(self):
...             self.x = 3
...     def dump(self):
...             print self.x
...
>>> dir(A)

['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', 
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 
'__weakref__', 'dump']

>>> a = A()
>>> dir(a)

['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__',
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__',
'__weakref__', 'dump', 'x']

The Python Type System
Python has many types. Much more than you find in most languages (at least explicitly). This means that the interpreter has a lot of information at runtime and the programmer can take advantage of it by manipulating types at runtime. Most types are defined in the types module, which is shown in the code immediately below. Types come in various flavors: There are built-in types, new-style classes (derived from object), and old-style classes (pre Python 2.2). I will not discuss old-style classes since they are frowned upon by everybody and exist only for backward compatibility.

>>> import types
>>> dir(types)

['BooleanType', 'BufferType', 'BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType',
'ComplexType', 'DictProxyType', 'DictType', 'DictionaryType', 'EllipsisType', 'FileType', 
'FloatType', 'FrameType', 'FunctionType', 'GeneratorType', 'InstanceType', 'IntType', 'LambdaType',
'ListType', 'LongType', 'MethodType', 'ModuleType', 'NoneType', 'NotImplementedType', 'ObjectType',
'SliceType', 'StringType', 'StringTypes', 'TracebackType', 'TupleType', 'TypeType', 
'UnboundMethodType', 'UnicodeType', 'XRangeType', '__builtins__', '__doc__', '__file__', '__name__']

Python's type system is object-oriented. Every type (including built-in types) is derived (directly or indirectly) from object. Another interesting fact is that types, classes and functions are all first-class citizens and have a type themselves. Before I delve down into some juicy demonstrations let me introduce the built-in function 'type'. This function returns the type of any object (and also serves as a type factory). Most of these types are listed in the types module, and some of them have a short name. Below I've unleashed the 'type' function on several objects: None, integer, list, the object type, type itself, and even the 'types' module. As you can see the type of all types (list type, object, and type itself) is 'type' or in its full name types.TypeType (no kidding, that's the name of the type).

>>> type(None)
<type 'NoneType'>

>>> type(5)
<type 'int'>

>>> x = [1,2,3]
>>> type(x)
<type 'list'>

>>> type(list)
<type 'type'> 

>>> type(type)
<type 'type'>

>>> type(object)
<type 'type'>
>>>

>>> import types
>>> type(types)
<type 'module'>

>>> type==types.TypeType
True

What is the type of classes and instances? Well, classes are types of course, so their type is always 'type' (regardless of inheritance). The type of class instances is their class.

>>> class A(object):
...     pass


>>> a = A()

>>> type(A)
<type 'type'>

>>> type(a)
<class '__main__.A'>

>>> a.__class__
<class '__main__.A'>

It's time for the scary part—a vicious cycle: 'type' is the type of object, but object is the base class of type. Come again? 'type' is the type of object, but object is the base class of type. That's right—circular dependency. 'object' is a 'type' and 'type' is an 'object'.

>>> type(object)
<type 'type'>

>>> type.__bases__
(<type 'object'>,)

>>> object.__bases__
()

How can it be? Well, since the core entities in Python are not implemented themselves in Python (there is PyPy but that's another story) this is not really an issue. The 'object' and 'type' are not really implemented in terms of each other.

The one important thing to take home from this is that types are objects and are therefore subject to all the ramifications thereof. I'll discuss those ramifications very shortly.

Instances, Classes, Class Factories, and Metaclasses
When I talk about instances I mean object instances of a class derived from object (or the object class itself). A class is a type, but as you recall it is also an object (of type 'type'). This allows classes to be created and manipulated at runtime. This code demonstrates how to create a class at runtime and instantiate it.

def init_method(self, x, y):
      self.x = x
      self.y = y
def dumpSum_method(self):
      print self.x + self.y

D = type('DynamicClass',
                   (object,),
                   {'__init__':init_method, 'dumpSum':dumpSum_method})
d = D(3, 4)
d.dumpSum()

As you can see I created two functions (init_method and dumpSum_method) and then invoked the ubiquitous 'type' function as a class factory to create a class called 'DynamicClass,' which is derived from 'object' and has two methods (one is the __init__ constructor).

It is pretty simple to create the functions themselves on the fly too. Note that the methods I attached to the class are regular functions that can be called directly (provided their self-argument has x and y members, similar to C++ template arguments).

Functions, Methods and other Callables
Python enjoys a plethora of callable objects. Callable objects are function-like objects that can be invoked by calling their () operator. Callable objects include plain functions (module-level), methods (bound, unbound, static, and class methods) and any other object that has a __call__ function attribute (either in its own dictionary, via one of its ancestors, or through a descriptor).

It's truly complicated so the bottom line is to remember that all these flavors of callables eventually boil down to a plain function. For example, in the code below the class A defines a method named 'foo' that can be accessed through:

1. an instance so it is a bound method (bound implicitly to its instance)
2. through the class A itself and then it is an unbound method (the instance must be supplied explicitly)
3. directly from A's dictionary, in which case it is a plain function (but you must still call it with an instance of A).

So, all methods are actually functions but the runtime assigns different types depending on how you access it.

class A(object):
    def foo(self):
        print 'I am foo'

>>> a = A()
>>> a.foo
<bound method A.foo of <__main__.A object at 0x00A13EB0>>

>>> A.foo
<unbound method A.foo>

>>> A.__dict__['foo']
<function foo at 0x00A0A3F0>
>>> a.foo

>>> a.foo()
I am foo
>>> A.foo(a)
I am foo
>>> A.__dict__['foo'](a)
I am foo

Let's talk about static methods and class methods. Static methods are very simple. They are similar to static methods in Java/C++/C#. They are scoped by their class but they don't have a special first argument like instance methods or class methods do; they act just like a regular function (you must provide all the arguments since they can't access any instance fields). Static methods are not so useful in Python because regular module-level functions are already scoped by their module and they are the natural mapping to static methods in Java/C++/C#.

Class methods are an exotic animal. Their first argument is the class itself (traditionally named cls) and they are used primarily in esoteric scenarios. Static and class methods actually return a wrapper around the original function object. In the code that follows, note that the static method may be accessed either through an instance or through a class. The class method accepts a cls instance as its first argument but cls is invoked through a class directly (no explicit class argument). This is different from an unbound method where you have to provide an instance explicitly as first argument.

class A(object):
    def foo():
        print 'I am foo'
    def foo2(cls):
        print 'I am foo2', cls 
    def foo3(self):
        print 'I am foo3', self       
    foo=staticmethod(foo)
    foo2=classmethod(foo2)
        
>>> a = A()
>>> a.foo()
I am foo

>>> A.foo()
I am foo

>>> A.foo2()
I am foo2 <class '__main__.A'>

>>> a.foo3()
I am foo3 <__main__.A object at 0x00A1AA10>

Note that classes are callable objects by themselves and operate as instance factories. When you "call" a class you get an instance of that class as a result.

A different kind of callable object is an object that has a __call__ method. If you want to pass around a function-like object with its context intact, __call__ can be a good thing. Listing 1 features a simple 'add' function that can be replaced with a caching adder class that stores results of previous calculations. First, notice that the test function expects a function-like object called 'add' and it just invokes it as a function. The 'test' function is called twice—once with a simple function and a second time with the caching adder instance. Continuations in Python can also be implemented using __call__ but that's another article.

Metaclasses
Metaclasse is a concept that doesn't exist in today's mainstream programming languages. A metaclass is a class whose instances are classes. You already encountered a meta-class in this article called 'type'. When you invoke "type" with a class name, a base-classes tuple, and an attribute dictionary, the method creates a new user-defined class of the specified type. So the __class__ attribute of every class always contains its meta-class (normally 'type').

That's nice, but what can you do with a metaclass? It turns out, you can do plenty. Metaclasses allow you to control everything about the class that will be created: name, base classes, methods, and fields. How is it different from simply defining any class you want or even creating a class dynamically on the fly? Well, it allows you to intercept the creation of classes that are predefined as in aspect-oriented programming. This is a killer feature that I'll be discussing in a follow-up to this article.

After a class is defined, the interpreter looks for a meta-class. If it finds one it invokes its __init__ method with the class instance and the meta-class gets a stab at modifying it (or returning a completely different class). The interpreter will use the class object returned from the meta-class to create instances of this class.

So, how do you stick a custom metaclass on a class (new-style classes only)? Either you declare a __metaclass__ field or one of your ancestors has a __metaclass__ field. The inheritance method is intriguing because Python allows multiple inheritance. If you inherit from two classes that have custom metaclasses you are in for a treat—one of the metaclasses must derive from another. The actual metaclass of your class will be the most derived metaclass:

class M1(type): pass
class M2(M1):   pass

class C2(object): __metaclass__=M2    
class C1(object): __metaclass__=M1
class C3(C1, C2): pass


classes = [C1, C2, C3]
for c in classes:
    print c, c.__class__
    print '------------'                 

Output:
    
<class '__main__.C1'> <class '__main__.M1'>
------------
<class '__main__.C2'> <class '__main__.M2'>
------------
<class '__main__.C3'> <class '__main__.M2'>

Day In The Life of a Python Object
To get a feel for all the dynamics involved in using Python objects let's track a plain object (no tricks) starting from its class definition, through its class instantiation, access its attributes, and see it to its demise. Later on I'll introduce the hooks that allow you to control and modify this workflow.

The best way to go about it is with a monstrous simulation. Listing 2 contains a simulation of a bunch of monsters chasing and eating some poor person. There are three classes involved: a base Monster class, a MurderousHorror class that inherits from the Monster base class, and a Person class that gets to be the victim. I will concentrate on the MurderousHorror class and its instances.

Class Definition
MurderousHorror inherits the 'frighten' and 'eat' methods from Monster and adds a 'chase' method and a 'speed' field. The 'hungry_monsters' class field stores a list of all the hungry monsters and is always available through the class, base class, or instance (Monster.hungry_monsters, MurderousHorror.hungry_monsters, or m1.hungry_monsters). In the code below you can see (via the handy 'dir' function) the MurderousHorror class and its m1 instance. Note that methods such as 'eat,' 'frighten,' and 'chase' appear in both, but instance fields such as 'hungry' and 'speed' appear only in m1. The reason is that instance methods can be accessed through the class as unbound methods, but instance fields can be accessed only through an instance.

class NoInit(object):
    def foo(self):
        self.x = 5
        
    def bar(self):
        print self.x
                            
if __name__ == '__main__':            
    ni = NoInit()
    assert(not ni.__dict__.has_key('x'))
    try:
        ni.bar()
    except AttributeError, e:
        print e
    ni.foo()
    assert(ni.__dict__.has_key('x'))
    ni.bar()
    
Output:

'NoInit' object has no attribute 'x'
5

Object Instantiation and Initialization
Instantiation in Python is a two-phase process. First, __new__ is called with the class as a first argument, and later as the rest of the arguments, and should return an uninitialized instance of the class. Afterward, __init__ is called with the instance as first argument. (You can read more about __new__ in the Python reference manual.)

When a MurderousHorror is instantiated __init__ is the first method called. __init__ is similar to a constructor in C++/Java/C#. The instance calls the Monster base class's __init__ and initializes its speed field. The difference between Python and C++/Java/C# is that in Python there is no notion of a parameter-less default constructor, which, in other languages, is automatically generated for every class that doesn't have one. Also, there is no automatic call to the base class' default __init__ if the derived class doesn't call it explicitly. This is quite understandable since no default __init__ is generated.

In C++/Java/C# you declare instance variables in the class body. In Python you define them inside a method by explicitly specifying 'self.SomeAttribute'. So, if there is no __init__ method to a class it means its instances have no instance fields initially. That's right. It doesn't HAVE any instance fields. Not even uninitialized instance fields.

The previous code sample (above) is a perfect example of this phenomenon. The NoInit class has no __init__ method. The x field is created (put into its __dict__) only when foo() is called. When the program calls ni.bar() immediately after instantiation the 'x' attribute is not there yet, so I get an 'AttributeError' exception. Because my code is robust, fault tolerant, and self healing (in carefully staged toy programs), it bravely recovers and continues to the horizon by calling foo(), thus creating the 'x' attribute, and ni.bar() can print 5 successfully.

Note that in Python __init__ is not much more then a regular method. It is called indeed on instantiation, but you are free to call it again after initialization and you may call other __init__ methods on the same object from the original __init__. This last capability is also available in C#, where it is called constructor chaining. It is useful when you have multiple constructors that share common initialization, which is also one of the constructors/initializers. In this case you don't need to define another special method that contains the common code and call it from all the constructors/initializers; you can just call the shared constructor/initializer directly from all of them.

Attribute Access
An attribute is an object that can be accessed from its host using the dot notation. There is no difference at the attribute access level between methods and fields. Methods are first-class citizens in Python. When you invoke a method of an object, the method object is looked up first using the same mechanism as a non-callable field. Then the () operator is applied to the returned object. This example demonstrates this two-step process:

class A(object):
    def foo(self):
        print 3
                            
if __name__ == '__main__':            
    a = A()
    f = a.foo
    print f
    print f.im_self
    a.foo()
    f()
    
    
Output:

<bound method A.foo of <__main__.A object at 0x00A03EB0>>
<__main__.A object at 0x00A03EB0>
3
3

The code retrieves the a.foo bound method object and assigns it to a local variable 'f'. 'f' is a bound method object, which means its im_self attribute points to the instance to which it is bound. Finally, a.foo is invoked through the instance (a.foo()) and by calling f directly with identical results. Assigning bound methods to local variables is a well known optimization technique due to the high cost of attribute lookup. If you have a piece of Python code that seems to perform under the weather there is a good chance you can find a tight loop that does a lot of redundant lookups. I will talk later about all the ways you can customize the attribute access process and why it is so costly.

Destruction
The __del__ method is called when an instance is about to be destroyed (its reference count reaches 0). It is not guaranteed that the method will ever be called in situations such as circular references between objects or references to the object in an exception. Also the implementation of __del__ may create a new reference to its instance so it will not be destroyed after all. Even when everything is simple and __del__ is called, there is no telling when it will actually be called due to the nature of the garbage collector. The bottom line is if you need to free some scarce resource attached to an object do it explicitly when you are done using it and don't wait for __del__.

A try-finally block is a popular choice for garbage collection since it guarantees the resource will be released even in the face of exceptions. The last reason not to use is __del__ is that its interaction with the 'del' built-in function may confuse programmers. 'del' simply decrements the reference count by 1 and doesn't call '__del__' or cause the object to be magically destroyed. In the next code sample I use the sys.getrefcount() function to determine the reference count to an object before and after calling 'del'. Note that I subtract 1 from sys.getrefcount() result because it also counts the temporary reference to its own argument.

import sys

class A(object):
    def __del__(self):
        print "That's it for me"
                            
if __name__ == '__main__':            
    a = A()
    b = a
    print sys.getrefcount(a)-1
    del b
    print sys.getrefcount(a)-1

Output:

2
1
That's it for me

Hacking Python

Let the games begin. In this section I will explore different ways to customize attribute access. The topics include the __getattribute__ hook, descriptors, and properties.

- __getattr__, __setattr__ and __getattribute__

These special methods control attribute access to class instances. The standard algorithm for attribute lookup returns an attribute from the instance dictionary or one of its base class's dictionaries (descriptors will be described in the next section). They are supposed to return an attribute object or raise AttributeError exception. If you define some of these methods in your class they will be called upon during attribute access under some conditions.

• __getattr__ and __setattr__ work with old-style and new-style classes.
• __getattr__ is called to get attributes that cannot be found using the standard attribute lookup algorithm.
• __setattr__ is called for setting the value of any attribute. This asymmetry is necessary to allow adding new attributes to instances.
• __getattribute__ works for new-style classes only. It is called to get any attribute (existing or non-existing). __getattribute__ has precedence over __getattr__, so if you define both, __getattr__ will not be called (unless __getattribute__ raises AttributeError exception).

Listing 3 is an interactive example. It is designed to allow you to play around with it and comment out various functions to see the effect. It introduces the class A with a single 'x' attribute. It has __getattr__, __setattr__, and __getattribute__ methods. __getattribute__ and __setattr__ simply forward any attribute access to the default (lookup or set value in dictionary). __getattr__ always returns 7. The main program starts by assigning 6 to the non-existing attribute 'y' (happens via __setattr__) and then prints the preexisting 'x', the newly created 'y', and the still non-existent 'z'. 'x' and 'y' exist now, so they are accessible via __getattribute__. 'z' doesn't exist so __getattribute__ fails and __getattr__ gets called and returns 7. (Author's Note: This is contrary to the documentation. The documentation claims if __getattribute__ is defined, __getattr__ will never be called, but this is not the actual behavior.)

Descriptors
A descriptor is an object that implements three methods __get__, __set__, and __delete__. If you put such a descriptor in the __dict__ of some object then whenever the attribute with the name of the descriptor is accessed one of the special methods is executed according to the access type (__get__ for read, __set__ for write, and __delete__ for delete).This simple enough indirection scheme allows total control on attribute access.

The following code sample shows a silly write-only descriptor used to store passwords. Its value may not be read nor deleted (it throws AttributeError exception). Of course the descriptor object itself and the password can be accessed directly through A.__dict__['password'].

class WriteOnlyDescriptor(object):
    def __init__(self):
        self.store = {}

    def __get__(self, obj, objtype=None):
        raise AttributeError 

    def __set__(self, obj, val):
        self.store[obj] = val
    
    def __del(self, obj):
        raise AttributeError

class A(object):
    password = WriteOnlyDescriptor()
      
if __name__ == '__main__': 
    a = A()
    try:
        print a.password
    except AttributeError, e:
        print e.__doc__
    a.password = 'secret'
    print A.__dict__['password'].store[a]

Descriptors with both __get__ and __set__ methods are called data descriptors. In general, data descriptors take lookup precedence over instance dictionaries, which take precedence over non-data descriptors. If you try to assign a value to a non-data descriptor attribute the new value will simply replace the descriptor. However, if you try to assign a value to a data descriptor the __set__ method of the descriptor will be called.

Properties
Properties are managed attributes. When you define a property you can provide get, set, and del functions as well as a doc string. When the attribute is accessed the corresponding functions are called. This sounds a lot like descriptors and indeed it is mostly a syntactic sugar for a common case.

This final code sample is another version of the silly password store using properties. The __password field is "private." Class A has a 'password' property that, when accessed as in 'a.password,' invokes the getPassword or setPassword methods. Because the getPassword method raises the AttributeError exception, the only way to get to the actual value of the __password attribute is by circumventing the Python fake privacy mechanism. This is done by prefixing the attribute name with an underscore and the class name a._A__password. How is it different from descriptors? It is less powerful and flexible but more pleasing to the eye. You must define an external descriptor class with descriptors. This means you can use the same descriptor for different classes and also that you can replace regular attributes with descriptors at runtime.

class A(object):
    def __init__(self):
        self.__password = None

    def getPassword(self):
        raise AttributeError

    def setPassword(self, password):        
        self.__password = password

    password = property(getPassword, setPassword)    
      
if __name__ == '__main__':
    a = A()
    try:
        print a.password
    except AttributeError, e:
        print e.__doc__
    a.password = 'secret'
    print a._A__password
    
Output:
    
Attribute not found.
secret 

Properties are more cohesive. The get, set functions are usually methods of the same class that contain the property definition. For programmers coming from languages such as C# or Delphi, Properties will make them feel right at home (too bad Java is still sticking to its verbose java beans).

Python's Richness a Mixed Blessing
There are many mechanisms to control attribute access at runtime starting with just dynamic replacement of attribute in the __dict__ at runtime. Other methods include the __getattr__/__setattr, descriptors, and finally properties. This richness is a mixed blessing. It gives you a lot of choice, which is good because you can choose whatever is appropriate to your case. But, it is also bad because you HAVE to choose even if you just choose to ignore it. The assumption, for better or worse, is that people who work at this level should be able to handle the mental load.

In my next article, I will pick up where I've left off. I'll begin by contrasting metaclasses with decorators, then explore the Python execution model, and explain how to examine stack frames at runtime. Finally, I'll demonstrate how to augment the Python language itself using these techniques. I'll introduce a private access checking feature that can be enforced at runtime.

Gigi Sayfan is a software developer working on CELL applications for Sony Playstation3. He specializes in cross-platform object-oriented programming in C/C++/C#/Python with emphasis on large-scale distributed systems.

[Feb 20, 2006] freshmeat.net Project details for Meld

Meld is a GNOME 2 visual diff and merge tool. It integrates especially well with CVS. The diff viewer lets you edit files in place (diffs update dynamically), and a middle column shows detailed changes and allows merges. The margins show location of changes for easy browsing, and it also features a tabbed interface that allows you to open many diffs at once.

Information about CWM - TimBL's Closed World Machine

CWM is a popular Semantic Web program that can do the following tasks:-

• Parse and pretty-print the following RDF formats: XML RDF, Notation3, and NTriples
• Store triples in a queryable triples database
• Perform inferences as a forward chaining FOPL inference engine
• Perform builtin functions such as comparing strings, retrieving resources, all using an extensible builtins suite

CWM was written in Python from 2000-10 onwards by Tim Berners-Lee and Dan Connolly of the W3C.

This resource is provided so that people can use CWM, find out what it does (documentation used to be sparse), and perhaps even contribute to its development.

What's new in Python 2.4

Dive Into Python Python for experienced programmers

Dive Into Python is a free Python book for experienced programmers. You can read the book online, or download it in a variety of formats. It is also available in multiple languages.

This book is still being written. The first three chapters are a solid overview of Python programming. Chapters covering HTML processing, XML processing, and unit testing are complete, and a chapter covering regression testing is in progress. This is not a teaser site for some larger work for sale; all new content will be published here, for free, as soon as it’s ready. You can read the revision history to see what’s new. Updated 28 July 2002

Wing IDE for Python Python IDS that includes source browser and editor.  The editor supports folding

• Wing IDE is a powerful development environment for the Python programming language.
• Designed to maximize programmer productivity, Wing IDE and Python can reduce development and maintenance costs by 50 to 90 percent of that seen with languages such as C, C++, Java, VB, and Perl.
• Whether you are building dynamic web sites, desktop applications, or complex enterprise solutions, Wing IDE and Python provide you with a fast, scalable, and portable development platform that lets you concentrate on building application-specific functionality.
• Wing and Python are easy to learn and integrate well with other tools and your existing non-Python code base.
• Key features of Wing IDE include:
  •  Networked graphical debugger, supports both stand-alone application development and remote debugging of externally launched code (like web CGIs and servlets).
  •  Source code analyzer and browser, offers high-level graphical inspection of source code structure, making it easier to understand, redesign, and maintain code.
  •  Powerful source editor, provides syntax highlighting for many languages, auto-completion, auto-indent, indentation analysis and conversion, keyboard macros, structural code folding, and customization of key bindings. An optional emacs personality module is included.
  •  Project manager, organizes your code and speeds access to your files.
  • Wing IDE comes in two product levels: Wing IDE Standard and Wing IDE Lite, a scaled down version available for non-commercial use only. For a detailed listing of features found in each product level, see the product feature list.

Reference Manual Wing IDE Version 1.1.4 Pythod IDS that includes source broser and editor.

developerWorks Linux Open source projects Charming Python Iterators and simple generators

What's New in Python 2.2 -- generators is a very interesting feature of Python 2.2 that is essentially a co-routine.

Generators are another new feature, one that interacts with the introduction of iterators.

You're doubtless familiar with how function calls work in Python or C. When you call a function, it gets a private namespace where its local variables are created. When the function reaches a return statement, the local variables are destroyed and the resulting value is returned to the caller. A later call to the same function will get a fresh new set of local variables. But, what if the local variables weren't thrown away on exiting a function? What if you could later resume the function where it left off? This is what generators provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function:

def generate_ints(N):
    for i in range(N):
        yield i

A new keyword, yield, was introduced for generators. Any function containing a yield statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result. Because a new keyword was introduced, generators must be explicitly enabled in a module by including a from __future__ import generators statement near the top of the module's source code. In Python 2.3 this statement will become unnecessary.

When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the yield statement, the generator outputs the value of i, similar to a return statement. The big difference between yield and a return statement is that on reaching a yield the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's .next() method, the function will resume executing immediately after the yield statement. (For complicated reasons, the yield statement isn't allowed inside the try block of a try...finally statement; read PEP 255 for a full explanation of the interaction between yield and exceptions.)

Here's a sample usage of the generate_ints generator:

>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in generate_ints
StopIteration

You could equally write for i in generate_ints(5), or a,b,c = generate_ints(3).

Inside a generator function, the return statement can only be used without a value, and signals the end of the procession of values; afterwards the generator cannot return any further values. return with a value, such as return 5, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function.

You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For example, returning a list of integers could be done by setting self.count to 0, and having the next() method increment self.count and return it. However, for a moderately complicated generator, writing a corresponding class would be much messier. Lib/test/test_generators.py contains a number of more interesting examples. The simplest one implements an in-order traversal of a tree using generators recursively.

# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x

Two other examples in Lib/test/test_generators.py produce solutions for the N-Queens problem (placing queens on an chess board so that no queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon (http://www.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from ``An Overview of the Icon Programming Language'' at http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:

sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)

In Icon the find() function returns the indexes at which the substring ``or'' is found: 3, 23, 33. In the if statement, i is first assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon retries it with the second value of 23. 23 is greater than 5, so the comparison now succeeds, and the code prints the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a central concept. Generators are considered a new part of the core Python language, but learning or using them isn't compulsory; if they don't solve any problems that you have, feel free to ignore them. One novel feature of Python's interface as compared to Icon's is that a generator's state is represented as a concrete object (the iterator) that can be passed around to other functions or stored in a data structure.

See Also:

PEP 255, Simple Generators

Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.

Dive Into Python Dive Into Python is a free Python book for experienced programmers.

 You can read the book online, or download it in a variety of formats. It is also available in multiple languages.

Seventh International Python Conference Papers

Applications I: The Internet

• Implementing the SMS server, or why I switched from Tcl to Python
• Mailman – An Extensible Mailing List Manager Using Python
• A Python Based Production System for High Volume Electronic Publishing

Optimizing Python

• A Peephole Optimizer for Python
• Converting Python Virtual Machine Code to C
• Virtual Method Tables in Python

Extending and Compiling

• A facility for creating Python extensions in C++
• Compiling Little Languages in Python
• PyFront: Conversion of Python to C Extension Modules
• Evolutionary Prototyping:  "Add Later" Static Types for Python

Applications II: Science and Simulation

• Beyond: A Portable Virtual World Simulation Framework
• Python as a Discrete Event Simulation environment
• Python Vuh: Mayan Calendrical Mathematics with Python

Slashdot: What Makes a Powerful Programming Language

[Dec 28, 2001] http://www.dotfunk.com/projects/pylint/pylint.0.1.tar.gz

 Pylint is a static type checker for Python (compare with  PyChecker 0.4) David Jeske and Scott Hassan proved that it is possible to do this. They are working on a type inference engine that understands the Python language and can detect type errors and violations. Adding type checking to python without
changing the language will ease the maintenance of a large python project with lots of developers. Please remember that eGroups before it was bought by Yahoo was a huge Python project (more than 180,000 lines of Python doing everything from a 100% dynamic website to all email delivery, pumping out 200 messages/second on a single 400 MHz Pentium!)

[Dec 20, 2001] Semi-coroutines in Python 2.2
 

Search Result 31 From: Steven Majewski (sdm7g@Virginia.EDU) Subject: Re: (semi) stackless python Newsgroups: comp.lang.python