With the application development cycles being measured in "web-years" (3-6 months) and the need/rush to "webify" applications a programmer can benefit from expanding his/her knowledge to a wide variety of languages and apply one which is ideal for the task at hand. When one goes beyond the publishing of static pages and into the realm of interactivity and publishing of information from databases , the need arises to have a tool at hand which will help one achieve the job quickly and efficiently. Python http://www.python.org is one such language.
There are various ways to interact with a RDBMS -
1. Use the vendor supplied API and be forced to program in C/C++
2. Use ODBC and be database independent and have the freedom to choose the language appropriate for the job such as a scripting language.
For Web based applications , scripting languages are the ideal tool for rapid development. Python is a <insert favorite buzzword - Object Oriented, byte code, portable, extensible, embeddable .... > scripting language.
In this article , I intend to show how we can develop a Web based, RDBMS interfacing, ODBC using application . We will develop a Library Management System which will allow users to search for and reserve books . We will also develop an administrative interface which will allow the library administrator to checkin and checkout books and add delete users . The complete code is available from ftp://ftp.python.org/contrib/lms-0.2.x.tar.gz. I developed this application on NT3.51/4.0 , IIS{1,2,3}.0, Access95 and using PythonWin (Python on NT with extra win32 specific modules ) http://www.python.org/ftp/python/pythonwin/index.html. Since I used ODBC , any database that has an ODBC driver can be used in place of Access - such as Postgres95 (http://www.postgres.org ) or Solid (http://www.solidtech.com/) or mysql (http://www.tcx.se/) on Linux OS or the usual Oracle, Sybase, Informix, MS SQL Server .
The ODBC module supplied with PythonWin actually includes two modules dbi and ODBC . The dbi module is used to interface with the database for binding to a database specific formate - for eg - DATE. The ODBC interface is similar to the database API specification http://www.python.org/ftp/www.python.org/sigs/db-sig/DatabaseAPI.html
that is being proposed by the Python DB-SIG http://www.python.org/ftp/www.python.org/sigs/db-sig/. Interfaces to popular databases such as Oracle, Informix which are compliant with the above spec are available from the Python ftp site ftp://ftp.python.org/contrib .
Having peaked your curiosity (I hope) by now or tested your patience ( I hope not) let us get down to actual code. Following is a sample code that opens a connection to a database and outputs all the rows in the table book .
import dbi
import odbc
try:
s = odbc.odbc('PostODBCTest/hiren/')
cur = s.cursor()
cur.execute('select * from book')
print cur.description
for tup in cur.description:
print tup[0],
print
while 1:
rec = cur.fetchone()
if not rec: break
print rec
except:
print 'error while processing ', sys.exc_type,sys.exc_value
s = None
cur = None
The first two lines import the two modules dbi and ODBC. Note: The order is important and necessary. The next line puts the rest of the code in a try/except block to catch any errors . In Python errors are "raised" as exceptions.The try/catch block in the above is not strictly necessary as any uncaught exceptions are propagated to the top level and the interpreter exits. You can use it for any exceptions that you generate from your code and perform any necessary cleanups or generate notifications or log a message. In the next line we are returned a connection object to which we should pass a string in the following format - 'DSN/Userid/Password' . If password is not needed then that field can be left blank as shown above. We use the connection object to get a "cursor" object ( a representation of a database cursor ) which can be used to execute sql and fetch rows from the database. You can get multiple cursor object from the connection object if you need to get data from two or more tables separately (as opposed to doing a join ) . A few of the interfaces of the cursor class are as shown below -
execute(sqlstmt) - argument is a sql statement which should be passed in as a string .
description() - returns a tuple of values which give the column names of the rows returned as a result of the sql statment executed.
fetchone() - returns one row of data
fetchmany(size) - returns the number of rows specified by size as a list or an empty list.
fetchall() - returns all the rows as a list or an empty list.
close() - closes the cursor.
The connection object supports cursor() and close() interfaces.
In the above sample code we have shown how easy it is to interface with a database. Python's dynamic typing comes in very handy as we do not have to declare the type of the data being received and its strong run time typing is useful as I will show below. So far we have just queried the database and output the results to the screen. We'll "web-enable" this by generating the output in a tabular format and in HTML. To webify our application we will use HTMLgen http://www.python.org/sigs/web-sig/ - an excellent Python Module written by Robin Friedrich which makes generating HTML a trivial matter. Here is the code -
import cgi
import sys
import dbi
import odbc
import glob
import string, regex, regsub, os
from HTMLgen import *
from HTMLcolors import *
print "Content-type: text/html" # HTML is following
print # blank line, end of headers
s = odbc.odbc('lms2/admin/')
cur = s.cursor()
cur.execute('select * from book')
doc = Document('HTML.rc') # The configuration file which generates the
# footers
doc.title = 'Test ODBC'
doc.subtitle = 'Tables'
aft,fore,top,home = None,None,None,None # Navigational bars.
doc.goprev,doc.gonext,doc.gotop,doc.gohome = aft,fore,top,home
doc.append(HR())
body = []
heading = []
for tup in cur.description:
heading.append(tup[0])
#fetch the data as we did before and append it to "body" . Since
# fetchmany returns a list , and we are appending that to the body
# - body is a list of lists which is what HTMLgen requires to generate
# table. In the following code recs is a list of tuples - a row of data
# is a tuple. We first change the tuple to a list and then append that
# list to another list .
while 1:
recs = cur.fetchmany(10)
for row in recs:
recL = []
for field in row:
recL.append(field)
body.append(recL)
if not recs: break
#print recs
#print body
t = Table( 'Table Books' ,
heading = heading,
body = body,
border = 4 ,
cell_align = 'right')
doc.append(Heading(3, 'test'))
doc.append(t)
doc.write()
The above code will generate a page which looks like -
HTMLgen is a Python module which provides support for generating HTML and it allows you to maintain the same look and feel for the pages generated. If you know HTML tags , it is easy to use HTMLgen as it uses classes which have names corresponding to an equivalent HTML tag. For eg: if you want to generate a line with a link in HTML you would say - <A HREF="http://localhost/xx.html"> Test link </a> . The equivalent in HTMLgen is
HREF("http://localhost/xx.html",Test link). The HTMLgen module comes with very good documentation and I'd advise you to refer to that . I don't think I'd do any better on that.
Now that we have interfaced with a database and even generated a HTML page , it is time for interfactions i.e CGI and forms and of course we will be using HTMLgen. Before we do that let us generate two base classes which will represent a table and a row in that table . We will provide interfaces on it to make it look like a sequence and then other classes can derive from it to be plug compatible with the HTMLgen module.
We will create a module (Database.py) which will contain the above two classes. class RTable represents a database table and RRec represents a row of a table. Dynamic typing is very useful here as it lets us put functionality into classes without having to declare beforehand the datatypes of the incoming data. The RTable class is pretty simple as we provide only a couple of interfaces and provide functions which will let an instance of the RTable emulate a sequence or a list i.e we will be able to use indexes to get at a row of data. Here is the code for RTable and RTable
import dbi
import odbc
import time
class RTable:
def __init__(self,dsn):
self.conn = odbc.odbc(dsn)
self.cur = self.conn.cursor()
def execute(sql):
self.sql = sql
self.cur.execute(self.sql)
def __len__(self):
pass
def __setitem__(self,key,val):
pass
def __getitem__(self,index):
rec = self.cur.fetchone()
if not rec:
raise IndexError, "index too large"
return rec
class RRec:
def __init__(self,rec,description):
self.record = []
if not rec: return
i = 0
for field in rec:
if description[i][1] == 'DATE':
if field != None and field != 'N/A':
local = time.localtime(field.value)
s = str(local[1]) + '/' \
+ str(local[2]) + '/' +str(local[0])
setattr(self,description[i][0],s)
self.record.append(s)
else:
self.record.append('N/A')
elif description[i][1] == 'RAW':
dummy = 'RAW Not Implemented'
setattr(self,description[i][0],dummy)
self.record.append( dummy)
else:
setattr(self,description[i][0],field)
self.record.append(field)
i = i+1
self.data = self.record
def dump(self):
for i in self:
print i,
print
def __len__(self):
return len(self.record)
def __setitem__(self,key,val):
self.record[key] = val
def __getitem__(self,index):
return self.record[index]
if __name__ == '__main__':
print 'script as main'
sqlstmt = '\
SELECT EmployeeID, LastName,FirstName \
FROM Employees'
dsn='nwin/admin/'
s = RTable(dsn,sqlstmt)
for rec in s:
b = RRec(rec,s.cur.description)
for field in b:
print field,
print
Of importance in the above code are the functions with leading and trailing underscores. These are special functions which lets us define the behaviour of an instance of the class. When you define __init__ you defining a 'constructor' for that class. __len__,__setitem__ and __getitem__ are necessary to make the instance of RTable look like a sequence . We will primarily be using the RTable as a read-only and do any updates to the database thro' the execute statement. __setitem__ at present is just a place-holder. Providing functionality on it to update the database and keep it synchronized is beyond the scope of this article.
In __getitem__ we intercept any indexing operations and call fetchone() on the cursor which we had created in the execute, if fetchone returns None we raise an IndexError which is consistent with accessing and out-of-bounds element in a sequence. It is in the definiton of RRec where we have to do some work. This is because we have to format some fields like DATE into a human-readable form. This is where Python's strong run-time type definiton is convenient.
The constructor of RRec is passed a row of data retrieved from a table along with its description. Since we have the description which tells us the type of the column or the field in the record , we loop through the row and when we detect a DATE format we format it to 'MM/DD/YY' . Note the call to setattr(self,description[i][0],s). What we are doing here is introducing an attribute of name description[i][0] which is the name of the field with its value into the instance of the class RRec. We are also appending the values into an internal list. What this buys us is the ability to access the data either as a sequence or with name. An example will better illustrate it -
If a row returned from the database as follows , i.e the name of the first column is 'fname', the name of the second column is 'lname' and the values are 'Hiren' and 'Hindocha' respectively, then an instance of RRec say record can be accessed thus -
print record.fname # this will print Hiren print record[1] # this will print 'Hindocha' , remember indexing starts at 0.
| fname | lname |
| 'Hiren' | 'Hindocha' |
One of the nice things about Python is the ability to be able to test a module as a standalone application and yet be able to include it another application without any code modification. In the code above , the line 'if __name__ == '__main__' is true if you invoke the module as a standalone script and is false when it is imported into another module or app. This proves to be extremely useful as the designer of a class can show how to use the class and it can also act as a testing tool.
Now that the base classes are defined , we will develop classes that represent the 'book' table and a 'book' record. The book table will provide interfaces that will let us query the database for a particular title , reserve books and all other necessary interfaces. We will concentrate on only the querying part and as such only that part is show below. The filename is LMS.py
import HTMLgen
import HTMLcolors
import Database
import Dates
class BookRep(Database.RTable):
def __init__(self):
dsn='lms2/admin/'
self.rows = []
Database.RTable.__init__(self,dsn)
def books_checked_out(self):
pass
def __len__(self):
return len(self.rows)
def __setitem__(self,index,val):
self.rows[index] = val
def __getitem__(self,index):
if index > len(self.rows):
raise IndexError, "index too large"
return self.rows[index]
def checkin(self,bookids,dates):
pass
def checkout(self,bookids,empids,dates):
pass
def find(self,titlepattern='UNIX',author='oreilly'):
self.rows = []
sqlstr = "\
select a.bookid,a.title,b.checkOutDate,b.dueDate,b.returnDate \n\
FROM { oj book a left outer join checkout b\n\
on a.bookid = b.bookid\n}\
where a.title like '%s' \
and ( ( b.checkOutDate >= ( select max(c.checkOutDate) \
from checkout c \
where c.bookid = b.bookid) ) \
or \
( b.checkOutDate is NULL) \
) "
titlestr = '%' + titlepattern + '%'
sqlstmt = sqlstr % titlestr
self.cur.execute(sqlstmt)
self.rows = []
while 1:
rec = self.cur.fetchone()
if not rec: break
book = Book(rec,self.cur.description)
url='/scripts/lms/reserve.py?bookid=' + str(book.bookid)
bookURL = HTMLgen.Href(url=url,text='Reserve')
book.data[0] = bookURL
self.rows.append(book.data)
def reserve(self,bookid,empid):
sqlfmt = "insert into checkout (bookid,empid,reserved,checkOutDate)\
values (%s,%s,'1',\'%s/%s/%s\')"
today = Dates.today()
sqlstmt = sqlfmt % (bookid,empid,today.month,today.day,today.year)
print sqlstmt
rc = self.cur.execute(sqlstmt)
if rc != 1:
print "Sql Error in insert", sqlstmt
self.cur.close()
self.cur = None
return 1
def status(self,book):
pass
class Book(Database.RRec):
def __init__(self,rec,description):
Database.RRec.__init__(self,rec,description)
self.data = self.record
if __name__ == '__main__':
s = BookRep()
b = s.findBook('123')
print b.data
BookRep represents a database table and we have provide interfaces on it such that HTMLgen will accept an instance of BookRep as a list of lists . HTMLgen then formats the data into a HTML format and hence we have provided the __setitem__ method. We could have done that in the base class itself but then we would not be able to reuse it for another application as __setitem__ is specific for our case. The above code provides for the ability to search for a book and also reserve it if needed. The LMS database contains three tables - book,checkout and employee. The 'checkout' table has the following fields of interest to us - bookid , empid , checkOutDate, dueDate, reserved . The result of a query should output particulars of the book and its availability or when it will be available for checkout. We should also provide a link that will let the user reserve a book from the query list. Hence we need to do an outer join on book and checkout . The rest is straight SQL. ODBC imposes certain constraints on the syntax of outer joins and the above SQL statement might be different if done interactively from a prompt as opposed to using ODBC. The core of the application is done and all that is needed is a script that uses this and outputs a HTML form .
We'll use one script that outputs a form when invoked without any options and outputs search results if certain fields are passed to it. The action of the form will be the script itself . The search form is a very simple form which has only one input field where you enter the title of the book that you are searching for. The search results form generates the results of the search and also provides a link on each row of the results to which will reserve a book . The reserve script will not be shown here but you can download the entire application and peruse the source for a better understanding . As Linus Torvald (creator of Linux ) (supposedly) said "Use the source Luke, be one with the code" .
The user interface script is search.py and the code follows -
import cgi
from HTMLgen import *
from HTMLcolors import *
import LMS
def search_results(form):
doc = Document('HTML.rc')
doc.title = 'Search Results'
home='/scripts/lms/search.py'
aft = top = home
fore= None
doc.goprev,doc.gonext,doc.gotop,doc.gohome = aft,fore,top,home
body = LMS.BookRep()
body.find(titlepattern=form['title_pattern'].value)
heading = []
for tup in body.cur.description:
heading.append(tup[0])
t = Table( 'Result of the Search',
heading = heading,
body = body,
border = 4 ,
cell_align = 'right')
doc.append(t)
doc.write()
def print_form():
doc = Document('HTML.rc', banner=('/images/forms.gif', 472, 40))
doc.title = 'Search the Library Database'
aft,fore,top,home = None,None,None,None
doc.goprev,doc.gonext,doc.gotop,doc.gohome = aft,fore,top,home
F = Form('/scripts/lms/search.py')
inP = Input(name='title_pattern',llabel='Search Titles',
size=40,maxlength=100)
F.append(inP)
F.append(P())
F.submit = Input(type='submit', value='Fire off to the server')
doc.append(F)
doc.write()
print "Content-type: text/html" # HTML is following
print # blank line, end of headers
form = cgi.FieldStorage()
if form.has_key('title_pattern'):
search_results(form)
else:
print_form()
The above script when invoked invokes the cgi module supplied with the Python distribution. The cgi module provides a dictionary like interface to the name value pairs and handles all the tedious chores of decoding the browser input. We check the input to see if a field 'title_pattern' is passed , if it is not , we print the search form. The name of the input box in our search form is 'title_pattern'. When the user fills out the form and hits Submit , the title to be searched is the value of the name 'title_pattern' in our input. The existence of the 'title_pattern' field gives us the cue to execute the search and output the result.
The search form is shown here search.htm and the search results form is shown here search_results.htm.
We have demonstrated how with Python and ODBC , we can quickly develop Web based applications rapidly and test it out at the same time. Python has many more attractive features that I cannot cover in this article. For more information , please check the newsgroup comp.lang.python and regularly check the web site . Python Web-SIG and the DB-SIG are good mailing lists to be on . Many more complex applications can be web enabled either by using Python or by extending Python to include your code.