Go to the previous, next section.

Porting ILU to Common Lisp Implementations

The ILU runtime for Common Lisp is largely written in vanilla Common Lisp. The lisp-implementation-specific details are confined to a small number of macros and functions which need to be defined. The major work is to write ilu-xxx.lisp, where "xxx" is the specifier for the particular implementation of Common Lisp in use, and any necessary xxx-to-C shims in ilu-xxx-skin.c. There are a number of things that have to be done in ilu-xxx.lisp. They can be regarded in three major sections: providing the ILU notion of foreign-function calls, connecting the Lisp's garbage collector to the ILU network GC, and providing either a threaded or event-loop model of operation. In addition, there is a small hook that has to be provided to convert between character sets.

Providing the ILU notion of foreign-function calls.

Perhaps the trickiest is to provide an implementation of the macro "define-c-function". This maps the ILU notion of a call into C into the native lisp notion. "define-c-function" has the signature

Macro: ilu::define-c-function (LISP-NAME symbol) (DOC-STRING string) (C-NAME string) (ARGS list) (RETURN-TYPE keyword) &key (INLINE boolean cl:nil)

The LISP-NAME is a symbol which will be the name of the function in Common Lisp. The C-NAME is a string which will be the "regular" C name of the C function to be called; that is, the name as it would be named in a C program, rather than the name of the symbol for the entry point of the function. ARGS is a list of arg which describe the signature of the C function, where each arg is either a keyword or a 2-tuple. If a keyword, the keyword indicates the type of the argument. Allowable argument types are

If the arg is a 2-tuple, the cadr is the type, and the car is the "direction", which may be either :in, :out, or :inout. Args with no "direction" are by default of direction :in. The RETURN-TYPE argument is a keyword for the return type of the function, which is drawn from the same set of keywords as the argument types. Return-types may also use the keyword :void, which specifies that no value is returned. The INLINE keyword is a boolean value which, if cl:t, indicates that the necessary type-checking has been assured by the application code, and that the C function may be called directly without type-checking the parameters.

define-c-function defines a Common Lisp function with a possibly different signature from the C function. This function has arguments which consist of all the :in and :inout arguments of the C function, in the order in which they occur in the signature of the C function. It returns possibly multiple values, which consist of the specified return type, if not :void, followed by any :out and :inout arguments to the C function, in the order in which they occur in the signature of the C function.

define-c-function assumes that the C function will call back into Common Lisp, and that gc may occur during the invocation of the C function. Therefore, any objects passed to C which are not values must be registered in some way to prevent them from moving during the call. Often this means that the actual call must be surrounded by code which makes static copies of, for example, strings, calls the C function, then frees the static copy after the call. In addition, when "catching" :out arguments and :inout arguments, it is usually necessary to pass a pointer to the appropriate argument, rather than the argument directly. Typically 1-element arrays have to be allocated to do this. The Franz ACL implementation uses a resource of arrays to minimize consing for this.

We should probably add another keyword, NO-CALLBACKS, to indicate that the C function will not call back into Common Lisp (and therefore some of the GC protection can be skipped when calling this function). Providing for NO-CALLBACKS in your implementation would probably be a good idea.

Network Garbage Collection

The Common Lisp-specific runtime must provide three calls which allow the kernel to map the kernel's C ILU object to a CLOS object. These are register-lisp-object, lookup-registered-lisp-object, and unregister-lisp-object. The idea behind these is to provide the C runtime with a handle on a CLOS object that is a small integer that will not be moved by Common Lisp GC, and to provide a layer which weak references can hide behind.

Function: ilu::register-lisp-object (OBJ ilu:ilu-object) &key (REFTYPE keyword :strong) => fixnum

The OBJ is an ILU CLOS object (the Franz ACL implementation accepts any Common Lisp value except NIL, but this is only because it uses it internally in `ilu-franz.lisp'). The REFTYPE keyword may be either the keyword :weak or the keyword :strong, which determines whether the reference to the object is a weak reference or a strong reference. A weak reference is one that is not "followed" by the Common Lisp collector. The returned value is a fixnum that can be used with lookup-registered-lisp-object and unregister-lisp-object to find the object or remove the reference to the object, respectively.

Function: ilu::lookup-registered-lisp-object (INDEX fixnum) => ilu:ilu-object

This function follows the reference indicated by INDEX and returns the object, or cl:nil if the INDEX is invalid.

Function: ilu::unregister-lisp-object (INDEX fixnum)

Causes any reference indicated by INDEX to be removed.

Macro: ilu::optional-finalization-hook (OBJ ilu:ilu-object)

This is a macro which should be defined in such a way as to indicate a finalization action for OBJ when the Common Lisp collector collects it. This finalization action will interact with the ILU kernel to ensure that remote peers of this Common Lisp will know that it no longer has an interest in the object. In addition, the finalization action will be able to prevent OBJ from being actually collected, should any peer have an active reference to it.

The Franz ACL implementation only allows the collector to run the finalization when it knows that no peer has a reference, by keeping the Common Lisp reference to the object as a strong reference until the C ILU kernel informs the Common Lisp ILU runtime that no peer has a reference, in which case the Common Lisp reference is changed to a weak reference. In time this allows the collector to GC the object, and the finalization action is called. The action that needs to be taken is "null out" both the pointer from the CLOS object to the C object, via (setf (ilu-cached-kernel-obj lisp-obj) nil), and "null out" the reference from the C object to the CLOS object, via (register-language-specific-object (kernel-obj lisp-obj) 0). See `ilu-franz.lisp', ilu::franz-shutdown-ilu-object, for an example. The Franz ACL example also does these shutdowns in a separate thread, instead of doing them directly in the GC finalization process. This is because the shutdown actions may cause arbitrary callbacks into Common Lisp, some of which may not occur on the stack of the ACL scheduler, which may invoke the collector.

If you feel that it just isn't possible to hook your Common Lisp collector into the network GC, you can simply define register-lisp-object to ignore the REFTYPE parameter, and define optional-finalization-hook to expand to nothing. The result will be that no ILU object in your address space will ever be GC'ed, and that no true instance of a collectible ILU object type referenced by your process will ever be GC'ed anywhere in its true address space until your Common Lisp image disappears. This might also be a good starting point, just to get the other parts working.

Thread and/or Event Loops

Every address space into which ILU is loaded is implicitly a server. This is partially because ILU uses method calls internally, such as pinging garbage collection callbacks, and partially because it provides for recursive protocols, in which a "server" might call back to a "client" during the execution of a method call. This means that any implementation of ILU has to provide a way to execute incoming calls; which means that it has to provide a stack and thread of control in which to execute the "true" code of the method call. There are two mechanisms supported by ILU to associate a thread of control with an incoming request, threads and event loops. In the thread model, each request is executed in a thread associated with either the specific request (thread-per-request) or the connection on which the thread arrives (thread-per-client). In the event loop model, one thread of control is multiplexed between all uses by means of calls into particular "event handler" routines when some "event" is delivered to the process. Typical events are timer expirations, I/O available on file descriptors, UNIX signals. Other more application-specific events are possible, such as X Window System events or XView toolkit events.

For a threaded Common Lisp, the thread model is preferred. To support this, the implementor of the Common Lisp runtime must call the C procedure ilu_SetWaitTech() with two C-callable routines that provide ways to block the current thread until input or output is available on a particular file descriptor. He must call ilu_SetMainLoop() with a main loop struct that provides NULL procedures for the ml_run, ml_exit, ml_register_input, and ml_register_output fields, simple procedures that return ilu_FALSE for the ml_unregister_input and ml_unregister_output fields, and three C-callable procedures that implement creation, setting, and unsetting of alarms for the ml_create_alarm, ml_set_alarm, and ml_unset_alarm fields. Finally, he must provide C-callable procedures to describe his thread system's mutex and condition variable system to the ILU C kernel, and register them by calling ilu_SetLockTech(). See the Franz ACL implementation for an example of this. Note that the file `ilu-process.lisp' provides an implementation-independent veneer over various process systems. It would be useful to extend that, then use it in providing the specific thread mechanisms, rather than using your Common Lisp's threads directly.

For an non-threaded Common Lisp, the event loop model is available. In this, you divide up all computation in your application into event handlers, separate functions that are run when some event occurs, and initialize the system by calling some event handler dispatcher routine, often called the "main loop" of the system. ILU provides a default main loop in the kernel, which provides support for two kinds of events: timer expiration (ILU calls timers "alarms"), and input or output available on a UNIX file descriptor. This means that handler functions can be registered to be called when an event of one of these types occurs. The ILU event loop is also "recursive"; this means that event handlers can call back into the main loop to wait for something to occur. To use the ILU main loop, you must provide mainly a way to invoke the main loop, probably something like ilu:xxx-main-loop, where "xxx" is the name of your flavor of Common Lisp.

If the ILU main loop is for some reason not satisfactory, a Common Lisp-runtime-specific main loop can be substituted via a call to the ILU C kernel routine ilu_SetMainLoop(). This is often necessary to interoperate with UI toolkits like XView or Tk which believe that they own the main loop. Note that this main loop must provide all the functionality provided by the ILU main loop. A less-powerful main loop can be used in addition to the ILU main loop, by calling the ILU C kernel routine ilu_AddRegisterersToDefault(). See the comments in `ILUSRC/runtime/kernel/iluxport.h' for documentation of all of this.

In addition to making the appropriate calls into the ILU kernel to set up either threaded mode or event-loop mode, the Common Lisp runtime implementor must provide a few required function calls:

Function: ilu::initialize-locking

This misnamed function is called by the generic ILU Common Lisp runtime to set up the interaction mode, start the scheduler if necessary, and in general do anything necessary to initialize the Common Lisp-flavor-specific Common Lisp runtime.

Function: ilu::setup-new-connection-handler (FN function) (SERVER C-pointer) (PORT C-pointer)

This is called when a client connects to a kernel server, SERVER, implemented in this address space. It should arrange to apply FN to (list SERVER PORT) if a new incoming connection is received on PORT. FN should return cl:nil if no handler could be established, non-cl:nil otherwise. SERVER is the C address of an ILU kernel ilu_Server, PORT is the C address of an ILU kernel ilu_Port. The ILU C kernel routine ilu_FileDescriptorOfMooringOfPort() will return the UNIX file descriptor of the ilu_Mooring of an ilu_Port. In threaded Common Lisps, this will typically cause a thread to be forked, which will watch for connections to this port. In event-loop Common Lisps, this will typically register FN as an event handler for "input available on the file descriptor of the mooring of PORT".

Function: ilu::setup-connection-watcher (FN function) (CONN C-pointer) (SERVER C-pointer)

This is called when a new connection is setup. It should arrange things so that FN is applied to (list CONN SERVER) whenever input is available on CONN. FN should return non-cl:nil if the input was successfully handled, cl:nil otherwise. If FN ever returns cl:nil, the connection-watcher should be demolished. CONN is the C address of an ILU kernel ilu_Connection, and SERVER is the C address of an ILU kernel ilu_Server. The ILU C kernel routine ilu_FileDescriptorOfConnection() will return the UNIX file descriptor for an ilu_Connection. In threaded Common Lisps, this will typically fork a thread which will handle requests coming in on this connection. In event-loop Common Lisps, this will typically register FN as an event handler for "input available on the file descriptor of the connection".

Converting between character sets.

This section is not currently correct, but we are changing the Lisp runtime to make it correct.

ILU uses the ISO Latin-1 and Unicode (ISO 10646) character sets. Common Lisp uses a somewhat different version of `character'. To provide for a mapping back and forth between ILU and Common Lisp, the runtime implementor must provide four macros:

Macro: ilu::construct-lisp-character-from-unicode (UNICODE (unsigned-byte 16))) => character

Macro: ilu::determine-unicode-of-character (LISP-CHAR character) => Unicode-code

Macro: ilu::construct-lisp-character-from-latin-1 (LATIN-1-CODE (unsigned-byte 8)) => character

Macro: ilu::determine-latin-1-of-character (LISP-CHAR character) => ISO-Latin-1-code

which I trust are self-explanatory.

Go to the previous, next section.