SoLoad Progression
When I first created SoLoad, I intended to just make available C libraries and functionality, at which point I would be done with the project. But now the scope seems to have grown.
SoLoad has been, if not entirely rewritten, at least extensively renovated. It now uses a plugin architecture, and a concept of "interfaces" which implement a certain set of functionality for a programming language or API.
The C interface retains feature-parity with where it was before the recent changes (but the names of commands have changed) and the Python interface is capable of rudimentary operations.
Yes, that is correct. Python.
The new plugin architecture allows different languages and calling conventions to coexist within the same program. The interfaces to different languages are each encapsulated in their own shared object file, making it possible to, for example, compile a plugin which embeds a python interpreter and then use that interpreter to act upon data calculated by a C function.
Currently, the Python plugin is capable of loading modules, extracting a function, and executing said function, but only if the function takes a single integer argument and returns another integer. Over the next few days, I intend to flesh out the type conversion routines so that any Python function can be executed.
Other languages being considered for support include:
- Common Lisp (through ECL)
- Scheme (through MzScheme)
- Java (through JNI)
And now I get to rewrite all the documentation. Again.
Whoo! Callbacks!
SoLoad now has support for user-defined callback functions.
The syntax goes
cbnew rtype atypes
to create a new callback function. The value returned from this command will always be used when the callback is invoked in the execution of another SoLoad command, so hang onto it.
When the callback is invoked, it will return a message in the form of
callback callbackID arg1 ... argN
instead of a normal function return.
If you want to delete the callback, simply go
cbdel callbackID
but as always, if you don't bother, it will be cleaned up at exit.
Other Changes in SoLoad
The "load" and "type" commands no longer require you to explicitly state the number of types they take. Support for giving referring to libraries, functions, and such by name was removed, as it complicated the program a great deal, and didn't really help at all when SoLoad was invoked from another program. Support for loadable definition files was axed too, with the realization that the information would generally be repeated the language-side bindings anyway, so the functions might as well be loaded dynamically there too.
Unfortunately, these changes mean that my old guides are now subtly inaccurate. I will add a note to that effect later, but will hesitate to write new documentation until I am reasonably sure the interface won't be changing for a while.
A Foreign Function Interface for Ikarus Using SoLoad
A while back I wrote a foreign function server called SoLoad. It is intended for situations where a regular FFI, for whatever reason, is not available to you, but the permissions necessary to run and communicate with programs are available.
Once this was in place, I proceeded to write some code to take advantage of my program, and provide rudimentary C function capabilities to the Ikarus scheme implementation. I got most of the code done, implementing portable type conversion functions, an implementation-independent interface to encapsulate the initialization and communication code, and all the little macros necessary to make the FFI easy to use. It had its bugs, but it worked pretty well. Well enough, at least, to create a window using SDL, and then close it again.
And then I just sort of forgot. Somehow, after getting it all to work and provide a basic level of C library integration to Ikarus, I just wandered off and allowed myself to get involved in other projects.
Today, I was looking through my projects folder, and happened to notice the code I had previously forgotten. Working under the assumption that someone could probably find a use for it, I decided that I would clean it up and post it online.
It required considerably less cleaning that I had expected.
Basic Structure of the Scheme->SoLoad Link
A major design goal in this project was designing it in a way that would allow it to be easily ported to a variety of Scheme implementations and communication channels. The design I ultimately settled on was as follows:
There is one function, SOLOAD-INIT, which handles essentially all of the necessary setup, communication, and teardown code associated with using SoLoad. When called, it is passed the path to the SoLoad executable, and calls it. It then returns a pair, whose CAR is a function to send data to SoLoad and return whatever data SoLoad sends back, and whose CDR is a function that will tell SoLoad to exit and perform any other teardown functions that may be necessary.
Its implementation for Ikarus looks like this:
;;;
;;; soload-init
;;; Starts an instance of SoLoad
;;; Returns a cons cell containing two functions:
;;; car: Send a command to SoLoad, return the output.
;;; cdr: Kill SoLoad and close all sockets.
;;;
(define (soload-init soload-path)
;; Start SoLoad, and bind its input, output, and error ports.
(let-values ([(pid ip op errp)
(process soload-path "stdin")])
;; Transcode the ports we'll be using.
(let ([soload-in (transcoded-port op (native-transcoder))]
[soload-out (transcoded-port ip (native-transcoder))])
(cons
;; Send commands to SoLoad
(lambda (command)
(unless (port-closed? soload-out) ; Make sure it's open
(put-string soload-out command) ; Send the command
(newline soload-out) ; Newline
(flush-output-port soload-out)) ; Flush the port
(unless (port-closed? soload-in) ; Make sure it's open
(get-line soload-in))) ; Get the output
;; Kill SoLoad and close the ports
(lambda ()
(unless (port-closed? soload-out) ; Make sure it's open
(put-string soload-out "exit\n"); Tell it to close
(flush-output-port soload-out) ; Flush output
(close-port soload-out)) ; Close the port
(unless (port-closed? soload-in) ; Make sure it's open
(close-port soload-in)) ; Close the port
(unless (port-closed? errp) ; Make sure it's open
(close-port errp))))))) ; Close the port
And that concludes the implementation-dependent portion of the interface. All of the code above this level is (hopefully) completely independent of whatever particular Scheme it is running on.
Type Conversion
The next important part of the Ikarus/SoLoad FFI is the conversion between Scheme native types, and their text-based representation.
This consists of a myriad of small little functions, to escape/unescape strings, convert numbers, and flatten lists into a single string. They are each easy to implement, and should be entirely portable Scheme code.
All together, they run a little long, but I may as well post them here as well:
;; soload-escape
;; Escape text so SoLoad will read it properly.
;;
(define (soload-escape text)
(string-append "\"" text "\""))
;;
;; soload-unescape
;; Remove the escaping from text that SoLoad returns.
;;
(define (soload-unescape text)
(list->string
(reverse
(cdr
(reverse
(cdr
(string->list text))))))) ; Remove quotation marks
;;
;; native->soload
;; Converts a native type to a
;; string representation that can
;; be passed to SoLoad.
;;
(define (native->soload type value)
(case type
[(int float double long short
uint8 sint8 uint16 sint16
uint32 sint32 uint64 sint64
ushort sshort uint sint
ulong slong char uchar
schar) (number->string value)]
[(string char*) (soload-escape value)]
[(pointer void*) value]
[(void) ""]
[else "0"]))
;;
;; soload->native
;; Converts a SoLoad string
;; into a native type.
;;
(define (soload->native type value)
(case type
[(int float double long short
uint8 sint8 uint16 sint16
uint32 sint32 uint64 sint64
ushort sshort uint sint
ulong slong char uchar
schar) (string->number value)]
[(string char*) (soload-unescape value)]
[(pointer void*) value]
[(void) #t]
[else #f]))
(define (list:native->soload types values)
(if (> (length types) 0)
(cons
(native->soload (car types) (car values))
(list:native->soload (cdr types) (cdr values)))
'()))
(define (list:soload->native types values)
(if (> (length types) 0)
(cons
(soload->native (car types) (car values))
(list:soload->native (cdr types) (cdr values)))
'()))
(define (soload-flatten strings)
(apply string-append
(map
(lambda (element) (string-append " " element))
strings)))
Wrapping It All Up
These functions technically provide all the functionality required to use SoLoad from within Ikarus, but they lack a little something. What they need now is to be made easy. It should be possible to declare a function such that it just works from anywhere in your program, with no need to know that it's calling a helper behind the scenes.
This is possibly the most involved portion of the interface, because it requires individually wrapping every function of SoLoad, and then adding a few macros to neaten things up. In the end, though, I think it works out well:
(define soload-path "soload")
(define (soload-set-path path)
(set! soload-path path))
(define (soload-send command)
(if (equal? soload-process #f)
(set! soload-process (soload-init soload-path)))
((car soload-process) command))
(define (soload-kill)
(if (not (equal? soload-process #f))
(begin
((cdr soload-process))
(set! soload-process #f))))
(define (soload-library path)
(soload-send (string-append "open " path)))
(define (soload-fn-load library name rtype atypes)
(soload-send
(string-append
"load " library " " name " "
(symbol->string rtype) " "
(number->string (length atypes))
(soload-flatten (map symbol->string atypes)))))
(define (soload-fn-call function rtype atypes args)
(soload->native rtype
(soload-send
(string-append
"call " function
(soload-flatten
(list:native->soload atypes args))))))
(define (soload-import definitions)
(soload-send (string-append "def " definitions)))
(define (soload-type-define name . types)
(soload-send
(string-append "type " name " "
(number->string (length types))
(soload-flatten (map symbol->string types)))))
(define (soload-type-create name . args)
(soload-send
(string-append "new " name " "
(soload-flatten (list:native->soload types args)))))
(define (soload-delete name pointer)
(soload-send (string-append "delete " name " " pointer)))
(define-syntax soload-function
(syntax-rules ()
((_ library name (atypes ...) rtype)
(let ([function (soload-fn-load library name
'rtype '(atypes ...))])
(lambda arguments
(soload-fn-call function 'rtype
'(atypes ...) arguments))))
((_ library name (atypes ...))
(soload-function library name (atypes ...) void))
((_ library name rtype)
(soload-function library name () rtype))
((_ library name)
(soload-function library name () void))))
;; Warning! SOLOAD-TYPE does not delete the stuff
;; that it creates. It will stick around till
;; SoLoad is closed.
(define-syntax soload-type
(syntax-rules ()
((_ name types ...)
(let ([type (soload-type-define name types ...)])
(lambda arguments
(soload-send
(string-append
"new " name " "
(soload-flatten
(list:native->soload '(types ...) arguments)))))))))
Using It
Along with a working executable of SoLoad, this code should be all that is needed to use C libraries from within Ikarus. I tested it roughly 5 minutes ago, so I'm fairly certain it works. A working example, assuming all the previous code is in a file called "ikarus-soload.ss", with the soload executable (or a symlink) in the same directory, is as follows:
(soload-set-path "./soload")
;;;
;;; Math Check
;;; Test the value of sin for progressively closer values of pi.
;;; Should return numbers moving closer to 0
;;;
;; Import the library
(define libm (soload-library "/lib/libm.so.6"))
;; Import the function
(define s-sin (soload-function libm "sin" (double) double))
;; Do some tests
(newline)
(pretty-print (s-sin 3))
(pretty-print (s-sin 3.1))
(pretty-print (s-sin 3.14))
(pretty-print (s-sin 3.141))
;;;
;;; SDL Test
;;; Load SDL, create a window, then quit
;;;
;; Load the library
(define lib-sdl
(soload-library "/usr/lib/libSDL.so"))
;; Load a bunch of functions
(define sdl-init
(soload-function lib-sdl "SDL_Init" (uint32) int))
(define sdl-quit
(soload-function lib-sdl "SDL_Quit"))
(define sdl-set-video-mode
(soload-function lib-sdl "SDL_SetVideoMode" (int int int uint32) pointer))
(define sdl-fill-rect
(soload-function lib-sdl "SDL_FillRect" (pointer pointer uint32) int))
(define sdl-flip
(soload-function lib-sdl "SDL_Flip" (pointer) int))
(define sdl-map-rgb
(soload-function lib-sdl "SDL_MapRGB" (pointer uint8 uint8 uint8) uint32))
;; Declare a type
(define SDL_Rect*
(soload-type "SDL_Rect" 'sint16 'sint16 'uint16 'uint16))
(sdl-init 0)
(define sdl-surface (sdl-set-video-mode 640 480 16 0))
(sdl-fill-rect sdl-surface (SDL_Rect* 0 0 640 480) 32535)
(sdl-flip sdl-surface)
(sdl-quit)
(soload-kill)
Now that I've taken a break from this for a while, I can see a few areas it could be made more robust. Next feature: an optional timeout period for SoLoad, so it can close gracefully even if the caller has crashed.
The SoLoad Foreign Function Server (Part III) – Intermediate Usage
This is the third post in a series or four. It comes after Introduction and Basic Usage.
In the last two posts, I discussed the reason for SoLoad, and how to use it. The basic usage guide skipped a few of the more complex features in the interest of simplicity. The guide only showed how to use numeric types, and not pointers, strings, or user-defined types. Also, it didn't mention the use of loadable definition files. This post will discuss those features.
Strings
Strings in SoLoad are fairly simple to use. Any sequence of characters, separated by spaces, will be read as a string if its type says so. To add spaces to a string, simply use quotes. To add quotes to a string, use a backslash to escape them, and to add a backslash, use two backslashes. Strings in SoLoad look pretty much like standard C strings.
Say that we have a hypothetical "message" function, which we want to load from a library and use. It takes a string (char*) as an argument, and puts it in a message box. It has no return value.
Loading this function is simple. It looks like
(
is also acceptable)
Calling it is just that simple, too. For a single word output, you can just say
and get "Hello" as a message. Quotes are needed if we want to use spaces in the string:
And quotation marks can be escaped with a backslash:
Pointers
Using pointers in SoLoad is very simple. If all you need to do is to call functions that return pointers, and pass those values to other functions, you can just do that.
0x2364DADB
> call get_value 0x2364DADB
37
If you want to be able to create new pointers or get their values, however, there are a few new commands to master.
type
The
command registers a new type with SoLoad. It has pretty much the same syntax as the argument part of loading a function.
new
The
command creates an instance of a type in memory, and returns a pointer to it.
value
The
command returns the values contained in a struct at the specified address.
37 "Hello, World!"
delete
The
command frees the struct at the given address.
deltype
The del
command deletes a type that was already registered with SoLoad. WARNING: Doing this will probably make SoLoad unable to delete remaining instances of this type.
Def Files
All of the commands for opening a library and loading functions work well enough for one-shot uses, but they can get a little tedious when loading an entire library. Also, I think that bindings developed for SoLoad should easily be usable by any program that uses SoLoad. I intend to provide this functionality through loadable definitions files. These files are simply text files that list various bits of information necessary to load a library and its functions. They are incapable of deleting data, they can only open libraries, load functions, and define types. They look like this:
name sdl paths /usr/lib/libSDL.so /usr/lib/SDL.so /usr/local/lib/libSDL.so function SDL_Init int 1 uint32 function SDL_Quit void 0 function SDL_SetVideoMode pointer 4 int int int uint32 function SDL_WM_ToggleFullScreen int 1 pointer type struct_one 2 int int type struct_two 2 int string
The "name" statement declares a short name for the library.
The "paths" statement lists paths that the library might be found at. They will be tried one at a time in order until one is opened successfully.
The "function" statement loads a function from the library. "function ..." is equivalent to "load <short-name> ..."
The "type" statement does the exact same thing as the "type" statement does in interactive mode.
And that concludes the intermediate level usage tutorial for SoLoad. It should now be possible for you to load and use any kind of function that SoLoad supports. You should also be able to interact with structures, pointers, and strings. In the next (and probably final) post in this series, I will detail the process of creating an FFI library for a dialect of Scheme (just as soon as I get around to writing the library myself).
The SoLoad Foreign Function Server (Part II) – Basic Usage
I'm sorry this post has taken so long. I started writing it over a week ago, but by the time it was finished, it looked more like a reference document than a guide to basic usage. I decided that a document like that would be much better if placed on the SoLoad Wiki.
After much editing down, I ended up with this tutorial, which is much easier to read.
The SoLoad Foreign Function Server is a program that allows interpreters and scripting languages to call C functions. The only requirements for use are those necessary to start a process and communicate with it.
Previously, I explained the need for and the purpose of SoLoad. In this post, I will explain how it is used in a command-line session.
Starting
SoLoad has, at the time of this writing, two methods of communication with the outside world. It can talk over the standard input/output streams, and via TCP. To start SoLoad and control it over stdin, use the command
Similarly, to use it over TCP, just say
Once you have connected to the port via telnet, there should be no further differences between the different communications methods.
Opening a Library
To open a library, use the
command. The
command takes one or two arguments. The first is the path to the library, and the second is an optional short name for the library. If provided, this short name can be used interchangeably with the library identifier that is returned by the command.
For the purposes of this example, we will try to open the
shared library. I assume that the path to it is "/lib/libm.so.6" . If it is different on your system, the command will need to be modified accordingly.
So, to open the libm library, the command goes
This opens the library and assigns it a short name of "libm".
Loading a Function
Loading a function from a library is done with the
command. The load command takes as its arguments: a library identifier, the name of the function to be loaded, the return type, the number of arguments to the function, and the types of all the arguments. For our purposes, the command will look like
If, however, we had opted not to provide a short name for the library when opening it, the call would probably look something more like
Calling a Function
To call a function, use the
command takes a function identifier, and a number of arguments. For us, it will look like
and will return the result of the call.
Exiting
To exit, simply type
. Everything should automatically be unloaded, and SoLoad will close.
Cleanup (Optional)
close <em>library-id</em>
The following functions don't usually need to be used. They are provided simply because doing so adds barely any complexity to the program, and allows use in situations where libraries must be continually loaded and unloaded without exiting, and without filling up all available memory with old library references.
Again, SoLoad will automatically do this cleanup when it exits, so you are free to just quit now if you wish.
To unload a function, use the
command. It takes a function identifier as its only argument. It would look like
Similarly, to close a library, simply say
The library is closed, and functions loaded from it are no longer valid. (Note: Closing a library does not automatically unload all of its functions, but those functions will still be unloaded at exit)
A More Complex Example
The following is a transcript of a simple interaction with the SDL library. SoLoad will be used to open SDL, create a window, and then quit.
> soload stdin > open /usr/lib/libSDL.so sdl 0x9fd7058 > load sdl SDL_Init int 1 uint32 0x9fd7be8 > load sdl SDL_SetVideoMode pointer 4 int int int uint32 0x9fd7bc8 > load sdl SDL_Quit void 0 0x9fd7d88 > call SDL_Init 0 0 > call SDL_SetVideoMode 640 480 16 0 0xa004998 > call SDL_Quit @� > exit exit
Stay tuned for the next post in the series, in which I will discuss intermediate level usage (structs and strings).
The SoLoad Foreign Function Server (Part I) – An Introduction
The largest obstacle in my path when I try to accomplish anything in scheme is the lack of a standardized foreign function interface (FFI). I understand the reasons for it, I understand that there are many implementations running on platforms and architectures where a foreign function interface wouldn't make any sense, I understand that in some cases, such as JVM-based implementations, any usable FFI would have to look radically different from the standard C library interface that we most often want. I understand all of this, but still it annoys me a little.
My needs in an FFI are simple. All I want to do is to load C libraries such as OpenGL and GTK, and call the functions they export. I don't need anything fancy, like callbacks or continuations. I just want something that lets me make stuff appear on the screen (the ability to use optimized math libraries from within an interpreted language would be a plus, too).
Most implementations offer a C FFI or some sort, but their usage is always radically different. They have different capabilities, and different failure conditions, and even if their mode of operation is the same, the command semantics are usually different. Once you start using one of them, you aren't coding in scheme any more, you're coding in PLT Scheme, or Gambit Scheme, or Chicken Scheme, or any of the myriad other implementations out there. And this is ignoring all those implementations, such as Ikarus, who for the sake of simplicity don't even offer an FFI at all.
There is hope, however. The basic solution looks something like Peter van Eerten's GTK-Server.
Most every scheme implementation makes some provision for starting external programs and talking to them, or at the very least communicating over TCP. Once we realize this, our problem becomes vastly simpler. Our task is no longer to write a standard foreign function interface that unifies all the different schemes out there. All we have to do is come up with an interface that can start an external program, and communicate with it. The semantics for this task are much simpler and often much more similar across implementations than those of an FFI.
Of course, just having this library is no good. If we have that, we're very well placed to use GTK-Server, but essentially no better off at interfacing with any C library that comes along. To make this work, we're going to need something to talk to.
That's what SoLoad is. SoLoad implements a simple text-based protocol which allows you to do some fancy things with C libraries. Using SoLoad, you can open libraries, load functions, and call them, as well as messing around with the types and values of pointers. There is also support for "definition" files, which allow you to load all the defined functionality from a library with a single call.
As well as bringing all of this functionality much closer to portable availability in scheme, the existence of this program has the rather pleasant side effect of making all that functionality availably to any language that has basic process starting and text sending abilities. Feel like programming the next killer FPS in awk, anyone?
Stay tuned for Part 2, in which we learn how to actually make use of SoLoad.