Scheme Load Return Values
Shortly after finishing my previous post, I began to wonder if maybe some of the features I had requested were available already. I was pleasantly surprised.
Implementations tested were:
Bad news first. In my hour or so of testing, I was unable to find any way to directly return values from a loaded file in Guile, Gambit, and STklos.
On the other hand, that means that four of the seven implementations that I tested could return values.
MzScheme and MIT Scheme both return the last form, whatever it was. If the last form was a DEFINE statement, MzScheme returns nothing, and MIT Scheme returns the identifier that was defined.
Bigloo doesn't seem to return anything usable, but it does print the results of intermediate evaluations, and claims to return the result of the last evaluation. This may indeed be true, but in my testing I was unable to get it to return anything other than the path of the just-loaded file.
Chicken, however, was an unqualified success. Although it is true that at first it returns nothing from its LOAD function, it provides a brilliant little mechanism to fix that. The LOAD function, in addition to taking a filename, will also take an optional argument called EVALPROC, which is called repeatedly with each form from the loaded file. This fact, combined with a little closure magic, ends us up with this:
(let ((return '()))
(load file
(lambda (exp)
(if (and (list? exp) (eqv? (list-ref exp 0) 'define))
(begin
(eval exp)
(set! return (cons (list-ref exp 1) return)))
(set! return (cons (eval exp) return)))))
(reverse return)))
This function calls the LOAD function, but passes it its own evaluation procedure, which will assemble a list of values when called, as well as evaluating each expression.
After seeing the beautiful and elegant way in which Chicken Scheme allows the user to dynamically alter the behaviour of the LOAD function at runtime, I feel like I should alter my recommendation for modifying that behaviour. Chicken has, I believe, found the best possible way to implement such functionality: have a reasonable default, but allow people to override it if they wish. Simple, elegant, and powerful: exactly like the language itself. Bravo, Chicken.
Bravo.
SoLoad Progression
When I first created SoLoad, I intended to just make available C libraries and functionality, at which point I would be done with the project. But now the scope seems to have grown.
SoLoad has been, if not entirely rewritten, at least extensively renovated. It now uses a plugin architecture, and a concept of "interfaces" which implement a certain set of functionality for a programming language or API.
The C interface retains feature-parity with where it was before the recent changes (but the names of commands have changed) and the Python interface is capable of rudimentary operations.
Yes, that is correct. Python.
The new plugin architecture allows different languages and calling conventions to coexist within the same program. The interfaces to different languages are each encapsulated in their own shared object file, making it possible to, for example, compile a plugin which embeds a python interpreter and then use that interpreter to act upon data calculated by a C function.
Currently, the Python plugin is capable of loading modules, extracting a function, and executing said function, but only if the function takes a single integer argument and returns another integer. Over the next few days, I intend to flesh out the type conversion routines so that any Python function can be executed.
Other languages being considered for support include:
- Common Lisp (through ECL)
- Scheme (through MzScheme)
- Java (through JNI)
And now I get to rewrite all the documentation. Again.
Whoo! Callbacks!
SoLoad now has support for user-defined callback functions.
The syntax goes
cbnew rtype atypes
to create a new callback function. The value returned from this command will always be used when the callback is invoked in the execution of another SoLoad command, so hang onto it.
When the callback is invoked, it will return a message in the form of
callback callbackID arg1 ... argN
instead of a normal function return.
If you want to delete the callback, simply go
cbdel callbackID
but as always, if you don't bother, it will be cleaned up at exit.
Other Changes in SoLoad
The "load" and "type" commands no longer require you to explicitly state the number of types they take. Support for giving referring to libraries, functions, and such by name was removed, as it complicated the program a great deal, and didn't really help at all when SoLoad was invoked from another program. Support for loadable definition files was axed too, with the realization that the information would generally be repeated the language-side bindings anyway, so the functions might as well be loaded dynamically there too.
Unfortunately, these changes mean that my old guides are now subtly inaccurate. I will add a note to that effect later, but will hesitate to write new documentation until I am reasonably sure the interface won't be changing for a while.
A Foreign Function Interface for Ikarus Using SoLoad
A while back I wrote a foreign function server called SoLoad. It is intended for situations where a regular FFI, for whatever reason, is not available to you, but the permissions necessary to run and communicate with programs are available.
Once this was in place, I proceeded to write some code to take advantage of my program, and provide rudimentary C function capabilities to the Ikarus scheme implementation. I got most of the code done, implementing portable type conversion functions, an implementation-independent interface to encapsulate the initialization and communication code, and all the little macros necessary to make the FFI easy to use. It had its bugs, but it worked pretty well. Well enough, at least, to create a window using SDL, and then close it again.
And then I just sort of forgot. Somehow, after getting it all to work and provide a basic level of C library integration to Ikarus, I just wandered off and allowed myself to get involved in other projects.
Today, I was looking through my projects folder, and happened to notice the code I had previously forgotten. Working under the assumption that someone could probably find a use for it, I decided that I would clean it up and post it online.
It required considerably less cleaning that I had expected.
Basic Structure of the Scheme->SoLoad Link
A major design goal in this project was designing it in a way that would allow it to be easily ported to a variety of Scheme implementations and communication channels. The design I ultimately settled on was as follows:
There is one function, SOLOAD-INIT, which handles essentially all of the necessary setup, communication, and teardown code associated with using SoLoad. When called, it is passed the path to the SoLoad executable, and calls it. It then returns a pair, whose CAR is a function to send data to SoLoad and return whatever data SoLoad sends back, and whose CDR is a function that will tell SoLoad to exit and perform any other teardown functions that may be necessary.
Its implementation for Ikarus looks like this:
;;;
;;; soload-init
;;; Starts an instance of SoLoad
;;; Returns a cons cell containing two functions:
;;; car: Send a command to SoLoad, return the output.
;;; cdr: Kill SoLoad and close all sockets.
;;;
(define (soload-init soload-path)
;; Start SoLoad, and bind its input, output, and error ports.
(let-values ([(pid ip op errp)
(process soload-path "stdin")])
;; Transcode the ports we'll be using.
(let ([soload-in (transcoded-port op (native-transcoder))]
[soload-out (transcoded-port ip (native-transcoder))])
(cons
;; Send commands to SoLoad
(lambda (command)
(unless (port-closed? soload-out) ; Make sure it's open
(put-string soload-out command) ; Send the command
(newline soload-out) ; Newline
(flush-output-port soload-out)) ; Flush the port
(unless (port-closed? soload-in) ; Make sure it's open
(get-line soload-in))) ; Get the output
;; Kill SoLoad and close the ports
(lambda ()
(unless (port-closed? soload-out) ; Make sure it's open
(put-string soload-out "exit\n"); Tell it to close
(flush-output-port soload-out) ; Flush output
(close-port soload-out)) ; Close the port
(unless (port-closed? soload-in) ; Make sure it's open
(close-port soload-in)) ; Close the port
(unless (port-closed? errp) ; Make sure it's open
(close-port errp))))))) ; Close the port
And that concludes the implementation-dependent portion of the interface. All of the code above this level is (hopefully) completely independent of whatever particular Scheme it is running on.
Type Conversion
The next important part of the Ikarus/SoLoad FFI is the conversion between Scheme native types, and their text-based representation.
This consists of a myriad of small little functions, to escape/unescape strings, convert numbers, and flatten lists into a single string. They are each easy to implement, and should be entirely portable Scheme code.
All together, they run a little long, but I may as well post them here as well:
;; soload-escape
;; Escape text so SoLoad will read it properly.
;;
(define (soload-escape text)
(string-append "\"" text "\""))
;;
;; soload-unescape
;; Remove the escaping from text that SoLoad returns.
;;
(define (soload-unescape text)
(list->string
(reverse
(cdr
(reverse
(cdr
(string->list text))))))) ; Remove quotation marks
;;
;; native->soload
;; Converts a native type to a
;; string representation that can
;; be passed to SoLoad.
;;
(define (native->soload type value)
(case type
[(int float double long short
uint8 sint8 uint16 sint16
uint32 sint32 uint64 sint64
ushort sshort uint sint
ulong slong char uchar
schar) (number->string value)]
[(string char*) (soload-escape value)]
[(pointer void*) value]
[(void) ""]
[else "0"]))
;;
;; soload->native
;; Converts a SoLoad string
;; into a native type.
;;
(define (soload->native type value)
(case type
[(int float double long short
uint8 sint8 uint16 sint16
uint32 sint32 uint64 sint64
ushort sshort uint sint
ulong slong char uchar
schar) (string->number value)]
[(string char*) (soload-unescape value)]
[(pointer void*) value]
[(void) #t]
[else #f]))
(define (list:native->soload types values)
(if (> (length types) 0)
(cons
(native->soload (car types) (car values))
(list:native->soload (cdr types) (cdr values)))
'()))
(define (list:soload->native types values)
(if (> (length types) 0)
(cons
(soload->native (car types) (car values))
(list:soload->native (cdr types) (cdr values)))
'()))
(define (soload-flatten strings)
(apply string-append
(map
(lambda (element) (string-append " " element))
strings)))
Wrapping It All Up
These functions technically provide all the functionality required to use SoLoad from within Ikarus, but they lack a little something. What they need now is to be made easy. It should be possible to declare a function such that it just works from anywhere in your program, with no need to know that it's calling a helper behind the scenes.
This is possibly the most involved portion of the interface, because it requires individually wrapping every function of SoLoad, and then adding a few macros to neaten things up. In the end, though, I think it works out well:
(define soload-path "soload")
(define (soload-set-path path)
(set! soload-path path))
(define (soload-send command)
(if (equal? soload-process #f)
(set! soload-process (soload-init soload-path)))
((car soload-process) command))
(define (soload-kill)
(if (not (equal? soload-process #f))
(begin
((cdr soload-process))
(set! soload-process #f))))
(define (soload-library path)
(soload-send (string-append "open " path)))
(define (soload-fn-load library name rtype atypes)
(soload-send
(string-append
"load " library " " name " "
(symbol->string rtype) " "
(number->string (length atypes))
(soload-flatten (map symbol->string atypes)))))
(define (soload-fn-call function rtype atypes args)
(soload->native rtype
(soload-send
(string-append
"call " function
(soload-flatten
(list:native->soload atypes args))))))
(define (soload-import definitions)
(soload-send (string-append "def " definitions)))
(define (soload-type-define name . types)
(soload-send
(string-append "type " name " "
(number->string (length types))
(soload-flatten (map symbol->string types)))))
(define (soload-type-create name . args)
(soload-send
(string-append "new " name " "
(soload-flatten (list:native->soload types args)))))
(define (soload-delete name pointer)
(soload-send (string-append "delete " name " " pointer)))
(define-syntax soload-function
(syntax-rules ()
((_ library name (atypes ...) rtype)
(let ([function (soload-fn-load library name
'rtype '(atypes ...))])
(lambda arguments
(soload-fn-call function 'rtype
'(atypes ...) arguments))))
((_ library name (atypes ...))
(soload-function library name (atypes ...) void))
((_ library name rtype)
(soload-function library name () rtype))
((_ library name)
(soload-function library name () void))))
;; Warning! SOLOAD-TYPE does not delete the stuff
;; that it creates. It will stick around till
;; SoLoad is closed.
(define-syntax soload-type
(syntax-rules ()
((_ name types ...)
(let ([type (soload-type-define name types ...)])
(lambda arguments
(soload-send
(string-append
"new " name " "
(soload-flatten
(list:native->soload '(types ...) arguments)))))))))
Using It
Along with a working executable of SoLoad, this code should be all that is needed to use C libraries from within Ikarus. I tested it roughly 5 minutes ago, so I'm fairly certain it works. A working example, assuming all the previous code is in a file called "ikarus-soload.ss", with the soload executable (or a symlink) in the same directory, is as follows:
(soload-set-path "./soload")
;;;
;;; Math Check
;;; Test the value of sin for progressively closer values of pi.
;;; Should return numbers moving closer to 0
;;;
;; Import the library
(define libm (soload-library "/lib/libm.so.6"))
;; Import the function
(define s-sin (soload-function libm "sin" (double) double))
;; Do some tests
(newline)
(pretty-print (s-sin 3))
(pretty-print (s-sin 3.1))
(pretty-print (s-sin 3.14))
(pretty-print (s-sin 3.141))
;;;
;;; SDL Test
;;; Load SDL, create a window, then quit
;;;
;; Load the library
(define lib-sdl
(soload-library "/usr/lib/libSDL.so"))
;; Load a bunch of functions
(define sdl-init
(soload-function lib-sdl "SDL_Init" (uint32) int))
(define sdl-quit
(soload-function lib-sdl "SDL_Quit"))
(define sdl-set-video-mode
(soload-function lib-sdl "SDL_SetVideoMode" (int int int uint32) pointer))
(define sdl-fill-rect
(soload-function lib-sdl "SDL_FillRect" (pointer pointer uint32) int))
(define sdl-flip
(soload-function lib-sdl "SDL_Flip" (pointer) int))
(define sdl-map-rgb
(soload-function lib-sdl "SDL_MapRGB" (pointer uint8 uint8 uint8) uint32))
;; Declare a type
(define SDL_Rect*
(soload-type "SDL_Rect" 'sint16 'sint16 'uint16 'uint16))
(sdl-init 0)
(define sdl-surface (sdl-set-video-mode 640 480 16 0))
(sdl-fill-rect sdl-surface (SDL_Rect* 0 0 640 480) 32535)
(sdl-flip sdl-surface)
(sdl-quit)
(soload-kill)
Now that I've taken a break from this for a while, I can see a few areas it could be made more robust. Next feature: an optional timeout period for SoLoad, so it can close gracefully even if the caller has crashed.
A Few Handy File Transfer Tools (All Written in Python)
File transfer: the eternal problem.
The file is on your computer, you want to get it onto your friend's computer, and you're in a hurry. Luckily, there are tools to help. All of these tools are single files, written in python, and (probably) work on Windows (although I haven't checked)
Single-File Tools
Droopy
Droopy is a wonderful little script. It's wonderfully useful when your friend has a file they need to send to you, and you don't want to make them install anything. Just run it (possibly with some arguments) and it will start a little miniature web-server to which anyone can upload a file, and then it exits. Command line options make it possible to optionally display a message and/or an icon on the upload page.
Woof
Woof (Web Offer One File) is the opposite to Droopy. Instead of recieving a single file, Woof sends one. It can also optionally send an entire folder as a single tar archive. Another nice feature of woof is its "-s" option, which will cause the client to be served an identical copy of the woof.py script.
Heavyweight Contenders
These tools are meant for jobs a little larger than the ones above. The tools already mentioned concern themselves with a single all-important file transfer, which is all well and good. But like the difference between claw hammer and a sledgehammer, sometimes you just need a tool with a little more heft. That's where these scripts come in.
Python HTTP Server
This tool is only barely fancy enough to need mentioning, but the operative word here is "barely". Unknown to almost everyone, a full-featured static HTTP server lurks within the standard python executable under the guise of an "example". All that you have to do is call "python -m SimpleHTTPServer" in a console window, and it will happily begin serving up the current directory on port 8000.
pyftpdlib
All of these scripts are wonderful tools, just perfect for their intended niches. But sometimes, your needs are more complex. What if you need to both send and recieve multiple files? In this case, our old standby FTP comes to hand. In this case, the pyftpdlib library should be our tool of choice. While at first glance it looks rather daunting, a tarball filled with numerous files and subdirectories, the library itself consists of just one file, ftpserver.py. With a few modifications, this file can be the perfect portable, install-less FTP server. I spent about an hour today putting together these changes, and now have about the nicest little FTP server I've ever had the joy of playing with. Just download the FTP server library, replace the test() and main functions at the end with my 50-line changes (basically just giving it command-line options), and you'll have a user-friendly tool for sharing any kind of files you may need.
I love collecting little tools like this. One never knows when they might come in handy.
It's Alive!
Wow. From idle complaints to a working implementation in 12 days. And most of those days I did exactly no work on this project either. I'm surprised.
Long story short, it turns out it's actually not that hard to write a parser, and a lexer is even easier to do if you just use a preexisting regular expression library.
So now I can officially announce the existence of the Helicon Parser Toolkit. It's small, it works for my purposes, and it isn't Lex/Yacc. I don't know what the official classification of the parser is, I just did whatever seemed like a good idea at the time. It has no lookahead currently, but since I just wanted to parse Scheme, that's fine by me.
The lexer and parser together weigh in at 438 lines of code (not including the two header files), and all of it except for about five functions is just bookkeeping.
The lexer uses PCRE for its regular expression matching, and the parser is entirely coded by hand.
As far as Valgrind can tell me, there are no memory leaks other that what GNU Readline brings to the table (maybe this is my fault and I'm using it wrong. Will check later.)
It seems to work so far.
YACCs are Large and Unwieldy, as a Rule
A while back, I decided to try and write an implementation of Scheme, or at least a mostly Scheme-like Lisp. After six tries, I was finally forced to admit that a hand-coded parser just wasn't going to cut it. So I decided that I would make things easier on myself by using Lex and Yacc*.
Now I had two problems.
Now don't get me wrong, Lex and Yacc are probably a couple of very great tools for very certain problems. Unfortunately, they are also a very big pain in the ass to use.
My first issue has to do with input. Lex is a sort of prima donna, refusing to handle any input that isn't handed to it on a gold-plated FILE* handle. The most reliable method I can find for feeding it arbitrary strings is basically "don't bother". Instead, I am told to write out my text to a temp file and read it back in through Lex.
But that isn't so bad. It's certainly possible to work around finicky input conventions. Hell, that's half of what programming seems to be about most of the time.
So then I come to my next issue. I ask myself "What if, during the lexing of one file, I want to go lex another one instead?" As best as I can tell, the answer is again "don't bother". The only facility I can see for opening another file is in the yywrap() function, and I don't even want to begin to contemplate what would happen if I switched pointers mid-lexing.
Now I'm beginning to get uneasy. Are these really the tools that I want to use?
But so far, at least my code is working. It's taken some trial and error, but I have a working program. It reads input, tokenizes it, and even parses it a little. Then something breaks.
Now I run into the next problem. The code output by lex and yacc is opaque. It may just barely be possible to understand what's going on if you have a very good understanding of finite state automata and LR parser theory, but for mere mortals such as myself, no dice. I spend about four hours trying to figure out where in my six parser rules the error lies, but have no luck.
So I decide to go work on something else for a while. I really don't care about parsing. I want to write a scheme interpreter, and parsing scheme code is just a messy prerequisite to this. To reflect this viewpoint, I believe that the lexer and parser should logically be subordinate to the main program. But again, yacc screws me over. I can't tell it "hey, take this bit of text, lex and parse it, then give me back the results." No, instead I have to write stuff to a file, hope that all relevant variables (global of course, no encapsulation for this library) are cleared, and then let yacc take over.
So I take a step back. In times like this, there's a quote that it pays to remember. It says that "When you're up to your ass in alligators, it's hard to remember that your initial objective was to drain the swamp."
I had just spent four hours of my (very limited) free time trying to achieve a goal radically different from what I had set out to do. I couldn't care less about the complexities of yacc, I just wanted to parse some scheme code.
So now I had a decision to make. Do I continue muddling along with lex and yacc, hoping against hope that I'll get something useful out of it, or do I toss the whole thing and hope I can figure out a better way instead?
From where I stood, the wheel under consideration was old, and chipped, and looked more lika a square than a circle. So I decided to reinvent it. Using the state-of-the-art technology of C (Hey, it's one letter better than the original Lex/Yacc), I decided to write my own lexer and parser libraries.
It should be interesting.
* Okay, technically I was using Flex and Bison, but I never got far enough to use the non-backwards-compatible bits.
Long Time, No Post
Until last Sunday, I hadn't posted anything recently. Mostly, this is because I got distracted by Super Mario 64 two weeks ago, and went directly on to Paper Mario 64 the week after that. Partly, it's because I experienced a sudden upsurge in my homework load two weeks ago, which took until yesterday to finish all of. A little bit can be blamed on college applications, and many other little things that ate into my time. But I think a large part of the problem was that the first few posts I wrote on this blog were rather long, and I felt something of an obligation to keep up the quantity.
So. Change. Shorter posts, but posts nonetheless. Hopefully now I'll be able to get myself to write more often.
I ♥ Unit Testing
I have recently been converted. For a while I've been aware that I should probably be doing unit tests to verify that my code is doing what I expect it to, but I could never find a framework that was unobtrusive enough to work for me. Since most of my code is currently in C, I needed a good C unit testing framework to go with it.
Finally, I found one. MinUnit is possibly the simplest unit testing framework imaginable. Measuring in at three lines long, with one of those lines just defining a counter for the number of tests run, I figured it was probably simple enough to start using without any hassle. I was right.
So I started using it. It's a really nice feeling not having to worry about making a change and breaking functionality somewhere else. It's also become a lot easier to write code in small chunks. I can sit down and tell myself "Okay, I'm just going to write tests for this and this case, and when they pass, I'll go do something else." This comes in very handy when you only have a 20 minute break between homework assignments to work in.
Unfortunately, about a week after I started using unit tests, my workload drastically increased, and I got distracted by playing Super Mario 64 and Paper Mario 64. But now things seem to have quieted down again with my completion of Paper Mario 64 yesterday.
And thanks to unit testing, I was able to pick up my coding right where I left off.
The SoLoad Foreign Function Server (Part III) – Intermediate Usage
This is the third post in a series or four. It comes after Introduction and Basic Usage.
In the last two posts, I discussed the reason for SoLoad, and how to use it. The basic usage guide skipped a few of the more complex features in the interest of simplicity. The guide only showed how to use numeric types, and not pointers, strings, or user-defined types. Also, it didn't mention the use of loadable definition files. This post will discuss those features.
Strings
Strings in SoLoad are fairly simple to use. Any sequence of characters, separated by spaces, will be read as a string if its type says so. To add spaces to a string, simply use quotes. To add quotes to a string, use a backslash to escape them, and to add a backslash, use two backslashes. Strings in SoLoad look pretty much like standard C strings.
Say that we have a hypothetical "message" function, which we want to load from a library and use. It takes a string (char*) as an argument, and puts it in a message box. It has no return value.
Loading this function is simple. It looks like
(
is also acceptable)
Calling it is just that simple, too. For a single word output, you can just say
and get "Hello" as a message. Quotes are needed if we want to use spaces in the string:
And quotation marks can be escaped with a backslash:
Pointers
Using pointers in SoLoad is very simple. If all you need to do is to call functions that return pointers, and pass those values to other functions, you can just do that.
0x2364DADB
> call get_value 0x2364DADB
37
If you want to be able to create new pointers or get their values, however, there are a few new commands to master.
type
The
command registers a new type with SoLoad. It has pretty much the same syntax as the argument part of loading a function.
new
The
command creates an instance of a type in memory, and returns a pointer to it.
value
The
command returns the values contained in a struct at the specified address.
37 "Hello, World!"
delete
The
command frees the struct at the given address.
deltype
The del
command deletes a type that was already registered with SoLoad. WARNING: Doing this will probably make SoLoad unable to delete remaining instances of this type.
Def Files
All of the commands for opening a library and loading functions work well enough for one-shot uses, but they can get a little tedious when loading an entire library. Also, I think that bindings developed for SoLoad should easily be usable by any program that uses SoLoad. I intend to provide this functionality through loadable definitions files. These files are simply text files that list various bits of information necessary to load a library and its functions. They are incapable of deleting data, they can only open libraries, load functions, and define types. They look like this:
name sdl paths /usr/lib/libSDL.so /usr/lib/SDL.so /usr/local/lib/libSDL.so function SDL_Init int 1 uint32 function SDL_Quit void 0 function SDL_SetVideoMode pointer 4 int int int uint32 function SDL_WM_ToggleFullScreen int 1 pointer type struct_one 2 int int type struct_two 2 int string
The "name" statement declares a short name for the library.
The "paths" statement lists paths that the library might be found at. They will be tried one at a time in order until one is opened successfully.
The "function" statement loads a function from the library. "function ..." is equivalent to "load <short-name> ..."
The "type" statement does the exact same thing as the "type" statement does in interactive mode.
And that concludes the intermediate level usage tutorial for SoLoad. It should now be possible for you to load and use any kind of function that SoLoad supports. You should also be able to interact with structures, pointers, and strings. In the next (and probably final) post in this series, I will detail the process of creating an FFI library for a dialect of Scheme (just as soon as I get around to writing the library myself).