The SoLoad Foreign Function Server (Part II) – Basic Usage
I'm sorry this post has taken so long. I started writing it over a week ago, but by the time it was finished, it looked more like a reference document than a guide to basic usage. I decided that a document like that would be much better if placed on the SoLoad Wiki.
After much editing down, I ended up with this tutorial, which is much easier to read.
The SoLoad Foreign Function Server is a program that allows interpreters and scripting languages to call C functions. The only requirements for use are those necessary to start a process and communicate with it.
Previously, I explained the need for and the purpose of SoLoad. In this post, I will explain how it is used in a command-line session.
Starting
SoLoad has, at the time of this writing, two methods of communication with the outside world. It can talk over the standard input/output streams, and via TCP. To start SoLoad and control it over stdin, use the command
Similarly, to use it over TCP, just say
Once you have connected to the port via telnet, there should be no further differences between the different communications methods.
Opening a Library
To open a library, use the
command. The
command takes one or two arguments. The first is the path to the library, and the second is an optional short name for the library. If provided, this short name can be used interchangeably with the library identifier that is returned by the command.
For the purposes of this example, we will try to open the
shared library. I assume that the path to it is "/lib/libm.so.6" . If it is different on your system, the command will need to be modified accordingly.
So, to open the libm library, the command goes
This opens the library and assigns it a short name of "libm".
Loading a Function
Loading a function from a library is done with the
command. The load command takes as its arguments: a library identifier, the name of the function to be loaded, the return type, the number of arguments to the function, and the types of all the arguments. For our purposes, the command will look like
If, however, we had opted not to provide a short name for the library when opening it, the call would probably look something more like
Calling a Function
To call a function, use the
command takes a function identifier, and a number of arguments. For us, it will look like
and will return the result of the call.
Exiting
To exit, simply type
. Everything should automatically be unloaded, and SoLoad will close.
Cleanup (Optional)
close <em>library-id</em>
The following functions don't usually need to be used. They are provided simply because doing so adds barely any complexity to the program, and allows use in situations where libraries must be continually loaded and unloaded without exiting, and without filling up all available memory with old library references.
Again, SoLoad will automatically do this cleanup when it exits, so you are free to just quit now if you wish.
To unload a function, use the
command. It takes a function identifier as its only argument. It would look like
Similarly, to close a library, simply say
The library is closed, and functions loaded from it are no longer valid. (Note: Closing a library does not automatically unload all of its functions, but those functions will still be unloaded at exit)
A More Complex Example
The following is a transcript of a simple interaction with the SDL library. SoLoad will be used to open SDL, create a window, and then quit.
> soload stdin > open /usr/lib/libSDL.so sdl 0x9fd7058 > load sdl SDL_Init int 1 uint32 0x9fd7be8 > load sdl SDL_SetVideoMode pointer 4 int int int uint32 0x9fd7bc8 > load sdl SDL_Quit void 0 0x9fd7d88 > call SDL_Init 0 0 > call SDL_SetVideoMode 640 480 16 0 0xa004998 > call SDL_Quit @� > exit exit
Stay tuned for the next post in the series, in which I will discuss intermediate level usage (structs and strings).
A Scheme Syntax-Rules Primer
Scheme has a wonderfully powerful hygienic macro system. Unfortunately, explanations on how to use it are few and far between. R5RS is utterly incomprehensible to anyone who doesn't already have a firm grounding in hygienic macro systems, and TYSiFD's section on macros dates from a time before syntax-rules and hygiene.
So this here is my attempt to share what I've learned over the past few days with regards to syntax-rules macros. Bear in mind that at the time of writing this, I have known how to use syntax-rules since yesterday, but my knowledge seems to be complete enough to write a working module system, so here's my attempt to pass on what I've learned.
First things first. You use define-syntax to create a top level binding, and let-syntax bears the same relationship to define-syntax as you'd expect.
;; a top-level binding of a macro
(define-syntax macro
<syntax transformer>)
That much is pretty clear from reading R5RS, now we need to figure out how to write the syntax transformer. There are a few ways to do this, but the best way for beginners is to use syntax-rules, which avoids inadvertent variable capture and other such nasties automatically, and uses a rather elegant pattern matching language.
We'll start by trying to write a simple macro, while. This will just be a standard looping construct that keeps executing over and over until its condition becomes false. We want its usage to look something like this:
(define x 0)
(while (< x 5)
(set! x (+ x 1))
(print x))
So first, we need the define-syntax form.
<syntax transformer>)
Next, we need to write the syntax transformer itself. This is where syntax-rules comes into play. Syntax-rules uses pattern matching and text substitution to allow you to make some pretty advanced macros. It looks like:
(syntax-rules (<keywords>)
((<pattern>) <template>)
...
((<pattern>) <template>)))
I will explain keywords later. For now, just leave that bit blank. What we're interested in are those ((<pattern>) <template>) pairs. Each <pattern> is just that, a pattern of code that will be matched. In our case, we want to match the pattern:
Where the '...' signifies that body may contain one or more forms. Luckily for us, this is exactly the syntax that syntax-rules wants to see, so we can just plug it in, giving us:
(syntax-rules ()
((while condition body ...) <template>)))
So far so good. Now we just have to fill in the other half, with a suitable <template>
Before we can write the <template>, though, we have to decide what we want the code to end up looking like. Since this isn't a guide to scheme code in general, I'll just go ahead and say that we want the output to look like:
(let loop ()
(if condition
(begin
body ...
(loop))
#f))
Got that? Okay, now we've just got to put this in our syntax-rules macro as a template. By another startling coincidence, this is exactly what the template code is expected to look like. We just plug in that code, and our final result is:
(syntax-rules ()
((while condition body ...)
(let loop ()
(if condition
(begin
body ...
(loop))
#f)))))
Just plug that into your scheme interpreter, and our while loop from earlier should execute perfectly.
Now let's try to write something a little more complicated. We want to write a for loop similar to the one that Python has. This should be a pretty easy task, since it's basically just syntactic sugar for scheme's map function.
Our goal is to be able to write a piece of code taking the form:
<body ...>)
And have it expand to:
(for-each (lambda (<element>)
<body ...>) <list>)
Our first try would probably look something like this:
(syntax-rules ()
((for element in list body ...)
(map (lambda (element)
body ...)
list))))
This works, but there's one issue with it. All of the following are valid and work exactly the same:
(for i fnord '(0 1 2 3 4) (print i))
(for i some-other-keyword '(0 1 2 3 4) (print i))
This is not so much of a problem in the case of a for loop, but what if you wanted to add another rule later, so that
(print i))
will also work? The solution to this problem is in that <keywords> argument that we glossed over earlier. Change the keywords list to include 'in' (and, for good measure, 'as'), and it will allow those symbols, and only those symbols, in places where they are mentioned. This change leaves us with:
(syntax-rules (in as)
((for element in list body ...)
(map (lambda (element)
body ...)
list))
((for list as element body ...)
(map (lambda (element)
body ...)
list))
Or, for simplicity (thanks to Dan Prager for pointing this out)
(syntax-rules (in as)
((for element in list body ...)
(map (lambda (element)
body ...)
list))
((for list as element body ...)
(for element in list body ...))))
And if we load this code into our scheme interpreter of choice, we should have two fully functional little bits of new syntax.
Hopefully this guide will help shed some light on the arcane subject that is the Scheme macro system, and hopefully I will never have to learn enough about syntax-case to write a tutorial on it.
The SoLoad Foreign Function Server (Part I) – An Introduction
The largest obstacle in my path when I try to accomplish anything in scheme is the lack of a standardized foreign function interface (FFI). I understand the reasons for it, I understand that there are many implementations running on platforms and architectures where a foreign function interface wouldn't make any sense, I understand that in some cases, such as JVM-based implementations, any usable FFI would have to look radically different from the standard C library interface that we most often want. I understand all of this, but still it annoys me a little.
My needs in an FFI are simple. All I want to do is to load C libraries such as OpenGL and GTK, and call the functions they export. I don't need anything fancy, like callbacks or continuations. I just want something that lets me make stuff appear on the screen (the ability to use optimized math libraries from within an interpreted language would be a plus, too).
Most implementations offer a C FFI or some sort, but their usage is always radically different. They have different capabilities, and different failure conditions, and even if their mode of operation is the same, the command semantics are usually different. Once you start using one of them, you aren't coding in scheme any more, you're coding in PLT Scheme, or Gambit Scheme, or Chicken Scheme, or any of the myriad other implementations out there. And this is ignoring all those implementations, such as Ikarus, who for the sake of simplicity don't even offer an FFI at all.
There is hope, however. The basic solution looks something like Peter van Eerten's GTK-Server.
Most every scheme implementation makes some provision for starting external programs and talking to them, or at the very least communicating over TCP. Once we realize this, our problem becomes vastly simpler. Our task is no longer to write a standard foreign function interface that unifies all the different schemes out there. All we have to do is come up with an interface that can start an external program, and communicate with it. The semantics for this task are much simpler and often much more similar across implementations than those of an FFI.
Of course, just having this library is no good. If we have that, we're very well placed to use GTK-Server, but essentially no better off at interfacing with any C library that comes along. To make this work, we're going to need something to talk to.
That's what SoLoad is. SoLoad implements a simple text-based protocol which allows you to do some fancy things with C libraries. Using SoLoad, you can open libraries, load functions, and call them, as well as messing around with the types and values of pointers. There is also support for "definition" files, which allow you to load all the defined functionality from a library with a single call.
As well as bringing all of this functionality much closer to portable availability in scheme, the existence of this program has the rather pleasant side effect of making all that functionality availably to any language that has basic process starting and text sending abilities. Feel like programming the next killer FPS in awk, anyone?
Stay tuned for Part 2, in which we learn how to actually make use of SoLoad.
A Minimalistic Module System for Scheme
I like scheme.
I really, really like scheme.
Unfortunately, it seems like a lot of the time the only way to get worthwhile things done in scheme is to abandon the conceptual purity and the elegance, all of the things that make scheme as wonderful as it is.
I figure I'm as good a person as any to try and fix at least a little of that.
The first (second actually, but more about that in the next post) step toward that is some sort of cross-implementation module system. As far as I know, the only such systems in existence so far are SLIB and Snowfort. While both of these are wonderful systems, they overshoot the mark, in my opinion. SLIB aims to be a complete all-singing all-dancing library system, and Snowfort integrates all kinds of package management with the simple act of breaking a program up into modules.
Well, that's the beauty of lisp, isn't it? If you need something, you can just code it yourself.
So I did. The preliminary result is only 19 lines of code, 6 of which are simply a function to build an alist from a key list and a value list. It can also double as a simple object system if you want to use it that way.
Its usage is simple. There are two macros, provide and module. Provide is basically the same as module, but anonymous, so it needs no documentation other than to say that you omit the name.
;;; Show off the usage of simple-module.scm
;;; By Will Donnelly
;; This program is free software. It comes without any warranty, to
;; the extent permitted by applicable law. You can redistribute it
;; and/or modify it under the terms of the Do What The Fuck You Want
;; To Public License, Version 2, as published by Sam Hocevar. See
;; http://sam.zoy.org/wtfpl/COPYING for more details.
(load "simple-module.scm")
;; (module <name> (<exports> ...) <body> ...)
(module mod-name (foo incr)
(define foo 37)
(define i 10) ; i cannot be accessed directly
(define (incr)
(set! i (+ i 1))
i))
(mod-name 'foo) ; => 37
((mod-name 'incr)) ; => 11
((mod-name 'incr)) ; => 12
((mod-name 'incr)) ; => 13
That's all you have to do to use it. The code inside the module system is about as simple as that too.
;;; A minimalistic module system for scheme
;;; By Will Donnelly
;; This program is free software. It comes without any warranty, to
;; the extent permitted by applicable law. You can redistribute it
;; and/or modify it under the terms of the Do What The Fuck You Want
;; To Public License, Version 2, as published by Sam Hocevar. See
;; http://sam.zoy.org/wtfpl/COPYING for more details.
(define (make-alist names values)
(if (equal? (length names) (length values))
(if (> (length names) 0)
(cons (cons (car names) (car values)) (make-alist (cdr names) (cdr values)))
'())
#f))
(define-syntax provide
(syntax-rules ()
((_ (exports ...) body ...)
((lambda ()
body ...
(define export-alist
(make-alist '(exports ...) (list exports ...)))
(lambda (symbol)
(cdr (or (assoc symbol export-alist) '(#f . #f)))))))))
(define-syntax module
(syntax-rules ()
((_ name (exports ...) body ...)
(define name (provide (exports ...) body ...)))))
Just load that code into your scheme interpreter, and you have a simple module system. That's all there is to it.