And here's the thing: to a sufficiently fuzzy approximation, all the R6RS Schemes out there are interchangeable. They all offer a foreign function interface with roughly the same capabilities, they all offer the ability to launch and control subprocesses, and just about anything else can be built up from there. All of the cool things that can be built from these primitives, there is no fundamental reason they can't be portable.

So in April I started working on a set of portability packages, followed by an install tool, followed by a web-based package upload and browsing interface. And now it's all sufficiently done that I feel like I should release it.

So here you go.

Caveats

It supports Ikarus, Larceny, Mosh, Racket, and Ypsilon, at least on my machine.

The install tool could certainly use more features, like a package upgrade ability, code precompilation for the various Schemes, and complex version constraints on packages. The web interface is horrible code, and I'm just happy it seems to work as well as it does.

But what I am proud of are the simple but extensible package format, with myriad available extension points for new features, the simple repository format which allows anyone to start serving up packages if they don't like my offering, and the libraries which allow FFI and process manipulation to operate identically across Schemes.

Similar Projects

This code doesn't exist in a vacuum, and there are other projects whose scope overlaps with mine.

The most noticeable, to me, is Marco Maggi's Nausicaa, a gargantuan library distribution for R6RS Schemes. I regret to say that I have no experience using it, but the sheer number of libraries is impressive. I would like nothing more than to see some of those libraries available through UCL Install at some point.

The only other R6RS package manager that I have heard of is Dorodango. It does not seem to have been officially released yet, and if I understand the documentation correctly, it only supports Ikarus and Ypsilon at this time. It does have an upgrade command, unlike UCL Install at this point.  I envy the awesome choice of name, and I look forward to seeing what the author does with this one in the future.

Finally we come to Snowfort, an older attempt at a Scheme packaging system. I do not believe it is R6RS compatible, as it defines its own module system, and it appears to have no FFI or other portability layers, so I believe packages are limited to pure Scheme code. I would like to look at either manually or automatically porting some of these packages at some point, if the authors are amenable.

                          

How useful.  To save me a handful of quotes, I lose the ability to programmatically generate my arguments.

As I was loath to reimplement the entire library just to regain that ability, I looked for another solution.  Thankfully, I found one.

As far as I can tell, this circumvents the macro's quoting features entirely to allow me to pass in a dynamically generated symbol.  While it won't work with functions which modify the environment, it worked well in my case, and allowed me to move on to more interesting code.

                          

No SRFI-0 by default:

• mzscheme
• ikarus
• scheme48

Name exported for cond-expand:

                          

In many cases, enough functionality appears in all major implementations to make some major steps toward portability.  So a while back, I decided to get a few things out of the way (you know, processes, FFI, things like that).

To make simple, portable libraries, we need to find out some details about the Scheme implementation the code is running under.  Unfortunately, there is no standard way to do this.  We could just call a function and see if it worked, but then we throw an error, and there's no standard way to trap those in R5RS.  And even if a function of the given name existed, we still don't know what arguments it takes, because that differs too.

So we're left with some sort of indirect detection method,  guessing the implementation based on a bunch of unrelated tests.

So I gathered a list of as many nonstandardized/undefined behaviours as possible (mainly from this table, and grepping the R5RS spec for the word "undefined").  Then I filtered those differences down to a bunch of tests that won't error out.  Then I took the tests that gave meaningful results, and because I had enough of those to be picky, I took only tests which fit on a single line.

The resulting set of 20 tests generates a sort of "signature" -- a list of twenty boolean values that identifies the host Scheme.

Currently the code works on:

• MzScheme
• CHICKEN
• Guile
• Bigloo
• Gambit
• Ikarus
• Scheme48
• MIT Scheme
• Gauche

This is actually a list of all the implementations I have installed right now, and other implementations can be supported just by adding another line to the signatures list.

On Scheme implementations which provide a compiled and an interpreted mode, only interpreted has been tested (I don't really know how to use a compiled Scheme, so I didn't).  Some Schemes have different behaviour interpreted and compiled, so caution should be used there.

Enough talk, here's the code:

Obviously, only the memoized DETECT:NAME is meant for general use, since a program generally won't the see the host changing as it runs, and repeatedly calculating the name would be inefficient.  I don't actually have much experience with Scheme, so the DETECT:MATCH-SIGNATURE function could probably be written more efficiently, but it works.

Anyway, I hope that someone finds this to be useful.  I have used it to write the beginnings of a portable process library.  More on that after I get around to polishing it up a bit more.

                          

Implementations tested were:

• MzScheme
• Bigloo
• Guile
• Chicken
• Gambit
• MIT Scheme
• STklos

Bad news first.  In my hour or so of testing, I was unable to find any way to directly return values from a loaded file in Guile, Gambit, and STklos.

On the other hand, that means that four of the seven implementations that I tested could return values.

MzScheme and MIT Scheme both return the last form, whatever it was.  If the last form was a DEFINE statement, MzScheme returns nothing, and MIT Scheme returns the identifier that was defined.

Bigloo doesn't seem to return anything usable, but it does print the results of intermediate evaluations, and claims to return the result of the last evaluation.  This may indeed be true, but in my testing I was unable to get it to return anything other than the path of the just-loaded file.

Chicken, however, was an unqualified success.  Although it is true that at first it returns nothing from its LOAD function, it provides a brilliant little mechanism to fix that.  The LOAD function, in addition to taking a filename, will also take an optional argument called EVALPROC, which is called repeatedly with each form from the loaded file.  This fact, combined with a little closure magic, ends us up with this:

This function calls the LOAD function, but passes it its own evaluation procedure, which will assemble a list of values when called, as well as evaluating each expression.

After seeing the beautiful and elegant way in which Chicken Scheme allows the user to dynamically alter the behaviour of the LOAD function at runtime, I feel like I should alter my recommendation for modifying that behaviour.  Chicken has, I believe, found the best possible way to implement such functionality: have a reasonable default, but allow people to override it if they wish.  Simple, elegant, and powerful: exactly like the language itself.  Bravo, Chicken.

Bravo.

                          

Once this was in place, I proceeded to write some code to take advantage of my program, and provide rudimentary C function capabilities to the Ikarus scheme implementation.  I got most of the code done, implementing portable type conversion functions, an implementation-independent interface to encapsulate the initialization and communication code, and all the little macros necessary to make the FFI easy to use.  It had its bugs, but it worked pretty well.  Well enough, at least, to create a window using SDL, and then close it again.

And then I just sort of forgot.  Somehow, after getting it all to work and provide a basic level of C library integration to Ikarus, I just wandered off and allowed myself to get involved in other projects.

Today, I was looking through my projects folder, and happened to notice the code I had previously forgotten.  Working under the assumption that someone could probably find a use for it, I decided that I would clean it up and post it online.

It required considerably less cleaning that I had expected.

Basic Structure of the Scheme->SoLoad Link

A major design goal in this project was designing it in a way that would allow it to be easily ported to a variety of Scheme implementations and communication channels.  The design I ultimately settled on was as follows:

There is one function, SOLOAD-INIT, which handles essentially all of the necessary setup, communication, and teardown code associated with using SoLoad.  When called, it is passed the path to the SoLoad executable, and calls it.  It then returns a pair, whose CAR is a function to send data to SoLoad and return whatever data SoLoad sends back, and whose CDR is a function that will tell SoLoad to exit and perform any other teardown functions that may be necessary.

Its implementation for Ikarus looks like this:

And that concludes the implementation-dependent portion of the interface.  All of the code above this level is (hopefully) completely independent of whatever particular Scheme it is running on.

Type Conversion

The next important part of the Ikarus/SoLoad FFI is the conversion between Scheme native types, and their text-based representation.

This consists of a myriad of small little functions, to escape/unescape strings, convert numbers, and flatten lists into a single string.  They are each easy to implement, and should be entirely portable Scheme code.

All together, they run a little long, but I may as well post them here as well:

Wrapping It All Up

These functions technically provide all the functionality required to use SoLoad from within Ikarus, but they lack a little something.  What they need now is to be made easy.  It should be possible to declare a function such that it just works from anywhere in your program, with no need to know that it's calling a helper behind the scenes.

This is possibly the most involved portion of the interface, because it requires individually wrapping every function of SoLoad, and then adding a few macros to neaten things up.  In the end, though, I think it works out well:

Using It

Along with a working executable of SoLoad, this code should be all that is needed to use C libraries from within Ikarus.  I tested it roughly 5 minutes ago, so I'm fairly certain it works.  A working example, assuming all the previous code is in a file called "ikarus-soload.ss", with the soload executable (or a symlink) in the same directory, is as follows:

Now that I've taken a break from this for a while, I can see a few areas it could be made more robust.  Next feature: an optional timeout period for SoLoad, so it can close gracefully even if the caller has crashed.

                          

So this here is my attempt to share what I've learned over the past few days with regards to syntax-rules macros.  Bear in mind that at the time of writing this, I have known how to use syntax-rules since yesterday, but my knowledge seems to be complete enough to write a working module system, so here's my attempt to pass on what I've learned.

First things first.  You use define-syntax to create a top level binding, and let-syntax bears the same relationship to define-syntax as you'd expect.

That much is pretty clear from reading R5RS, now we need to figure out how to write the syntax transformer.  There are a few ways to do this, but the best way for beginners is to use syntax-rules, which avoids inadvertent variable capture and other such nasties automatically, and uses a rather elegant pattern matching language.

We'll start by trying to write a simple macro, while. This will just be a standard looping construct that keeps executing over and over until its condition becomes false.  We want its usage to look something like this:

So first, we need the define-syntax form.

Next, we need to write the syntax transformer itself.  This is where syntax-rules comes into play.  Syntax-rules uses pattern matching and text substitution to allow you to make some pretty advanced macros.  It looks like:

I will explain keywords later.  For now, just leave that bit blank.  What we're interested in are those ((<pattern>) <template>) pairs.  Each <pattern> is just that, a pattern of code that will be matched.  In our case, we want to match the pattern:

Where the '...' signifies that body may contain one or more forms. Luckily for us, this is exactly the syntax that syntax-rules wants to see, so we can just plug it in, giving us:

So far so good.  Now we just have to fill in the other half, with a suitable <template>

Before we can write the <template>, though, we have to decide what we want the code to end up looking like.  Since this isn't a guide to scheme code in general, I'll just go ahead and say that we want the output to look like:

Got that?  Okay, now we've just got to put this in our syntax-rules macro as a template.  By another startling coincidence, this is exactly what the template code is expected to look like.  We just plug in that code, and our final result is:

Just plug that into your scheme interpreter, and our while loop from earlier should execute perfectly.

Now let's try to write something a little more complicated.  We want to write a for loop similar to the one that Python has.  This should be a pretty easy task, since it's basically just syntactic sugar for scheme's map function.

Our goal is to be able to write a piece of code taking the form:

And have it expand to:

Our first try would probably look something like this:

This works, but there's one issue with it.  All of the following are valid and work exactly the same:

This is not so much of a problem in the case of a for loop, but what if you wanted to add another rule later, so that

will also work? The solution to this problem is in that <keywords> argument that we glossed over earlier.  Change the keywords list to include 'in' (and, for good measure, 'as'), and it will allow those symbols, and only those symbols, in places where they are mentioned.  This change leaves us with:

Or, for simplicity (thanks to Dan Prager for pointing this out)

And if we load this code into our scheme interpreter of choice, we should have two fully functional little bits of new syntax.

Hopefully this guide will help shed some light on the arcane subject that is the Scheme macro system, and hopefully I will never have to learn enough about syntax-case to write a tutorial on it.

                          

I really, really like scheme.

Unfortunately, it seems like a lot of the time the only way to get worthwhile things done in scheme is to abandon the conceptual purity and the elegance, all of the things that make scheme as wonderful as it is.

I figure I'm as good a person as any to try and fix at least a little of that.

The first (second actually, but more about that in the next post) step toward that is some sort of cross-implementation module system. As far as I know, the only such systems in existence so far are SLIB and Snowfort. While both of these are wonderful systems, they overshoot the mark, in my opinion. SLIB aims to be a complete all-singing all-dancing library system, and Snowfort integrates all kinds of package management with the simple act of breaking a program up into modules.

Well, that's the beauty of lisp, isn't it? If you need something, you can just code it yourself.

So I did. The preliminary result is only 19 lines of code, 6 of which are simply a function to build an alist from a key list and a value list. It can also double as a simple object system if you want to use it that way.

Its usage is simple. There are two macros, provide and module. Provide is basically the same as module, but anonymous, so it needs no documentation other than to say that you omit the name.

That's all you have to do to use it. The code inside the module system is about as simple as that too.

Just load that code into your scheme interpreter, and you have a simple module system. That's all there is to it.