Plua revisited: Lua for PalmOS (and resurrecting plua2c)

I was not an early adopter of Palm PDAs, but my experience with handheld computers is actually rather longer; my first handheld was a Tandy Pocket Computer PC-4. It had 544 free bytes (1,568 with the 1K expansion pack) and you programmed it in BASIC. Other than the TI-85 in college my next step up from there was an HP 95LX, which was the first of Hewlett-Packard's well-regarded handheld DOS machines. But Palm was the rage in the mid-to-late-1990s when I was in medical school, and I picked up a brand-new colour m505 in 2001. From there I upgraded to the Zire 72 in 2005, which I kept using until the iPhone 3GS arrived because it had better apps and it could record video. Later on I got a used AlphaSmart dana, which is the closest thing to a PalmOS laptop; it had a wide screen, full keyboard, built-in word processor, a WiFi option, two SD/SDIO slots, and USB to connect it to a printer or to have the unit act as a keyboard for a PC or Mac. Just connect it to the computer, start your favourite word processor app, and the dana would "type" your entire document into the larger computer. No drivers necessary! Although I have a few other Palm units in my collection, including an original Palm 1000, a Tungsten T|X and a couple Centro phones, plus a Pre 2 and Veer from the webOS days, I still use my Zire 72 and Dana now and then for specific tasks.

The classic Palm OS (also known as Garnet in its final revisions), not to be confused with Palm's later and technologically unrelated webOS, actually feels a lot like classic MacOS. (The "Classic" mode in webOS 1.x for running Garnet apps doesn't seem like a coincidence to me, either.) Besides the common original architecture (68K), the heavy reliance on structured resources for both applications and data storage is very reminiscent of the Mac. When ARM-based Palm OS 5 devices emerged, not only was there a 68K emulator like the Power Mac's for running older software (called PACE, the Palm Application Compatibility Environment), but the normal state of the system was to be running 68K code.

I got a lot of wear out of pre-programmed Palm apps but I'm a nerd at heart, and I like to program things. The PC-4 was easy: it was BASIC, and it had 10 segmented program spaces, so I wrote simple games and tools for school classes. The 95LX ran DOS programs, and would happily run anything I wrote in Turbo Pascal 5.5 (though optimally if formatted for the smaller screen first). However, Palm development was primarily proprietary at the time, officially requiring CodeWarrior with specific Palm support and the appropriate hardware. I used Macs (at the time, a hand-me-down Power Mac 7300), so the hardware was no problem, but I was a starving student back then and CodeWarrior wasn't cheap.

So late in 2001 it was a real boon to discover a beta Palm OS 3.1 port of the Lua programming language to Palm OS, written by Marcio Migueletto de Andrade. The part I liked best (well, other than the fact it was free!) was it was fully self-hosted, with what today we would recognize as a simple IDE, such that you could develop right on the device. In those days Plua was based on Lua 4 and offered easy graphics, serial and UI support, so I used it for writing my own internal calculation apps which (thanks to an external separate runtime, the Palm's ubiquitous IR beaming, and everyone having a Palm device) everyone on the clinical team ended up using. Eventually Plua evolved into a full-fledged 1.0 release in 2003 instead of a time-limited beta.

Plua also included a small "cross-compiler" (really, a bytecode dumper) based on luac, though with additional code to link resources as well as emit a stub PRC header to call the runtime. This allowed you to develop on a desktop PC and build the PRC there, and then HotSync it over. More about that in a moment.

Plua was already pretty great by then, but what really moved it forward was support for TCP networking in Plua 1.1. Unfortunately, networking in Plua 1.1 had several significant bugs and Marcio was already working on Plua 2.0, which was based on Lua 5, so these weren't fixed. (One of my early apps that got bitten by this was Port-A-Goph, a gopher client for Palm OS. I got a mention in Wired and the code really did exist, but the socket bugs were difficult to work around. I probably have the source code around here somewhere.) Plua 2.0 also required Palm OS 3.5 and wasn't source compatible with Plua 1.1; the functions were similar, and many function calls could be rectified with text search-and-replace, but it still had some important differences plus the jump in the core language as well. For me personally it took awhile to convert over, but Plua 2.0 was a definite improvement and the bugfixes made it a very solid package.

This screenshot, of POSE (Palm OS Emulator 3.5) running under Mac OS 9 (emulating the emulator on my Raptor Talos II), shows that Plua 2 was perfectly capable of native controls and a standard application UI. With the socket issues resolved, networking was actually stupidly easy.

At the time I was an active participant in the Yahoo! Group for Plua (now gone, along with the rest of Yahoo! Groups), which was the only official place to get Plua 2. Marcio issued an analogous "cross-compiler" for Plua 2 called, analogously, plua2c, using 5.0.3's luac as the base. However, Plua was freeware but (Lua 5 is MIT-licensed) not open source, and the plua2c binaries — which, unlike Plua 1.0 and 1.1, were distributed separately — were only available for Windows and x86 Linux.

During the Plua 2.0 betas in 2006, I privately asked Marcio if I could build a PowerPC Mac OS X-compatible version of plua2c. He agreed to this with the condition that the source be kept private (I suspect, but do not know, that he had some interest in making it a commercial product or having a commercial support option). I agreed and over the next couple years ended up issuing four binary-only releases of the Mac OS X plua2c which I hosted on Floodgap. However, after Plua 2.0 left beta around 2008, although Marcio indicated he had interest in starting on a 2.1 based on Lua 5.1, I don't know if he ever actually did; the introduction of webOS in 2009 and the lack of interest in Palm OS Cobalt or further Garnet devices essentially ended classic Palm OS's market relevance in any case. I lost contact with Marcio and never received further replies from him regarding Plua or plua2c.

Still, I kept using it for various minor projects even if I didn't regularly keep a Palm in my pocket anymore. I eventually gave up on Port-A-Goph and started on a Plua 2 rewrite (the screenshot above), and turned my Zire 72 into a Plua-powered Hue light controller:

All of this development was done on my 32-bit Power Macs, using 32-bit builds, which was all that was ever supported. In the meantime, the Plua Yahoo! Group disappeared along with the rest of everything in the Yahoo! Groups genocide and along with it the only definitive source of Plua and plua2c. Although a few people have the PRCs, no one seems to have kept the cross-compiler, and other things like documentation and examples similarly evaporated.

When I recently decided to continue work using my Raptor Talos II, which is a 64-bit POWER9, I decided I would dust off the source code of plua2c still sitting in my G5 and develop on the new machine. plua2c compiled and appeared to function but ended up generating defective executables that weren't compatible with the Plua runtime (Plua2RT). They were dramatically bloated in size and caused the runtime to emit a low-level VM error.

Recall that plua2c is descended from luac, which more or less just dumps the Lua data structures in place. Lua's documentation says that "[t]he binary files created by luac are portable to all architectures with the same word size." To this end, luac 5.0.3 actually emits sizeof(int), sizeof(size_t) and sizeof(Instruction) (i.e., the typedef quantity for the size of individual bytecode instructions) into the bytecode header which should make an amphibious loader capable of selecting different bit widths, but Plua2RT doesn't swing both ways, at least not in that respect. To make the Plua VM happy, I had to force all of these to be 32 bits in size and change the emitter to only emit 4-byte int and size_t quantities.

This partially fixed the size, but it was still abnormally enlarged, suggesting 8-byte quantities were still being injected into the file somewhere else. After some detective work I found it was actually coming from plua2c's PDB header struct, so I hardcoded the correctly sized types in its typedefs, and the length matched up and the Plua VM could now execute the generated PRC. plua2c was now ported to 64-bit OpenPOWER.

You'll notice I said the length matched what my Power Macs emitted, but not the file itself. Besides a timestamp, the Lua bytecode is emitted using the native system's endianness, and the dump also has an endianness flag to indicate what that was. Interestingly, this is one situation in which the Plua VM does swing both ways: although the native endianness of the 68K Palm OS is big, and Plua was never ARM-native (which for Palm OS 5 is little, and PACE handles the endianness switch as part of thunking), it transparently converts the values just fine, just like Lua would. The docs even say, "binary files created on a 32-bit platform (such as Intel) can be read without change in another 32-bit platform (such as Sparc [sic]), even if the byte order ('endianness') is different." In fact, it has to, because Marcio's builds of plua2c were for little-endian 32-bit x86. Only my PowerPC Mac OS X builds actually emitted big-endian data, since that was the native endianness there. My POWER9 system runs Fedora in little-endian, so the endianness didn't match the Power Macs, but that was no problem for Plua.

I intend to honour my gentleman's agreement with Marcio about not disclosing the source code. Even if he's unable or unwilling to discuss changing the arrangement, a deal is a deal and I would want this to be a sign to anyone else who would share code with me for porting purposes that I keep my promises even a decade and a half after the fact. Still, I think Plua is a great way for retrocomputing enthusiasts to get back into Palm development. Yes, there are tools like OnboardC which compile on the Palm as well, and some but not all of the C cross-compiler infrastructure works on modern 64-bit systems, but Plua is a lot more straightforward for beginners and has tons of built-in functionality that would require external libraries or a lot of additional code with other development systems. The use of a separate runtime is a little obnoxious but hardly a dealbreaker for me personally.

So, in the spirit of our original arrangement to issue PowerPC Mac OS X binaries, I have compiled plua2c for modern 64-bit platforms, at least the ones I have a compiler or cross-compiler for. Besides the 32-bit PowerPC OS X version, which I still offer, and the 64-bit OpenPOWER ppc64le Linux binary I personally use, I also compiled it for Intel macOS 10.14+ with clang and 64-bit Intel Windows with a cross-compiling MinGW gcc. I'm willing to consider other platforms if I can easily set up compilation without a lot of additional work or disk space.

But these aren't much good without Plua itself, so I've additionally started hosting the Palm OS package on Floodgap with the runtime, onboard IDE and online help, along with Marcio's documentation and license terms. I also had a complete copy of the Plua 2 examples, so I've provided those, like the animated fishtank you saw on the introductory image. And, because building them from scratch needs the PILot Resource Compiler, I also made a minor 64-bit fix to its bitmap handling for modern systems too (it's GPL, so for that you get the full source code). Fortunately, pilot-link is still readily available for most modern platforms to sync your binaries over to the device.

It is my hope that one of two things will happen: Marcio will get in touch and bless an open-source release, or, with this tool, someone(tm) can work on a clean-room implementation of the runtime and maybe fix a few of the issues like memory usage and custom screen sizes. Sadly, that someone(tm) can't be me, because I've obviously seen the source code and know at least some of how it's implemented and that makes me a tainted implementor. But combined with the source for Lua 5.0.3 — and none of what I've divulged here can't be inferred from it — folks should be able to tease apart how the VM is constructed and how calls get to the OS, because except for the Palm-specific bits the VM core is still regular old Lua. Which is why, by the way, Plua was so great and is worth resurrecting. Maybe we could even get later releases of Lua 5 working. Who knows?

Even if we don't get either of those outcomes, at least now folks interested in Palm OS have another solid homebrew development option available once again. I got a lot of wear out of Plua and Marcio's hard work is why. I don't think he ever made a cent off it, but even with its minor warts it's still my favourite way to program the classic Palm OS. Now you can enjoy it too.

The binaries, documentation and examples are at Floodgap. As for the gopher client you saw? Well, that's a future post. But when I push that out, you can compile it yourself.