Gopher on the Palm Pilot and the pitfalls of PalmOS connectivity

To start, an apology for 18 years of tardiness.

In 2004 I was working on a gopher client for my Palm m505, written in Lua using the new hotness of Plua 1.x, which supported UI, graphics and networking built-in. I christened an early implementation as "Port-A-Goph" and it even got a mention in Wired. Due to socket bugs in that version that never got fixed, I deferred the release until I could rewrite Port-A-Goph for Plua 2. Over the next few years I worked on it intermittently but got distracted by other projects, and eventually after moved to iOS and then Android I stopped carrying a Palm around with me entirely. Since I have my own Gopher client on Android, the PalmOS version sat in suspended animation.

Well, it's time to dust off the resurrected Port-A-Goph, newly christened into the Overbite client family as Overbite Palm. Along the way we'll make sure it works on a selection of real hardware:

That's my O.G. m505 on the left, a 33MHz 68K DragonBall VZ system with 8MB RAM and PalmOS 4.0, the very unit I used in medical school in 2001. The battery is shot, so like a six-month-old it loses its mind when it's removed from its cradle and my cherished homegrown clinical calculator app has long since faded from its memory, but its 16MB (how cute!) SD card still has all my old games, a new battery's on order and as long as it stays plugged in we can test with it.

The two other units are later ARM-based PalmOS 5 devices. In the middle is a Palm Centro, the last and smallest of the classic Palm phones before webOS, running PalmOS 5.4.9 which was the last version released for classic Palm hardware. Despite being a red unit, this is actually an unlocked 2G GSM Centro 685 released shortly before the Centro was EOLed (the better-known red Sprint Centro 690 phones are CDMA, locked, and have a Sprint logo silkscreened on the front). The processor is an Intel XScale PXA270 CPU (the remnant of DEC's StrongARM series, subsequently sold to Marvell, though Intel reportedly still has an ARM license) at 312MHz. I like the Centro a lot because in that tiny package it has MicroSD expansion, a keyboard, a stylus, a crappy camera, non-volatile flash memory for storing applications so you can swap batteries without losing everything (64MB, unlike the refreshed 128MB Sprints), and an additional 64MB of regular SDRAM as working memory, the most of any classic Palm device. The cherry on top is that the OS really can be mostly operated one-handed with the rocker. The only downside is that it uses an Athena connector for hot-sync and charging, plus the handful of apps that don't like NVFS (EudoraWeb, I'm looking at yoooouuuu).

On the right is a blue Palm Zire 72 running 5.2.8, which replaced my original Z72 (repurposed to control Philips hue lights) where all the blue coating peeled off (eww). It also has a 312MHz Intel XScale PXA270, but no flash storage (32MB of regular volatile RAM for apps, with 8MB reserved for the operating system, so don't let the battery die). Unlike its elder sibling the Zire 71, which had a crappy slider for the crappy camera, this is a solid block with no moving parts. Another plus is that it uses a regular mini-B USB cable instead of a cradle, though it still requires its own charger. I kept the Z72 around for awhile even after I got an iPhone 2G because the Z72 could do video. Crappy video, mind you, but better than no video.

Our other test machine is an AlphaSmart dana wireless, pretty much the closest thing (other than the doomed Palm Foleo) to a PalmOS laptop. Notionally a word processor, its wide screen didn't get wide support but, based on a modified PalmOS 4.1.2, it runs most Palm apps without comment and almost all of its built-in apps support the extra real estate. It also has a 33MHz DragonBall VZ and was first released with 8MB of RAM, but the wireless (Wi-Fi) equipped variants like this one come with a more generous 16MB. Its most interesting feature is how it uploads to a host PC or Mac: connect over USB and it "types" your document into the host computer, emulating a keyboard!

(I have two other Palm units that aren't part of the test group. I was itching to try the longer screen of my Palm T|X but it wouldn't power up and the battery pack was dangerously bulgy, so that's down for repair, and the U.S. Robotics Pilot 1000 — before they were called Palm Pilots! — in my collection is too old to run Plua.)

So, let's test-drive it. You'll need a Palm device with at least 4MB of RAM — I'll explain how we determined this near the end — and PalmOS 3.5 or later. The client is written in Plua 2.0, a superset of Lua 5.0.3, and cross-compiled using plua2c (a modified luac). Build plua2c, then get Overbite Palm from the Github project and make it (there's just one file to compile), or download the binary I provide. The source and binary are under BSD 3-clause.

To run it, load the Plua2RT runtime and MathLib onto your Palm (included in the pre-built release), then the Overbite Palm .prc (I use pilot-xfer, part of the pilot-link suite, but you can also use the old school Palm Desktop app if you have a system compatible with it). Here, we're going to use the old-but-official Palm OS Emulator (POSE) which I'm running on my POWER9 Raptor Talos II under emulation because it directly supports networking. I built a system image for it with an m515 ROM and 16MB of RAM, and then make run loads Overbite Palm into POSE.

The main menu for the Floodgap Gopher, natch, which is the default home site. The main menu is loaded into a scrolling PalmOS list control. Notice that the lines are truncated; the Wordwrap button allows you to wrap them to the listbox boundaries at the cost of extra memory. More about that in a moment.

Plua offers exactly one pull-down menu as part of its UI kit, but that's all we need. You can set the home gopher and up to three bookmarks. This pretty much uses up all the menu slots we have available, too.

The internal "about" screen, implemented as a virtual text file using a read-only text gadget, showing the PalmOS version and free memory available to Overbite Palm. This is also how Overbite Palm shows text files generally.

In broad strokes Palm devices have four kinds of heap memory: a ROM heap, a storage heap, a low memory globals heap, and a dynamic heap; the dynamic heap serves as the workspace for the current application. Because most of the available RAM in a typical Palm is allocated for apps and databases, only a small amount of memory is left over for working heap even on this 16MB emulated Palm. The actual amount is OS-dependent and fixed. The earliest Palm devices provided only a 32K dynamic heap, increased it slightly to 64K (for devices with 1MB of total RAM) in PalmOS 2 and 96K (devices over 1MB) with PalmOS 3, and either 128K (devices under 4MB) or 256K in PalmOS 3.5. Some vendors changed this at the factory (the AlphaSmart dana wireless and the Sony Clié PEG-NR70 both have 16MB of RAM and provide 512K, but this 16MB emulated m515 only provides the default 256K). Dynamic heap space could be lessened even further if the OS required it (like, say, sockets) or additional system extensions were installed and competed for it. We'll see an example of this phenomenon later on.

The 245K Plua's os.mem() function reports as free would, at least superficially, seem to be sufficient to process almost any gopher menu. However, that memory holds not just the text of the current menu or document (as a blob or parsed into tables), but all the controls Plua draws on the screen, the cached previous menu (or parsing time would really suck), the overhead of any Lua variables and tables, the stack, and Plua's own memory requirements for its interpreter — and all the other stuff that needs that memory. Because wordwrap bloats the size of the listbox (among other things), Overbite detects the low memory situation on the emulated m515 and defaults wordwrap to off.

Even as Palm devices were built with greater and greater amounts of on-board RAM, the issue of the small dynamic heap remained a chronic problem. ARM Palms provide a much larger dynamic heap allocation depending on model: for our two units here the Zire 72 has over 4MB available, and the Centro nearly 10MB. Nevertheless, even that comparative largesse still wasn't enough for some apps and tools like UDMH could grab unused storage heap and add it to the dynamic heap in a fashion transparent to most programs. Unfortunately, UDMH doesn't work on PalmOS 4.

A further limitation is that individual allocations ("chunks") are generally limited to 64K. This prevents us from using large multidimensional tables in Plua, so we just piece them apart into completely different globals for internal working storage, which thus causes them to occupy separate chunks.

Wordwrap turned on. This formats long lines into individual listbox entries so they don't wrap, at the cost of (sometimes substantially) more memory.

So how do you see an entire entry if wordwrap isn't on? Let's scroll down to this text document (marked with 0>) and double-tap it with the stylus.

The entire line is shown, along with the selector (if this were simply a regular text item line with no associated selector, it becomes an informational dialogue box only). We can opt to navigate to it or not; we choose to do so.

The menu is cached and the text file (being itemtype 0, a text file) is fetched. The wordwrap button disappears since the OS handles wrapping in text controls for us.

There is only a built-in viewer for text files, but itemtype 7 (search servers, marked with 7>) are also supported. Here, we'll tap the Back button to return to the menu, then search Veronica-2 for an appropriate query.

Overbite Palm's memory heuristics triggered here and truncated the menu, since it predicted the overhead would require more dynamic heap than there was available. I think this beats crashing and you can still see the top matches.

That's it for the basic features. As for other filetypes, Overbite Palm doesn't support downloading because the internal memory isn't really a filesystem (there are various ways to treat it somewhat like one, but these aren't really files in the typical sense). VFS support for SD and microSD cards does present a filesystem, however, and that might be a feature for another day — presumably after I implement it for Android.

So, now that we've vetted it on the emulator it's time to try it on the real thing(s). Unfortunately, the trick here is connectivity.

While Palm introduced TCP/IP support as early as PalmOS 2.0, this was furnished only via RS-232 serial or modem access (except for the Palm VII's Mobitex packet radio transceiver, which apparently no longer has service in the United States, plus the various Handspring Visor Springboard slot modules). The first Wi-Fi PalmOS device, near as I can determine, was actually the AlphaSmart dana wireless in 2003 and many PalmOS devices never had it (my T|X does, but not my Z72 or Centro). Even of those that did, whether Palm made them or not, no Palm OS Wi-Fi device supports encryption greater than WEP and I'm not downgrading my access point security just to play around with this.

But Palm did support an alternative wireless system: Bluetooth. Palm even manufactured Bluetooth SDIO cards as upgrades (we'll demonstrate one). Unfortunately, Palm devices also predate the more typical Bluetooth PAN (Personal Area Network) that most devices use nowadays; they use the now-obsolete LAP, or LAN Access Profile, so you'll need a Bluetooth access point that still supports that. Enter the Belkin F8T030.

The Belkin F8T030 is a Bluetooth access point powered by a NetSilicon NET+50 CPU (ARM7TDMI 32-bit, no MMU) running μCLinux. The original firmware only supports LAP, though a later update allows both LAP and PAN (but even with the update it's not very good at PAN, so I use it just for LAP-only devices like these). It has a web configuration panel but can also be accessed by Telnet. It also has a USB port for printer sharing, though I haven't played with that feature.

The broad difference between LAP and PAN is that LAP works essentially like a PPP connection. This meshes very well with the Palm's existing TCP/IP implementation, which supported PPP over serial from the beginning.

Both the Z72 and the Centro support LAP via Bluetooth, so we'll test those first. These two devices have already been preconfigured to talk to the access point so I'll demonstrate that process a little later with the m505, which being totally brainless will need to be configured from scratch.

Starts fine.

Selects fine.

Displays fine. To be honest, given how much capacity this device has, I would have been surprised if there were any problems. So let's load it on the Centro.

Looks good (here with wordwrap). The rocker even mostly works with it — I couldn't get it to cycle through the buttons dynamically generated by Plua, but it scrolls through the listbox fine, and pressing left to back up worked too. It also cycles fine with the buttons on dialogue boxes, and the QWERTY keyboard was very nice for searching. I think we can conclude it works great with PalmOS 5.

Next up is the Dana. This is the wireless variant so it does have Wi-Fi (not Bluetooth), but we're not doing WEP, spank you very much.

Instead, we're going to take advantage of two other attributes of the Dana: it has SD card and SDIO capability (and it has two slots).

This means we can use the P10832US Palm Bluetooth Card, which runs on PalmOS 4 devices and provides a Bluetooth 1.1 compliant connection. The card comes with a CD containing various phone drivers (we don't need these), Bluetooth applications and .prc Bluetooth drivers in an InstallShield archive, which a tool like unshield can break apart if you're not on Windows. It also works with non-SDIO devices and includes SDIO drivers for them, which we'll talk about when we try out the m505.

For some reason (not sure if this is just a flaw in my unit or a marginal battery) having the backlight on and the Bluetooth in the Dana needs extra power via USB, or else it seems to have a power sag and reset when the Bluetooth radio fires up.

Connecting Bluetooth from the Network pane in the Preferences app. The "UUNet" service is just because it works (i.e., a more or less generic ISP); this is otherwise just PPP to the Bluetooth AP. Obviously disconnect the service before removing the card, or the OS gets a little daft.

Main screen works.

Selecting an item works.

Displaying a text file works. The extra dynamic heap on the Dana gives us a bit more headroom than the emulated m515.

Let's narrow the spec even further and see if my old m505 can do this. It's got an SD card slot too, but the OS version on it lacks SDIO, so it will need to have the full driver stack loaded. (I didn't turn the backlight on because I wasn't sure I could trust the battery, so sorry about the pictures.)

I set it up as a LAN ...

... and gave it a user name and password. Let's start up Overbite Palm.

The Palm automatically establishes a connection and tries to load the Floodgap default menu ...

... and this time Plua stops it with an out of memory error after just 2K of menu data loaded. Our heuristics aren't even enough to let the app launch.

The reason is that the Bluetooth driver has overhead of its own. While the 512K of dynamic heap in the Dana wireless can take the hit, the m505 gives us just half of that. I pulled the m505 out of the cradle, let the memory clear, and started anew.

Since we know from running it in POSE on an emulated m515 that it can run with just 256K of dynamic heap, we need another connectivity method, preferably something built-in. (POSE redirects NetLib calls to the operating system using a very thin layer.) So now we're going to try raw PPP itself, which almost certainly should have a smaller memory footprint than a full BT stack. This time, however, the connection problem is physical: if the m505 cradle connected over RS-232 I could conceivably hook it up to my Raptor Talos II with a dongle and serve it PPP that way, but inconveniently (for this purpose) it's USB.

If you're lucky the USB connection itself shows up as a serial port device, but we have an even better alternative. On my iMac G4 (which I used for Palm development) is a tool called USB-TCP Bridge, part of the OSX Palm Tools package. These are for early versions of OS X, but they run fine on Tiger. As they are PowerPC only you'll either need a Power Mac or to port them yourself from source. And doesn't everyone deserve to have an iMac G4 on their desk?

On the iMac G4, I install socat (netcat should also work), enable IP forwarding with sudo sysctl -w net.inet.ip.forwarding=1, and create a basic script to run a PPP daemon (this is built into 10.4, but my Monterey MacBook Air still seems to have /usr/sbin/pppd):

#!/bin/csh -f

# csh script haters can eat me

if ($uid != 0) then
        echo 'not root'
        exit
endif

exec /usr/sbin/pppd :PALM_IP \
local ms-dns PALM_DNS netmask NETMASK \
passive noauth proxyarp notty debug \
persist nodetach asyncmap 0 ktune

This script (I saved it as ~/pppd.exec) starts pppd, giving the Palm the static address PALM_IP (don't forget the colon), DNS server PALM_DNS and netmask NETMASK, further specifying we're waiting patiently until the Palm says it's ready, we're not authenticating the Palm ahem ahem ahem, we're putting the Palm on the local network, we're telling pppd this is a socket and to handle creating a TTY by itself, we're enabling debugging, we're keeping the connection up even if the Palm seems to disconnect, we're not letting it go to the background, we're not filtering any characters, and we're allowing pppd to tune the kernel if it wishes (and it will).

I then set socat to listen on a port (as root), something like socat TCP-LISTEN:9596,bind=127.0.0.1 EXEC:/home/screwtape/pppd.exec (or the equivalent for netcat), and then configure USB-TCP Bridge. On the screen you can see it's set to "Listen on USB and connect to TCP port" and the TCP port is specified as 9596, as we gave to socat. We start the bridge, which connects to the socat port, and waits.

On the Palm side, we set up a regular PPP connection via the cradle (if it asks you the service, UUNet will suffice; timeout is your choice, and make sure the "script" has exactly one entry, namely "End"). The username and password are irrelevant because we told pppd not to authenticate (so make sure your port isn't listening on something externally routable).

The connection automatically starts when we start Overbite Palm again.

Ta-daaa! Unfortunate I can't use Bluetooth on this device like I do with the others, but it works. By this way, this method also works for the dana's USB connection, which looks like a cradle to the Mac or PC when it's not acting as a keyboard.

I think it's clear that a unit with 128K or less of dynamic heap is going to have a bad time, so this means theoretically the oldest Palm capable of running Overbite needs to have at least 4MB of RAM (using Palm's default table). The earliest unit I know of with that amount capable of OS 3.5 is the 1999 Palm IIIx, or alternatively the IIIc or IIIxe since they both came with 3.5 out of the box (the 1999 Handspring Visor Deluxe has 8MB of RAM, but is stuck at 3.1H). These units should be able to connect over PPP the same way with their serial cradles, so anybody game to try?

Anyway, 18 years isn't too long to wait for a truly great Gopher client on PalmOS, right? Overbite Palm is provided to you under the BSD 3-clause license, with all files on Github.