Monday, May 30, 2022

So long, home T1 line; hello, hacking the T1 router

Floodgap has had a T1 line since I moved into this house in 2011. I'm one of those weirdos who runs my own hardware and prefers to avoid co-lo so that I can access things whenever I want despite the additional logistical complexity, and also acts as useful immunity against acquiring other expensive hobbies. The area was boonies-ish when I moved here (big house, cheap price, bottom of the market after the housing crash), arguably still is, and for at least several years there was no DSL due to its distance from the central office (fiber? hahahaha). Cable was around, but the only cable provider at the time refused to pull a run or even quote me a price to do so despite sending contractors on three separate occasions to scope out the site. After several weeks of downtime culminating in me making a formal complaint to the Public Utilities Commission, they agreed to stop messing around and released me from the contract. Of course, by then I was down for nearly a month.

T1 lines have never been cheap, though back in the day they were prized because at 1.536 Mbit/s each way they were comparatively high capacity. At my first job out of college in 1997 the university had a T1 connected to an AdTran CSU/DSU, adding more T1 lines on and bonding them for additional bandwidth until they upgraded to optical fiber. A friend of mine in the very late 1990s had his own residential T1 (this was when consumer DSL was uncommon and 56K dialup was still frequent) that his employer paid the bill for, reportedly close to a cool grand a month in those days; he would never been able to afford it otherwise. On a cost basis alone (and certainly dollars per megabit) a T1 would have been far from my first choice, but I needed a reliable server-grade connection and I couldn't find any other alternatives at the time, so if I wanted to get my hardware back online from the house I was going to have to pay up and get one. Rather than use the actual telephone company I went with an overlay vendor and was quoted $295 a month on an annual contract for a 16-address netblock plus $199 installation. Now going into my fourth week of downtime, I signed immediately. They called the telco who came out the next week, installed the smartjack (we'll talk about what this is), took over both telephone wire pairs to the house and wired it into the pedestal — conveniently, the local telco pedestal is literally in my backyard. Good thing I'd already moved the house alarm modem to a wireless connection since I could no longer have a landline now. I then ran lines from the smartjack into the server room (thanks to the telephone guys I used to work next to when I was consulting I already had good experience with a punchdown tool), the vendor came out the week after that with the T1 router, and finally Floodgap was back up.

The original idea was to use the T1 until something less expensive came along, but the T1 just plain worked and was always highest priority on service calls, so inertia and inflation eventually turned the $295 a month into $399 and a 12-month contract into 48. Still, it was a tariffed line with a service-level agreement, I had plenty of addresses and my personal bandwidth requirements have always been modest — I don't cloud, I rarely stream and YouTube is worse than television — so I ended up just using the T1 as my personal ISP at the same time and avoiding a second bill. This worked out fine for awhile except, of course, for love. My wife needed her Netflix and her iCloud, and by then the previously intransigent cable provider had been bought by someone else (fiber? hahahaha) who didn't know any of the previous history; they came out and finally pulled an RG-6 cable run five years after the fact, and switched us on. I moved the Wi-Fi to the new cable net and her bandwidth needs were thus met in the manner to which she was accustomed. We joke about the his'n'hers networks: I still had my lab and servers on the T1, and everything else including her devices was on the cable.

Well, it was good we did that because I mentioned in January this year that the vendor (which had changed owners twice over the years) was abruptly getting out of the residential T1 business and I had a month before it was switched off. I may well have been their last customer in the region. So I'd like to publicly thank John who reached out and offered a no-strings VPN arrangement — which I'm routing over the cable — to keep Floodgap online while we consider our housing options in a market as bad as it was good when I first bought the place. We turned the VPN on and the vendor turned the T1 off. They never asked for the router back and the smartjack still sits in the back of the house.

Now it's Memorial Day in the United States and I suppose I'll have to do something about that now superfluous wiring run sometime soon. Before I do, though, let's document the T1 for a generation who may have never seen one ... and figure out something fun to do with the router they left behind other than, you know, routing stuff.

Like every computing story, this one begins with a box.

And that box can go by various names, but most commonly is known as a network interface device, or NID. It is the meeting point where my wiring meets the telco's.

Before we look at that wiring, let's also discuss in general terms how analogue telephone service is delivered. The wiring to your house is called the local loop, and consists of one or two pairs, each pair containing a pair of wires referred to as "tip" and "ring." This name came from the old operator plugboards where the tip wire was connected to the tip of the connector and the ring to the conductive section of the "sandwich" on its shaft (like a quarter-inch headphone plug). These lines carry voltage, with a potential difference of about 48 volts or more between the two, and when the phone line is not engaged these lines float; they are not grounded (except perhaps briefly for signaling purposes) to avoid picking up alternating current hum.

For plain old telephone service (POTS) and consumer asymmetric DSL (ADSL, the most common residental data connection over copper wire), a single tip/ring pair is sufficient to serve as a "telephone line." Many houses have two pairs and thus two lines available, though the second isn't always hooked up at the exchange.

T1 works over the same sort of wiring. The T-carrier system was developed in 1962 by Bell Labs as a means for digitally transmitting multiplexed phone connections, but because T-carriers are digital systems they can transmit any digital data, not just telephone calls. The T1 is the lowest capacity of the T-carriers, transmitting at 1.544 Mbit/s using a carrier signal called Digital Signal 1 (DS-1), though after removing the framing the effective throughput is 1.536 Mbit/s. This is enough for 24 simultaneous 8-bit mono phone calls sampled at 8kHz (or a very slow night of Netflix) and a call center might get a T1 purely for telephone connectivity. It requires four wires, so two pairs, which fortunately would have been possible with my house without pulling more wire runs to the pedestal. If I wanted to bond multiple T1 connections together, as many places did, I would have needed extra two-pair sets for each additional T1. The European E1 and Japanese J1 lines are similar; later additions to the T-carrier spec enabled higher capacity lines such as the T3.

In modern installations over copper wire like this one, however, T1 lines are usually provisioned using an early type of DSL called high-bit-rate DSL, or HDSL. Baseline HDSL also requires two pairs, so from a wiring perspective it has the same physical plant requirements as traditional DS-1, but it can transmit over much longer distances without repeaters and correspondingly uses higher voltages. (Later DSL technologies like symmetric DSL [SDSL] cut the requirement down to a single pair, but you couldn't also use it as a phone line; ADSL, on the other hand, can co-exist with voice frequencies on the same pair.) Despite the name HDSL is not, in fact, faster than later DSL technologies. HDSL was specifically developed to replace legacy T1 runs and is not rate-adaptive like ADSL or SDSL; that is, although you only get 1.536 Mbit/s, you always get 1.536 Mbit/s, and you always get all of it up or down. The subsequent standard HDSL2 only needed a single pair to function, but the installation here is actually HDSL4, which still uses the second pair for enhanced robustness.

Let's open the box up. You'll notice the panel directly in front of us is divided into two sides, one with jacks and accessible wiring and the other tightly closed. This is the demarcation point or DMARC, not to be confused with the later E-mail authentication scheme. The telephone company has exclusive access to the closed left side, notionally so you don't get zapped (the telco guy installing the box said the differential was about 197 volts, "and it bites a bit"), but also so you don't screw around with their work. Everything on the left is their problem. Everything on the right is yours.

The wires coming into the closed section are the two pairs from the pedestal routed through my utility closet directly behind the NID. Those get fed through the black cable between the door and the panel to the smartjack, and the smartjack then sends signals back to the (in this case) top-most RJ-45 connector. This is now considered my wiring. The cable then feeds back into the utility closet on the right side; note the blue "ring" on it because we're going to find its other end in a minute. In this NID, only the top-most jack is active; the others would be for additional T1 lines if they were present.

The smartjack lives in the door, taking HDSL signals from the telephone pairs and emitting "conventional" T1 signals on the other end. This one is an AdTran H4TU-R, an HDSL4 remote-end transceiver, festooned with its bank of status lights that are now dark since the line is down. T1 lines always bidirectionally transmit at a fixed rate even if they're just zeroes, which is useful, because the line status is thus effectively under constant monitoring. If there is an issue in the line towards the telco side, then no signals or proper framing will be received from the telco by my smartjack and it will enter "red alarm," sending a "yellow alarm" (alternating signal) back to the telco if it can and a "blue alarm" (the so-called Alarm Indication Signal, all ones) down to my equipment such that the signal is recognized as obviously abnormal yet framing is preserved and clock recovery is possible. The other lights will then tell me what the likely issue is, almost always shenanigans at the central office causing total loss of HDSL reception (one day they forgot to plug me back in after an upgrade). If there is a break in the line towards the customer side, however, such as my T1 router releasing the magic smoke and failing to transmit to the smartjack, my smartjack will send a yellow alarm to me and a blue alarm to the telco.

To test where the problem might be, the telco can attempt to remotely put the smartjack into "loopback" and reflect any signals it sends back to it. In loopback, if the alarm state halts then the loss of signal must be on my side of the DMARC (and if it were on the telco side they might not be able to send the loopback sequence anyway). The smartjack is the property of the telephone company, so if it breaks, part of the tariff agreement is they come out and fix or replace it; the RS-232 port lets the tech plug in a serial cable and query the smartjack's status on field calls.

The smartjack board itself. The card is on a hinge, so it just comes out of the door. Notice the large number of fuses for voltage protection.
On the right, next to the ground clamp, you can see the pairs coming from the telco pedestal through the conduit under the yard, up to the red splice connectors to the wires feeding the telco side of the DMARC (my actual house wiring isn't even involved here; the phone jacks inside were dead as soon as they were disconnected). To the left of the splice connectors is the other end of that blue "ringed" RJ-45 cable and the hand-punched jack which marks the beginning of my home wiring. T1 cabling has separate shielding for each pair to avoid crosstalk on long runs like the 25-ish feet from here to the server room, but if you don't have T1-rated cable you can just separate each pair into a separate Cat. 5 run, which accomplishes the same thing. That's what I did here, wiring the jack to ensure continuity on each side, and using outdoor UV-resistant cables for durability. The red tape flag tells me which pair is which.
I then widened the hole where the power conduit to the outdoor lighting exited and put a little flex tubing there to thread the two runs out of. The green wire is the grounding wire the cable wiring is using, which is separate from the grounding wire in the phone lines' grounding clamp.
The other side of the cable run in the server room terminated at the T1 router. Old T1 installations had a separate box called the CSU/DSU (channel service unit/data service unit) which was basically the modem. It took the T1 signal from the telco and emitted V.35 over big fat connectors with long pins looking like honeycombs that most routers of the time would accept directly. V.35 is very low level, effectively OSI layer 1, and what the actual data looked like varied depending on the setup (usually this was frame relay, sometimes PPP, rarely SLIP). This situation is analogous to old Ethernet implementations where the MAU and PHY were separate devices, connected by AUI.

Eventually this became similarly inconvenient and specialized T1 routers emerged that took one or more RJ-45 T1 connections and presented a WAN Ethernet interface to devices on the other end, consolidating the CSU/DSU and the router into a single unit. This particular router is a Samsung Ubigate iBG-1000 (specifically the iBG-1000A), the littlest of the Ubigate line circa 2008. Although the Ubigate is absolutely capable of frame relay and many other such protocols, my connection to the vendor (via the telco central office) was over PPP.

Despite being the family runt the iBG-1000 is hardly Best Buy consumer crap. It can bond up to four T1 lines for an aggregate of 6.144 Mbit/s, which at the time of installation was still respectable bandwidth, and was fully configurable over serial (two ports, one as a console and one auxiliary, which could be connected to a modem) and Ethernet. The Ubigate line as a whole, however, is long obsolete; Samsung hasn't sold them in years and last time a vendor tech was out at the house he mentioned the company had moved to AdTran T1 routers, though he liked them less. The vendor never asked for the equipment back and I never asked to send it to them because I suspect they don't really want it anyway.

So here's another computing story that also starts with a box. Let's open it up.

First off we immediately see an SD card and a 512MB SO-DIMM of DDR2 SDRAM at the bottom. There are several large chips here; let's try identifying them. The Lattice LCMXO1200C chip in the top middle is a generic 1200 LUT FPGA. The PMC chip above it is a COMET™ QUAD PM4354, a four-channel all-singing, all-dancing T1/J1/E1 transceiver and framer; it even independently handles alarm status without having to involve the main CPU.

The big one at the top right was harder to identify. This is a Marvell GT-96124 and it seems like it's doing a lot. I couldn't find a datasheet for that specific IC, but the GT tag is a hint it was a design from Galileo Technology, an Israeli fabless semiconductor design firm Marvell bought in 2000, and after a little digging I was able to find a datasheet for the likely related GT-96132. Does this bus diagram look familiar?

There's our four T1 ports (via the quad T1 transceiver) and our two Ethernet ports, plus two serial ports and PCI (not apparently used here) and SDRAM controllers. This is obviously an all-in-one chipset to support the CPU and drive other onboard peripherals.

And where is the CPU? Why, that's the Freescale chip underneath it, an MPC8347EA (here labeled MPC8347EVVAGDB, indicating a 400MHz part [AG] with crypto acceleration [E] on a 266MHz bus [D] mounted via a lead-free tape ball-grid array package [VV]). The 8347EA is part of the PowerQUICC II Pro family introduced in 2004 intended as systems-on-a-chip for networking applications. This chip is a 32-bit PowerPC SoC using the e300 core, more or less a hopped-up PowerPC 603e, with its own two-port Ethernet and SDRAM controllers and a single UART (it also has on-board support for two USB ports, PCI, I2C and SPI).

Given the layout of the board my guess is the CPU is actually handling the SD card (via SPI), the Ethernet ports and the SDRAM itself, and probably the console port as well so that there's direct access to its bootloader. The Marvell chip likely services the auxiliary port and maintains the HDLC interface to the T1 transceiver.

If you've read my blogs for any length of time you knew my eyes lit up the minute I realized this was a PowerPC system. Now I had to make it all mine.

The box the vendor gave me is a complete kit and I got the router new with the protective stickywrap still on it. Conveniently this included the necessary RJ-45 to DE-9 (DB-9) serial cable, labelled IBG-CCOM10 (EQ39-00004A); if you don't have or can't find this cable, the pinout is in the manual. You'll need the serial cable to get into the bootloader, which we'll have to do to get control over it (stay tuned). Connect this cable to your serial port and set your terminal program to 9600bps, 8N1. Oddly, although the cable has hardware flow control signals, I had to turn hardware flow control off in minicom to get it to work properly.
We've got tunes and we've got signal. Main screen port turn on. The first stage is the bootloader.

Copyright (c) 2008-2010 Samsung Electronics, Co.

Samsung Ubigate iBG1000 (Freescale MPC8347EA, 512M memory)
BootROM
  Version: 1.4.0
  NORMAL Boot, Compiled Jan  4 2011, 17:17:55 by build
  Gold boot update baseline ver: 1.0.1
Boot flash(Normal boot ver: 1.4.0, Gold boot ver: 1.4.0)
Mainboard CPLD ver: 7.0


Press any key to stop auto-boot...
 0
auto-booting...

[BOOTM] Loading file(/cf0/factory_default)...

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<  
!!Warning!!
 Factory default 'Boot file name' was set 'factory_default'.
 System will load a default SNOS(the latest version) from Compact Flash.
 For the proper operation, user should change exact 'Boot file name'.
Compact Flash Device: CF0, Filename: /cf0/iBG1000_Advanced_1.5.0.3.Z
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 


[BOOTM] Loading package...
 Loading [100]
[BOOTM] System image loading done
 Loading [100]
[BOOTM] Bootrom image loading done
[BOOTM] Checksum validation is checked.[OK]
[BOOTM] Package validation pass
[BOOTM] Bootrom checksum validation is checked.[OK]
[BOOTM] Checking boot image for auto update...
[BOOTM] Done : No need to update Normal boot image
[BOOTM] System image validation pass
[BOOTM] System image is decompressing
[BOOTM] System image loading on the start address

"Loading on the one"

Despite the fact the storage is really an SD card, other Ubigate hardware actually had a Compact Flash card on board and the nomenclature apparently carried over.

The manual doesn't define what SNOS is, but an educated guess would say it's Samsung Network Operating System. That loads now.

==================================================
             SAMSUNG UBIGATE SNOS
==================================================
[ATA] Starting ATA driver(S) 2.0
This System is optimized for Evil Empire T1 Vendor site.
! Set to console terminal baud rate(9600)

SAMSUNG ELECTRONICS CO., LTD.
Suwon-si, Gyeonggi-do 442-600, Korea
Copyright (c) 2008-2010
Samsung SNOS Software, iBG1000 Software

Freescale MPC8347EA processor, 512 MB memory
-> 
[INIT] Performing Level-1 Init. SUCCESS
[STDDBG] tDbgTask (0x1ffbb548) is spawned ...
[CPUMON] tCpUMon (0x1ffb8c28)  is spawned ...
[SYSLOG] tSyslogTask (0x1ffb4b98) is spawned ...
[EVMGR] tEvMgr (0x1ffa8638) is spawned ...
[INIT] Performing Level-2 Init. SUCCESS
[COMNBUF]Buffer Mgmt: Allocating default # of buffers
[COMNBUF]totalBuf = 0x03bc0      tBufs = 0x1280  xtraBufs 0x0
[COMNBUF]Calculated Total-Buffers = 15296
[COMNBUF]SYS_COMN_BUF_ALLOC_MAX= 29058
[COMNBUF]Allocating 1952 size 15296 packet buffers
[AAA] User initialization done
[HTTPD] Initialization done
[CFGMGR] attached the link-restoral timer from Config-Mgr task. 
[L2Task] Started diff-Delay-timer in context of L2-task 
[TELCO] Call spawnAllTasks.
[TELCO] Threshold task spawned. 
[TELCO] Statistics task spawned. 
[TELCO] Log task spawned. 
[TELCO] Evlogtask msgQ = 0x1805a6e0
[TELCO] Test task spawned. 
[TELCO] Update task spawned. 
[TELCO] SpawnAllTasks done.
[TELCO] Card up message sent
[TELCO] Telco & TE1 Driver Init Done
[SECURITY] Security Function Initialization started...
[SECURITY] Security Function Initialization completed...
[INIT] Performing Level-3 Init. SUCCESS
[TSEC0] pTbdBase = 0x3422000
[TSEC0] pRbdBase = 0x3422400
[TSEC1] pTbdBase = 0x3422800
[TSEC1] pRbdBase = 0x3422c00
[SNMP] AGENT INIT ENDED.
[WIF] Init Done successfully!
[INIT] Forwarding Engine Init. SUCCESS
[QOS] QoS Initialization completed
[PBR] Local PBR Init is done successfully
[VoiceDrvs] Skip open FXS/FXO device driver config file 
[INIT] Performing Level-4 Init. SUCCESS
[ISDN]: LAYER1 Register ... status(0)
[ISDN]: LAYER2 Register ... status(0)
[ISDN]: LAYER3 Register ... status(0)
[ISDN]: LAYER4 Register ... status(0)
[VQM] Initialization completed

[LICENSE] No license key is present.
[SNMP] COMMON-SUB-AGENT : MIB REGISTER ENDED 

[VoiceInit] Voice Initialization Start...
[VoiceInit] NO VoIP DSP Card Found !!!
[TASKMON] taskMon (0x1fbacc00) is spawned ... Start task monitoring
[INIT] Performing Level-5 Init. SUCCESS
[VoiceInit] Voice memory partition 0xfac72f8 - 0x12dc72f8
[VoiceInit] Voice memory partition created, size=51M, id=0xfac72a0
[VoiceInit] Voice task configuration memory initialized
[VoiceInit] Voice task shared memory initialized
[VoiceInit] Loose Timer Task initialized
[VoiceInit] Provisioning Data Handler Task initialized
[VoiceInit] Resource Monitor and Maintenance Data Handler Task initialized
[VoiceInit] Call Statics Data Handler Task initialized
[VoiceInit] SIP Entity SBC Server Task initialized
[VoiceInit] Snmp Voice Subagent Agent initialized
configuring from file '/cf0/system.cfg' ... 
 Loading system.cfg file [100%]     


Ubigate iBG1000 Software, Advanced SNOS (Basic + Security/Voice)
Version: 1.5.0 ("/home1/build/release/ucrel_1.5.0.3/src")
Complied Apr 25 2011, 11:31:19 by build

#-----------------------------------------------------------------------
# SAMSUNG ELECTRONICS CO., LTD. Login
#-----------------------------------------------------------------------

login: 

Interesting that the operating system supports ISDN and voice T1 as well (useless on the 1000, however, because there's no way to connect any telephone hardware to it).

The default login and password are samsung (both). Naturally that didn't work, because that would have been too easy. But we can see it's loading from a configuration file, and that configuration file is on the SD card (/cf0/system.cfg), so maybe we can bust in by messing around with that. I removed the tamper tape from the SD card, pulled it out — an off-the-shelf Transcend 2GB card — set the lock tab and put it in the Talos II's card reader.

Before going any further I imaged the card and put it away for safekeeping, and copied the image to a new 2GB card I got out of my parts drawer. Unsurprisingly it's just a FAT16 filesystem, with an unallocated partition most likely left for wear-leveling purposes. Most of the card is empty space:

% ls -lR
.:
total 17056
drwxr-xr-x. 2 spectre spectre    32768 Jan  6  2000  CDR
-rw-r--r--. 1 spectre spectre        8 Jan  6  2000  Certificates.dat
-rw-r--r--. 1 spectre spectre    51200 Jan  6  2000  command.log
-rw-r--r--. 1 spectre spectre 16926715 Sep 27  2017  iBG1000_Advanced_1.5.0.3.Z
-rw-r--r--. 1 spectre spectre       28 Jan  6  2000  Keys.dat
-rw-r--r--. 1 spectre spectre     3207 Jan  6  2000  oldsystem.cfg
drwxr-xr-x. 2 spectre spectre    32768 Jan  6  2000  profiles
drwxr-xr-x. 2 spectre spectre    32768 Jan  6  2000  pwp
-rw-r--r--. 1 spectre spectre      672 Jan  6  2000  shdsakey
-rw-r--r--. 1 spectre spectre      610 Jan  6  2000  shdsakey.pub
-rw-r--r--. 1 spectre spectre      887 Jan  6  2000  shrsakey
-rw-r--r--. 1 spectre spectre      230 Jan  6  2000  shrsakey.pub
-rw-r--r--. 1 spectre spectre     3216 Jan  6  2000  system.cfg
drwxr-xr-x. 2 spectre spectre    32768 Sep 28  2017 'System Volume Information'
-rw-r--r--. 1 spectre spectre     6816 Jan  6  2000  __userext__.dat
drwxr-xr-x. 2 spectre spectre    32768 Jan  6  2000  vqm

./CDR:
total 0
-rw-r--r--. 1 spectre spectre 0 Jan  6  2000 CDR_VQM.csv

./profiles:
total 0

./pwp:
total 0

'./System Volume Information':
total 32
-rw-r--r--. 1 spectre spectre 76 Sep 28  2017 IndexerVolumeGuid

./vqm:
total 0

The system configuration file is too long to print here, but suffice it to say that there was no user authentication information in it. Although the SSH1 server was enabled in that file and replacing the RSA and DSA public keys might get me in through there, I couldn't even determine exactly what IP it was listening on (the config file didn't say) and whether it could be accessed outward from the inside ports. I poked at the other files but didn't see any obvious usernames and passwords there either.

As a first attempt I deleted the configuration file and rebooted it that way to see if I was wrong. The machine made lots of complaints about that, but the login prompt remained impervious. Since we're almost certainly going to rework the configuration anyway I didn't bother restoring it.

The manual was also (probably intentionally) opaque about how to change a "forgotten password" and there was no obvious factory reset switch. The only other permanent storage was the on-board flash where the bootloader lived, so that was my next port of call. Press a key in the autoboot sequence and it will drop to the bootloader prompt.

Copyright (c) 2008-2010 Samsung Electronics, Co.

Samsung Ubigate iBG1000 (Freescale MPC8347EA, 512M memory)
BootROM
  Version: 1.4.0
  NORMAL Boot, Compiled Jan  4 2011, 17:17:55 by build
  Gold boot update baseline ver: 1.0.1
Boot flash(Normal boot ver: 1.4.0, Gold boot ver: 1.4.0)
Mainboard CPLD ver: 7.0


Press any key to stop auto-boot...
 3
[BOOT]:

No authentication required, so let's see what we've got for commands. There's occasional Engrish here; I cut and pasted this directly out of minicom.

[BOOT]: h
 ?                     - print this list
 @                     - boot (load and go)
 #                     - boot in DIAG. mode
 p                     - print boot params
 c                     - change boot params
 v                     - print boot logo with version
 o                     - show bootrom header information
 L                     - show file list in Flash 
 F                     - format the Flash 
 A                     - show simple PCI device scan result
 r                     - show current boot image(Golden/Normal)
 w                     - print status of watchdog timer
 D                     - cold reset
 u                     - change User ID and PW default set
 U                     - reset to factory 
 W                     - Set Watch Dog Time. 
 J                     - Change Vender Information.
 t                     - Run HDLC Diagnostic
 i                     - show the current console baudrate
 ! [value]             - set the console baudrate
 Checksum enable:      
  0  - check image checksum 
  1  - skip checking image checksum 
 Show header enable:   
  0  - disable showing image header contents 
  1  - enable  showing image header contents 
 Save bootrom image:   
  0  - autoupdate NORMAL boot area rom image by checking version 
  1  - save boot image on the NORMAL flash area 
  3  - No bootrom update 
 Boot image redundancy: 
  NORMAL boot image - Default boot image saved on the NORMAL boot area 
  GOLDEN boot image - Backup boot image saved on the GOLDEN boot area 
                      GOLDEN image area should not be corrupted 

Oh, I do like our chances with the u option (not U, just in case I wreck something else in the process). Let's try that.

[BOOT]: u


WARNING: All system user information will be lost!!!
1)All user account information will be reset.
2)All user lock table information will be lost.
3)The default user ID and password will be set.

Are you sure (Y or N)? y

User information is set to factory-reset.

Default User & Password can be availabe.

I love it when the default user and password are availabe. Let's bring up SNOS again and see if the default username and password are now valid.

[BOOT]: @
[BOOTM] Loading file(/cf0/factory_default)...

[...]

Ubigate iBG1000 Software, Advanced SNOS (Basic + Security/Voice)
Version: 1.5.0 ("/home1/build/release/ucrel_1.5.0.3/src")
Complied Apr 25 2011, 11:31:19 by build

 *Jan 01,2000,01:08:44 #PLATFORMn-informational: Boot up procedure completed
#-----------------------------------------------------------------------
# SAMSUNG ELECTRONICS CO., LTD. Login
#-----------------------------------------------------------------------

login: samsung
password: 

                            SAMSUNG ELECTRONICS CO., LTD. CLI
evil_empire_T1_router#  *Jan 01,2000,01:08:49 #AAAn-notification: samsung logged
?

      clear                      access clear commands
      configure                  configure from ( flash / network / terminal )
      debug                      accesses debug commands
      exit                       Exit from this session
      file                       access file commands
      password                   Change the user password 
      ping                       invoke ping
      reboot                     reboot the system
      save                       save configuration to ( local / network )
      show                       access show commands
      ssh                        OpenSSH SSH client (remote login)
      telnet                     open a telnet connection
      test                       access test commands
      trace                      trace route to destination address or host name

      <cr>                       
evil_empire_T1_router# show version

     Model: Ubigate iBG1000
      SNOS: Advanced SNOS (Basic + Security/Voice)
   Version: 1.5.0
  Bld date: Apr 25 2011, 11:31:19 (by build)
  Bld Path: "/home1/build/release/ucrel_1.5.0.3/src"
SNOS package boot version: 1.4.0 (golden boot base version: 1.0.1)
Flash normal boot version: 1.4.0 (golden boot version: 1.4.0)
Flash boot version check : No action required

Chassis info:

Slot/SubSlot  Card-Type  Status  FPGA-Rev  CPLD-Rev
---------------------------------------------------
   0/-        CBU         NORMAL    N/A      0x07  

In like Flynn in the MCP's business. Dig the Y2K-proof default date and time.

Now that we're at a command prompt, let's talk about SNOS. Every time I write something gently derogatory about any kind of tech there's always someone to pop up and say I'm wrong, so I look forward to hearing from the three people who have used SNOS and like it in the comments. For everyone else, if you've used Cisco IOS on Cisco router hardware, SNOS is sort of the shabbier bootleg grey copyright cousin, like those Mackey Moose cartoons we know get sold on DVD-Rs from truck liftgates: very similar, as exact as it can be without being a blatant ripoff, but with enough subtle and sometimes obvious differences to land squarely in the network engineer uncanny valley.

I should also note that vendors could customize SNOS with their own add-ins or remove modules that were irrelevant to their business line. This is what my vendor seems to have done; their name, censored so I don't get in trouble, even appears in the boot messages. It's very possible there is differing functionality between this build of SNOS and the stock version, so just keep this in mind if you're using these steps for the Ubigate you found at the electronics recycler.

I'll discuss more about the architecture of SNOS when we look at monkeypatching it. For now, let's get this up on the network so I don't need to keep it plugged into the serial port. SNOS, like IOS, is a modal CLI: certain sets of commands are only allowed in certain modules which you must explicitly enter and exit, and the kernel keeps track which module a user is presently in so that two people logged into the device will be (largely) prevented from issuing dueling commands. That should give you enough context to understand the next set of entries.

evil_empire_T1_router# configure terminal
evil_empire_T1_router/configure# hostname george
george/configure# keymap Delete delete-left-char
george/configure# system logging
george/configure/system/logging# no logging_on
george/configure/system/logging# exit
george/configure# interface ethernet 0/0
george/configure/interface/ethernet (0/0)# ip address X.X.X.X Y.Y.Y.Y
george/configure/interface/ethernet (0/0)# exit ethernet
george/configure# ip domain-lookup
george/configure# ip pname_server Z.Z.Z.Z
george/configure# ip domain_name floodgap.com
george/configure# sntp server ntp.floodgap.com
george/configure# telnet_server
george/configure# ftp_server
george/configure# ip http server
george/configure# exit
george# save local

WARNING :
Do not reboot during this process.
Saving current system configuration.
Please wait... (up to a minute)
2382 bytes written
Done.
george# 

I named it George after the sales exec who sold me the line back in 2011; I have no idea if he's even still with the company in its current form. Although the IP addresses are censored so you stay out of my network backyard, you little script kiddie, these commands should be relatively self-explanatory.

In detail, I first enter the configuration module with configure terminal (versus configure file or configure network), which says I'll enter configuration information manually and locks the configuration session to me so another logged in administrator can't change it simultaneously. I then set the hostname, set the delete key to be sane, turned off logging (or it gets really spammy with the T1 ports disconnected), set ethernet port 0 to the correct address (X) and netmask (Y), enabled the DNS client, set the primary name server, set the domain name, set the NTP server (yay! clock!), and then enabled the Telnet, FTP and Web servers. (I left off the SSH server since it's not much good to me in its current form being merely SSH1, and george is currently only connected to the internal non-routable network anyway. There's an HTTPS server also, but it too is almost certainly fatally deficient by modern cryptographic standards.)

The final exit from configuration mode commits the changes to the current working state; save local then writes a clean, new /cf0/system.cfg with those settings to the card for next bootup. Hello, george, welcome to Floodgap.

I rebooted it (a power cycle will do) to verify it all stuck and logged in over Telnet this time, which is much faster than 9600bps serial. Obnoxiously the only one of these settings that does not properly persist is the delete key setting; it's properly written to the configuration file, but doesn't seem to be honoured by the Telnet server as the default. Either type it manually when you log in or use something like expect on top of telnet to remap ^? to ^H.

What else can we get into? There aren't many commands otherwise because this is supposed to be a router, darn it, and no part of the visible filesystem obviously contains those commands (more to say about that later), so I couldn't see a way to upload executables and have SNOS run them. The SSH1 client, as expected, failed to connect to any of the servers locally since they're all SSH2. The file module lets you manipulate the SD card from within the CLI interface, ping, trace and telnet do what you'd expect, and the test module just lets you run tests on the T1 line which, of course, doesn't do me any good anymore.

Let's start with the FTP server. Interestingly, the FTP credentials are separately maintained.

george# show ftp
FTP Setting:
--------------------------------------
      FTP Server:     Enabled
        (File list format: iBG private)

Allowed FTP Client:
--------------------------------------
        Username:       admin
        Password:       admin

Much like the Telnet server, the FTP server is a decent, unadorned, standards-compliant FTP server. Files go to and from the SD card. You could easily use it as a drop-box in this configuration.

% ftp george
Connected to george (X.X.X.X).
220 FTP server ready.
Name (george:spectre): admin
331 Password required for admin.
Password:
230 User admin logged in.
Remote system type is iBG:.
ftp> quote HELP
214- The following commands are recognized (* =>'s unimplemented).
   USER EPRT STRU REST CWD SYST XMKD CDUP 
   PASS PASV MODE RNFR XCWD STAT RMD XCUP 
   QUIT LPSV RETR RNTO LIST HELP XRMD STOU 
   PORT EPSV STOR ABOR NLST NOOP PWD SIZE 
   LPRT TYPE APPE DELE SITE MKD XPWD MDTM 
214 Direct comments to ftp-bugs@george.
ftp> quote SITE HELP
214- The following SITE commands are recognized (* =>'s unimplemented).
   IDLE HELP 
214 Direct comments to ftp-bugs@george.
ftp> quote SYST
215 iBG: Version: 1.5.0.3
ftp> quit
221 Goodbye.

That's pretty much all I have to say about the FTP server and the Telnet server.

The web server, on the other hand, is a stranger beast. By default, you get this error page.

% curl -v http://george/
*   Trying X.X.X.X:80...
* Connected to george (X.X.X.X) port 80 (#0)
> GET / HTTP/1.1
> Host: george
> User-Agent: curl/Q.QQ.Q
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: SNOS-HTTP-SERVER
< Connection: close
< Content-Type: text/html
< 
<H1>SNOS-HTTP-SERVER Error Report:</H1><HR>
<H2>Server Error: 404 Not Found</H2><HR>
<H2>File not found</H2><HR>
<H2>/start.html</H2><HR>
<H3> SNOS-HTTP-SERVER </H3><HR>
* Closing connection 0
Fair enough: we know the FTP server works, so I wrote up my customary tiny "hello world" page and uploaded it to start.html on the CF SD card.

% curl http://george/
Mortimer, we're back!
Now this leads to an interesting question. Is everything accessible on the card? Let's try for the brass ring right now:

% curl http://george/system.cfg
<H1>SNOS-HTTP-SERVER Error Report:</H1><HR>
<H2>Server Error: 404 Not Found</H2><HR>
<H2>File not found</H2><HR>
<H2>/system.cfg</H2><HR>
<H3> SNOS-HTTP-SERVER </H3><HR>

That's a relief, but it also means it won't serve arbitrary content. The exact filenames it will serve must be in the firmware. Let's look at that now.

Going back to our list of files on the card, iBG1000_Advanced_1.5.0.3.Z is obviously the boot image. At nearly 17MB pretty much everything is in it. You would think that the .Z suffix on the end implies it was shrunk with compress, but both uncompress and gunzip object to the file and won't process it. On the other hand, running strings on it yields nothing but garbage, so it is almost certainly compressed with something. The total length of this file is 16,926,715 bytes; remember this for the offsets we'll be using.

Let's start off by seeing what binwalk makes of it. It immediately determines the firmware is gzip-compressed with a separate 52-byte header, which explains gunzip's objections:

% binwalk iBG1000_Advanced_1.5.0.3.Z 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
52            0x34            gzip compressed data, from Unix, last modified: 2011-04-25 02:32:24
16048242      0xF4E072        MySQL MISAM compressed data file Version 5
16307655      0xF8D5C7        Copyright string: "Copyright 1984-2006 Wind River Systems, Inc.8`"
16326587      0xF91FBB        VxWorks operating system version "5.5.1" , compiled: "Jan  4 2011, 17:17:58"
16327012      0xF92164        Zlib compressed data, default compression

It also found two interesting strings indicating that SNOS is built on VxWorks 5.5.1, which is not at all surprising; it's a very common kernel for real-time applications like this, especially on PowerPC.

However, it would still take some trial and error to determine any subcomponents in the file as well as the exact meaning of the header, and I'm quite sure there's not really an ISAM database in it (probably). Fortunately it turns out the bootloader itself can tell us what's there. While I'm at it I'm going to hard-set the boot filename so we know exactly what's getting loaded.

[BOOT]: c
Boot dev  [ftp,active-ftp,cf0]: cf0 
Boot file name            : factory_default iBG1000_Advanced_1.5.0.3.Z
Server name               : host 
Server IP address         : 192.168.1.10 
My IP address             : 192.168.1.11 
My subnet mask            : 
Gateway IP address        : 
User name                 : demo 
Password                  : 
Checksum enable     [0:Check  ,1:Skip  ]: 0 
Show header enable  [0:Disalbe,1:Enable]: 0 1
Save bootrom image  [0:AutoUpdate,1:NormalBTupd,3:NoUpd]: 0 
Golden boot autosync [0: disable, 1: enable]: 0 
Save Diag Flag  [0:Normal mode,1:Diag mode]: 0 
Rollback SNOS [/cf0/file,null]: 

Other than setting the filename, the major change I made here was to enable showing the header on the file as it loads, i.e., that 52 bytes at the beginning that binwalk identified. We'll now boot the image and see what it displays.

[BOOT]: @

[BOOTM] Loading file(/cf0/iBG1000_Advanced_1.5.0.3.Z)...

* File Header *
version      : 01.05.00.03
no.of images : 2
packaging    : Advanced
model        : 1000
size         : 16926715 (0x10247fb)
checksum     : 0xa8a6d921
hdr checksum : 0x897a394f

* Image Header [0] *
version      : 01.05.00.03
type         : System
size         : 16307591 (0xf8d587)
checksum     : 0x8f385188

* Image Header [1] *
version      : 01.04.00.00
type         : Booting Rom
size         : 619072 (0x97240)
checksum     : 0x1b81b7d6

[BOOTM] Loading package...

This indicates there are exactly three pieces to the file, namely the header, the image and the boot ROM (the on-board flash). Let's look at the first 52 bytes in a hex editor.

00000000  01 05 00 03 
          02 
          01 
          03 e8  
          01 02 47 fb 
          a8 a6 d9 21  |..........G....!|
00000010  89 7a 39 4f 

          01 05 00 03 
          02 52 00 00 
          00 f8 d5 87  |.z9O.....R......|
00000020  8f 38 51 88 

          01 04 00 00  
          01 00 00 00 
          00 09 72 40  |.8Q...........r@|
00000030  1b 81 b7 d6 

This is 32-bit PowerPC, so all values are big-endian in the fashion G-d Himself intended humanity to think. In order, we see the version, number of images, the packaging type (1), the value 1000 (for the model) as a 16-bit short, and as 32-bit ints the total file size and the checksum. The two other entries should be similarly self-explanatory except for their type codes (0x02520000 and 0x01000000), which look like otherwise opaque 32-bit ints also. 52 + 16307591 + 619072 = 16926715, meaning all bytes are accounted for in the header, so we now have enough information to break the file apart.

% cat >break.pl 
#!/usr/bin/perl


read(STDIN, $buf, 52); # header
read(STDIN, $buf, 16307591); # sysimg
open(X, ">sys.gz") && print(X $buf) && close(X);
read(STDIN, $buf, 619072); # bootrom
open(X, ">boot.rom") && print(X $buf) && close(X);
^D
% perl break.pl < iBG1000_Advanced_1.5.0.3.Z
% ls -l
total 33072
-rw-rw-r--. 1 spectre spectre   619072 May 27 21:51 boot.rom
-rw-rw-r--. 1 spectre spectre      226 May 27 21:51 break.pl
-rw-rw-r--. 1 spectre spectre 16926715 May 27 21:45 iBG1000_Advanced_1.5.0.3.Z
-rw-rw-r--. 1 spectre spectre 16307591 May 27 21:51 sys.gz

It's the SNOS image we're most interested in, since we have no need right now to futz around with the bootloader. binwalk says it's gzipped, so let's try that.

% gunzip -c < sys.gz > sys
% file sys
sys: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), statically linked, not stripped

Ta-daa. It's actually one huge single ELF executable, probably loaded into real memory even though the MPC8347EA does have an MMU. And strings yields a lot of interesting things we might talk about in a future entry, including the list of filenames that the webserver will serve. Note that .htm and .html are not treated synonymously:

/start.htm         
/start.html        
/login.htm         
/login.html
/errlgn.htm        
/errlogin.htm

And that's it. CGI? Hahahaha.

For our last stupid pet trick, let's monkeypatch the firmware. Being one big executable all commands are embedded within it, and it should be possible to compile in new, arbitrary functionality, link it into the monolith and then modify some victim code somewhere else to call it. For the moment I don't know enough about VxWorks to do that currently and I probably should find something more productive to do with the rest of the weekend, so as proof of pwnership we'll settle for modifying strings in place. SAMSUNG and SPECTRE are both seven letters long, so I loaded the blob into hexer and changed it everywhere I saw it in the file.

The next step is to figure out how the checksums work to stream this back in. Yes, you did indeed see that the bootloader can be told to ignore checksums, but it's always better to maintain safeguards when you're this close to the metal; I wouldn't want to brick it on the eve I finally got it open. Running the pieces through crc32 did not match the checksums in the header, but by dumb luck cksum did, once you convert its hashes to hexadecimal. You should be able to find these values in the header, proving they check out.

% cksum sys.gz
2402832776 16307591 sys.gz
% cksum boot.rom
461486038 619072 boot.rom
% perl -e 'printf("%08x %08x\n", 2402832776, 461486038)'
8f385188 1b81b7d6
% cat sys.gz boot.rom | cksum | perl -ae 'printf("%08x\n", $F[0])'
a8a6d921

How about the checksum for the header itself? This is a tougher call because the header has its own checksum embedded in it, a classic chicken-and-egg problem. Some schemes require a specific placeholder value to be placed where the checksum would go and you checksum the data like that. It may be possible to brute force the starting value since we know the desired result, but a good initial guess is always to put zeroes where the checksum would go:

% perl -e 'undef $/;$_=<STDIN>;1while(s/[\r\l\n\s]+//);print(pack("H*",$_))' > x.out
  01 05 00 03 02 01 03 e8  01 02 47 fb a8 a6 d9 21  
00 00 00 00 01 05 00 03  02 52 00 00 00 f8 d5 87  
  8f 38 51 88 01 04 00 00  01 00 00 00 00 09 72 40  
  1b 81 b7 d6 
% cksum x.out
2306488655 52 x.out
% perl -e 'printf("%08x\n",2306488655)'
897a394f

I'd rather be lucky than good as that is indeed the correct method. Now we can assemble the replacement firmware: compute the checksum and length of the new compressed SNOS image, compute the checksum of the compressed SNOS image concatenated with the boot ROM (we didn't change the boot ROM, so that has the same length and checksum), create the header checksum, create the header with the checksum, and finally concatenate everything together and upload it to the SD card. (Whew!)

% gzip -c < sys > sys.gz
% cksum sys.gz | perl -ae 'printf("%08x %08x %d\n", $F[0], $F[1], $F[1])'
8e6adee9 00f8d584 16307588
% perl -e 'printf("%08x\n", 16307588+52+619072)'
010247f8
% cat sys.gz boot.rom | cksum | perl -ae 'printf("%08x\n", $F[0])'
4e8012dc
% perl -e 'undef $/;$_=<STDIN>;1while(s/[\r\l\n\s]+//);print(pack("H*",$_))' > x.out
01 05 00 03
02
01
03 e8
010247f8
4e8012dc
00000000
01 05 00 03
02 52 00 00
00f8d584
8e6adee9
01 04 00 00
01 00 00 00
00 09 72 40
1b 81 b7 d6
% cksum x.out | perl -ae 'printf("%08x\n", $F[0])'
57f4f58e
% perl -e 'undef $/;$_=<STDIN>;1while(s/[\r\l\n\s]+//);print(pack("H*",$_))' > x.out
01 05 00 03
02
01
03 e8
010247f8
4e8012dc
57f4f58e
01 05 00 03
02 52 00 00
00f8d584
8e6adee9
01 04 00 00
01 00 00 00
00 09 72 40
1b 81 b7 d6
% ls -l x.out
-rw-rw-r--. 1 spectre spectre 52 May 27 22:21 x.out
% cat x.out sys.gz boot.rom > iBG1000_Advanced_1.5.0.3.Z

Rebooting!

Copyright (c) 2008-2010 Samsung Electronics, Co.

Samsung Ubigate iBG1000 (Freescale MPC8347EA, 512M memory)
BootROM
  Version: 1.4.0
  NORMAL Boot, Compiled Jan  4 2011, 17:17:55 by build
  Gold boot update baseline ver: 1.0.1
Boot flash(Normal boot ver: 1.4.0, Gold boot ver: 1.4.0)
Mainboard CPLD ver: 7.0

$
Press any key to stop auto-boot...
 0
auto-booting...

[BOOTM] Loading file(/cf0/iBG1000_Advanced_1.5.0.3.Z)...

* File Header *
version      : 01.05.00.03
no.of images : 2
packaging    : Advanced
model        : 1000
size         : 16926712 (0x10247f8)
checksum     : 0x4e8012dc
hdr checksum : 0x57f4f58e

* Image Header [0] *
version      : 01.05.00.03
type         : System
size         : 16307588 (0xf8d584)
checksum     : 0x8e6adee9

* Image Header [1] *
version      : 01.04.00.00
type         : Booting Rom
size         : 619072 (0x97240)
checksum     : 0x1b81b7d6

[BOOTM] Loading package...
 Loading [100]
[BOOTM] System image loading done
 Loading [100]
[BOOTM] Bootrom image loading done
[BOOTM] Package validation pass
[BOOTM] Bootrom checksum validation is checked.[OK]
[BOOTM] Checking boot image for auto update...
[BOOTM] Done : No need to update Normal boot image
[BOOTM] System image validation pass
[BOOTM] System image is decompressing
[BOOTM] System image loading on the start address

==================================================
             SPECTRE UBIGATE SNOS
==================================================

For all those in-place changes this was the only spot where I saw SPECTRE actually visibly appear, but this is good enough to prove that the change took, the checksums verified and the new binary was accepted. If I have another lazy weekend I might look at the symbols in that executable, write up some headers and try to link in a hello-world module without blowing anything up. Linux? NetBSD? Well, it does have an MMU, kinda-sorta mass storage and sufficient RAM, so it should be possible.

In the meantime, the least the vendor could do after 10 years of fees and a precipitous exit from the market is not ask for the hardware back now after I finally managed to get into it. I mean, c'mon, at least leave me my boxy toys for such sunny holiday weekends. To do otherwise would not be a great ending to these two nerdy stories.

Wednesday, May 4, 2022

Gopher on the Palm Pilot and the pitfalls of PalmOS connectivity

To start, an apology for 18 years of tardiness.

In 2004 I was working on a gopher client for my Palm m505, written in Lua using the new hotness of Plua 1.x, which supported UI, graphics and networking built-in. I christened an early implementation as "Port-A-Goph" and it even got a mention in Wired. Due to socket bugs in that version that never got fixed, I deferred the release until I could rewrite Port-A-Goph for Plua 2. Over the next few years I worked on it intermittently but got distracted by other projects, and eventually after moved to iOS and then Android I stopped carrying a Palm around with me entirely. Since I have my own Gopher client on Android, the PalmOS version sat in suspended animation.

Well, it's time to dust off the resurrected Port-A-Goph, newly christened into the Overbite client family as Overbite Palm. Along the way we'll make sure it works on a selection of real hardware:

That's my O.G. m505 on the left, a 33MHz 68K DragonBall VZ system with 8MB RAM and PalmOS 4.0, the very unit I used in medical school in 2001. The battery is shot, so like a six-month-old it loses its mind when it's removed from its cradle and my cherished homegrown clinical calculator app has long since faded from its memory, but its 16MB (how cute!) SD card still has all my old games, a new battery's on order and as long as it stays plugged in we can test with it.

The two other units are later ARM-based PalmOS 5 devices. In the middle is a Palm Centro, the last and smallest of the classic Palm phones before webOS, running PalmOS 5.4.9 which was the last version released for classic Palm hardware. Despite being a red unit, this is actually an unlocked 2G GSM Centro 685 released shortly before the Centro was EOLed (the better-known red Sprint Centro 690 phones are CDMA, locked, and have a Sprint logo silkscreened on the front). The processor is an Intel XScale PXA270 CPU (the remnant of DEC's StrongARM series, subsequently sold to Marvell, though Intel reportedly still has an ARM license) at 312MHz. I like the Centro a lot because in that tiny package it has MicroSD expansion, a keyboard, a stylus, a crappy camera, non-volatile flash memory for storing applications so you can swap batteries without losing everything (64MB, unlike the refreshed 128MB Sprints), and an additional 64MB of regular SDRAM as working memory, the most of any classic Palm device. The cherry on top is that the OS really can be mostly operated one-handed with the rocker. The only downside is that it uses an Athena connector for hot-sync and charging, plus the handful of apps that don't like NVFS (EudoraWeb, I'm looking at yoooouuuu).

On the right is a blue Palm Zire 72 running 5.2.8, which replaced my original Z72 (repurposed to control Philips hue lights) where all the blue coating peeled off (eww). It also has a 312MHz Intel XScale PXA270, but no flash storage (32MB of regular volatile RAM for apps, with 8MB reserved for the operating system, so don't let the battery die). Unlike its elder sibling the Zire 71, which had a crappy slider for the crappy camera, this is a solid block with no moving parts. Another plus is that it uses a regular mini-B USB cable instead of a cradle, though it still requires its own charger. I kept the Z72 around for awhile even after I got an iPhone 2G because the Z72 could do video. Crappy video, mind you, but better than no video.

Our other test machine is an AlphaSmart dana wireless, pretty much the closest thing (other than the doomed Palm Foleo) to a PalmOS laptop. Notionally a word processor, its wide screen didn't get wide support but, based on a modified PalmOS 4.1.2, it runs most Palm apps without comment and almost all of its built-in apps support the extra real estate. It also has a 33MHz DragonBall VZ and was first released with 8MB of RAM, but the wireless (Wi-Fi) equipped variants like this one come with a more generous 16MB. Its most interesting feature is how it uploads to a host PC or Mac: connect over USB and it "types" your document into the host computer, emulating a keyboard!

(I have two other Palm units that aren't part of the test group. I was itching to try the longer screen of my Palm T|X but it wouldn't power up and the battery pack was dangerously bulgy, so that's down for repair, and the U.S. Robotics Pilot 1000 — before they were called Palm Pilots! — in my collection is too old to run Plua.)

So, let's test-drive it. You'll need a Palm device with at least 4MB of RAM — I'll explain how we determined this near the end — and PalmOS 3.5 or later. The client is written in Plua 2.0, a superset of Lua 5.0.3, and cross-compiled using plua2c (a modified luac). Build plua2c, then get Overbite Palm from the Github project and make it (there's just one file to compile), or download the binary I provide. The source and binary are under BSD 3-clause.

To run it, load the Plua2RT runtime and MathLib onto your Palm (included in the pre-built release), then the Overbite Palm .prc (I use pilot-xfer, part of the pilot-link suite, but you can also use the old school Palm Desktop app if you have a system compatible with it). Here, we're going to use the old-but-official Palm OS Emulator (POSE) which I'm running on my POWER9 Raptor Talos II under emulation because it directly supports networking. I built a system image for it with an m515 ROM and 16MB of RAM, and then make run loads Overbite Palm into POSE.

The main menu for the Floodgap Gopher, natch, which is the default home site. The main menu is loaded into a scrolling PalmOS list control. Notice that the lines are truncated; the Wordwrap button allows you to wrap them to the listbox boundaries at the cost of extra memory. More about that in a moment.
Plua offers exactly one pull-down menu as part of its UI kit, but that's all we need. You can set the home gopher and up to three bookmarks. This pretty much uses up all the menu slots we have available, too.
The internal "about" screen, implemented as a virtual text file using a read-only text gadget, showing the PalmOS version and free memory available to Overbite Palm. This is also how Overbite Palm shows text files generally.

In broad strokes Palm devices have four kinds of heap memory: a ROM heap, a storage heap, a low memory globals heap, and a dynamic heap; the dynamic heap serves as the workspace for the current application. Because most of the available RAM in a typical Palm is allocated for apps and databases, only a small amount of memory is left over for working heap even on this 16MB emulated Palm. The actual amount is OS-dependent and fixed. The earliest Palm devices provided only a 32K dynamic heap, increased it slightly to 64K (for devices with 1MB of total RAM) in PalmOS 2 and 96K (devices over 1MB) with PalmOS 3, and either 128K (devices under 4MB) or 256K in PalmOS 3.5. Some vendors changed this at the factory (the AlphaSmart dana wireless and the Sony Clié PEG-NR70 both have 16MB of RAM and provide 512K, but this 16MB emulated m515 only provides the default 256K). Dynamic heap space could be lessened even further if the OS required it (like, say, sockets) or additional system extensions were installed and competed for it. We'll see an example of this phenomenon later on.

The 245K Plua's os.mem() function reports as free would, at least superficially, seem to be sufficient to process almost any gopher menu. However, that memory holds not just the text of the current menu or document (as a blob or parsed into tables), but all the controls Plua draws on the screen, the cached previous menu (or parsing time would really suck), the overhead of any Lua variables and tables, the stack, and Plua's own memory requirements for its interpreter — and all the other stuff that needs that memory. Because wordwrap bloats the size of the listbox (among other things), Overbite detects the low memory situation on the emulated m515 and defaults wordwrap to off.

Even as Palm devices were built with greater and greater amounts of on-board RAM, the issue of the small dynamic heap remained a chronic problem. ARM Palms provide a much larger dynamic heap allocation depending on model: for our two units here the Zire 72 has over 4MB available, and the Centro nearly 10MB. Nevertheless, even that comparative largesse still wasn't enough for some apps and tools like UDMH could grab unused storage heap and add it to the dynamic heap in a fashion transparent to most programs. Unfortunately, UDMH doesn't work on PalmOS 4.

A further limitation is that individual allocations ("chunks") are generally limited to 64K. This prevents us from using large multidimensional tables in Plua, so we just piece them apart into completely different globals for internal working storage, which thus causes them to occupy separate chunks.

Wordwrap turned on. This formats long lines into individual listbox entries so they don't wrap, at the cost of (sometimes substantially) more memory.
So how do you see an entire entry if wordwrap isn't on? Let's scroll down to this text document (marked with 0>) and double-tap it with the stylus.
The entire line is shown, along with the selector (if this were simply a regular text item line with no associated selector, it becomes an informational dialogue box only). We can opt to navigate to it or not; we choose to do so.
The menu is cached and the text file (being itemtype 0, a text file) is fetched. The wordwrap button disappears since the OS handles wrapping in text controls for us.
There is only a built-in viewer for text files, but itemtype 7 (search servers, marked with 7>) are also supported. Here, we'll tap the Back button to return to the menu, then search Veronica-2 for an appropriate query.
Overbite Palm's memory heuristics triggered here and truncated the menu, since it predicted the overhead would require more dynamic heap than there was available. I think this beats crashing and you can still see the top matches.

That's it for the basic features. As for other filetypes, Overbite Palm doesn't support downloading because the internal memory isn't really a filesystem (there are various ways to treat it somewhat like one, but these aren't really files in the typical sense). VFS support for SD and microSD cards does present a filesystem, however, and that might be a feature for another day — presumably after I implement it for Android.

So, now that we've vetted it on the emulator it's time to try it on the real thing(s). Unfortunately, the trick here is connectivity.

While Palm introduced TCP/IP support as early as PalmOS 2.0, this was furnished only via RS-232 serial or modem access (except for the Palm VII's Mobitex packet radio transceiver, which apparently no longer has service in the United States, plus the various Handspring Visor Springboard slot modules). The first Wi-Fi PalmOS device, near as I can determine, was actually the AlphaSmart dana wireless in 2003 and many PalmOS devices never had it (my T|X does, but not my Z72 or Centro). Even of those that did, whether Palm made them or not, no Palm OS Wi-Fi device supports encryption greater than WEP and I'm not downgrading my access point security just to play around with this.

But Palm did support an alternative wireless system: Bluetooth. Palm even manufactured Bluetooth SDIO cards as upgrades (we'll demonstrate one). Unfortunately, Palm devices also predate the more typical Bluetooth PAN (Personal Area Network) that most devices use nowadays; they use the now-obsolete LAP, or LAN Access Profile, so you'll need a Bluetooth access point that still supports that. Enter the Belkin F8T030.

The Belkin F8T030 is a Bluetooth access point powered by a NetSilicon NET+50 CPU (ARM7TDMI 32-bit, no MMU) running μCLinux. The original firmware only supports LAP, though a later update allows both LAP and PAN (but even with the update it's not very good at PAN, so I use it just for LAP-only devices like these). It has a web configuration panel but can also be accessed by Telnet. It also has a USB port for printer sharing, though I haven't played with that feature.

The broad difference between LAP and PAN is that LAP works essentially like a PPP connection. This meshes very well with the Palm's existing TCP/IP implementation, which supported PPP over serial from the beginning.

Both the Z72 and the Centro support LAP via Bluetooth, so we'll test those first. These two devices have already been preconfigured to talk to the access point so I'll demonstrate that process a little later with the m505, which being totally brainless will need to be configured from scratch.

Starts fine.
Selects fine.
Displays fine. To be honest, given how much capacity this device has, I would have been surprised if there were any problems. So let's load it on the Centro.
Looks good (here with wordwrap). The rocker even mostly works with it — I couldn't get it to cycle through the buttons dynamically generated by Plua, but it scrolls through the listbox fine, and pressing left to back up worked too. It also cycles fine with the buttons on dialogue boxes, and the QWERTY keyboard was very nice for searching. I think we can conclude it works great with PalmOS 5.

Next up is the Dana. This is the wireless variant so it does have Wi-Fi (not Bluetooth), but we're not doing WEP, spank you very much.

Instead, we're going to take advantage of two other attributes of the Dana: it has SD card and SDIO capability (and it has two slots).
This means we can use the P10832US Palm Bluetooth Card, which runs on PalmOS 4 devices and provides a Bluetooth 1.1 compliant connection. The card comes with a CD containing various phone drivers (we don't need these), Bluetooth applications and .prc Bluetooth drivers in an InstallShield archive, which a tool like unshield can break apart if you're not on Windows. It also works with non-SDIO devices and includes SDIO drivers for them, which we'll talk about when we try out the m505.

For some reason (not sure if this is just a flaw in my unit or a marginal battery) having the backlight on and the Bluetooth in the Dana needs extra power via USB, or else it seems to have a power sag and reset when the Bluetooth radio fires up.

Connecting Bluetooth from the Network pane in the Preferences app. The "UUNet" service is just because it works (i.e., a more or less generic ISP); this is otherwise just PPP to the Bluetooth AP. Obviously disconnect the service before removing the card, or the OS gets a little daft.
Main screen works.
Selecting an item works.
Displaying a text file works. The extra dynamic heap on the Dana gives us a bit more headroom than the emulated m515.

Let's narrow the spec even further and see if my old m505 can do this. It's got an SD card slot too, but the OS version on it lacks SDIO, so it will need to have the full driver stack loaded. (I didn't turn the backlight on because I wasn't sure I could trust the battery, so sorry about the pictures.)

I set it up as a LAN ...
... and gave it a user name and password. Let's start up Overbite Palm.
The Palm automatically establishes a connection and tries to load the Floodgap default menu ...
... and this time Plua stops it with an out of memory error after just 2K of menu data loaded. Our heuristics aren't even enough to let the app launch.
The reason is that the Bluetooth driver has overhead of its own. While the 512K of dynamic heap in the Dana wireless can take the hit, the m505 gives us just half of that. I pulled the m505 out of the cradle, let the memory clear, and started anew.

Since we know from running it in POSE on an emulated m515 that it can run with just 256K of dynamic heap, we need another connectivity method, preferably something built-in. (POSE redirects NetLib calls to the operating system using a very thin layer.) So now we're going to try raw PPP itself, which almost certainly should have a smaller memory footprint than a full BT stack. This time, however, the connection problem is physical: if the m505 cradle connected over RS-232 I could conceivably hook it up to my Raptor Talos II with a dongle and serve it PPP that way, but inconveniently (for this purpose) it's USB.

If you're lucky the USB connection itself shows up as a serial port device, but we have an even better alternative. On my iMac G4 (which I used for Palm development) is a tool called USB-TCP Bridge, part of the OSX Palm Tools package. These are for early versions of OS X, but they run fine on Tiger. As they are PowerPC only you'll either need a Power Mac or to port them yourself from source. And doesn't everyone deserve to have an iMac G4 on their desk?

On the iMac G4, I install socat (netcat should also work), enable IP forwarding with sudo sysctl -w net.inet.ip.forwarding=1, and create a basic script to run a PPP daemon (this is built into 10.4, but my Monterey MacBook Air still seems to have /usr/sbin/pppd):

#!/bin/csh -f

# csh script haters can eat me

if ($uid != 0) then
        echo 'not root'
        exit
endif

exec /usr/sbin/pppd :PALM_IP \
local ms-dns PALM_DNS netmask NETMASK \
passive noauth proxyarp notty debug \
persist nodetach asyncmap 0 ktune
This script (I saved it as ~/pppd.exec) starts pppd, giving the Palm the static address PALM_IP (don't forget the colon), DNS server PALM_DNS and netmask NETMASK, further specifying we're waiting patiently until the Palm says it's ready, we're not authenticating the Palm ahem ahem ahem, we're putting the Palm on the local network, we're telling pppd this is a socket and to handle creating a TTY by itself, we're enabling debugging, we're keeping the connection up even if the Palm seems to disconnect, we're not letting it go to the background, we're not filtering any characters, and we're allowing pppd to tune the kernel if it wishes (and it will).

I then set socat to listen on a port (as root), something like socat TCP-LISTEN:9596,bind=127.0.0.1 EXEC:/home/screwtape/pppd.exec (or the equivalent for netcat), and then configure USB-TCP Bridge. On the screen you can see it's set to "Listen on USB and connect to TCP port" and the TCP port is specified as 9596, as we gave to socat. We start the bridge, which connects to the socat port, and waits.

On the Palm side, we set up a regular PPP connection via the cradle (if it asks you the service, UUNet will suffice; timeout is your choice, and make sure the "script" has exactly one entry, namely "End"). The username and password are irrelevant because we told pppd not to authenticate (so make sure your port isn't listening on something externally routable).

The connection automatically starts when we start Overbite Palm again.
Ta-daaa! Unfortunate I can't use Bluetooth on this device like I do with the others, but it works. By this way, this method also works for the dana's USB connection, which looks like a cradle to the Mac or PC when it's not acting as a keyboard.
I think it's clear that a unit with 128K or less of dynamic heap is going to have a bad time, so this means theoretically the oldest Palm capable of running Overbite needs to have at least 4MB of RAM (using Palm's default table). The earliest unit I know of with that amount capable of OS 3.5 is the 1999 Palm IIIx, or alternatively the IIIc or IIIxe since they both came with 3.5 out of the box (the 1999 Handspring Visor Deluxe has 8MB of RAM, but is stuck at 3.1H). These units should be able to connect over PPP the same way with their serial cradles, so anybody game to try?

Anyway, 18 years isn't too long to wait for a truly great Gopher client on PalmOS, right? Overbite Palm is provided to you under the BSD 3-clause license, with all files on Github.