MacLynx beta 3 -- really!

Yes, I'm back to further hijinx with the port of Lynx 2.7.1 to the classic Mac OS. I have great fondness for it because it was the first browser I ran on the first Mac I personally owned (a IIsi) and I'm delighted to be dusting it off. No, it's not just a monkeypatch like "beta 2" (in scarequotes) was: this is a real rebuild off the real original source code once I did some housecleaning and refactoring. Here it is, running in A/UX:

Or, here's some actual proof it's different. Even with Crypto Ancienne providing bolt-on TLS 1.2 capability, you couldn't view Hacker News with MacLynx previously because it was sent as text/html;charset=utf-8. Now you can (admittedly System 7 doesn't understand what UTF-8 is, but let's just handwave that away for now). There's no easy way to monkeypatch in a fix like that because it's new logic and new strings, not just changing old ones, so only a code change will fix it.

And, symmetrically, here's lobste.rs, though the reason you couldn't view it before was a bug in Cryanc that was fixed in current 1.5:

That's not the only change, either. Besides accepting UTF-8 (or at least not refusing), it also has a proper fix for beta 1's inappropriate Content-Encoding header, has a "live" scrolling area (click the top or bottom halves of the scroll area to advance or go back a page), and reduces the event loop's constant redrawing of the screen. There are no changes to the rendering core in this version.

Under the hood, although the alpha 5 source code (the only version that survives) claimed to have been built with CodeWarrior Pro 2, the projects are actually from an earlier IDE, and it isn't clear which version of GUSI or CWGUSI (GUSI being a shim library to map "conventional" Berkeley sockets and file handling onto MacTCP and MacOS) was used. In addition, even though CWGUSI 1.8.0 is included with CW Pro 2, MacLynx won't, er, link with it (it can't find stdout or _Stdout, depending on where the symbol occurs). I don't blame the GUSI authors for this; that's Metrowerks' fault because the Metrowerks Standard Library (MSL) made stdout a preprocessor define, not an actual compiler symbol, meaning the pre-built version of 1.8.0 included on the CD could not have been built with CW Pro 2 in the first place.

That said, because reproducibility with classic Mac OS homebrew is a problem (the cool kids now use Retro68 but this doesn't help much with old codebases), I'm sticking with CWGUSI 1.8.0 because it's already on the disk and for MACINTOSH software there's a veritable GARDEN worth of places to find CodeWarrior Pro 2. Rather than go to the inconvenience of rebuilding it again, this ugly hackery in the main MacLynx source file made the linker error go away and doesn't affect the SIOUX debugging window.

#undef stdout
FILE *stdout;

I also purged and cleaned up the filepaths in the project after converting it to the "new" IDE. The only other piece you'll need to build is Internet Config Programmer's Kit 1.4, which you can get from Info-Mac. Everything else is nice and relative to compiler and project so it's portable again:

There are now also separate prefix files for debug and release builds, too, so you can cut out debugging code completely from a release build. Right now this only excludes the SIOUX debug window from the release version, but I'll be looking for more targets to cut the fat in future versions.

That's not to say there weren't other significant problems with the toolchain upgrade, even after the application built successfully. As another symptom that CWGUSI 1.8.0 was mismatched, various errnos were wrong because GUSI's set originally encroached on other errnos used by the MSL and used a different set of values to compensate. That was no longer the case with CW Pro 2 and thus manifested as failure to open any network connection at all (because errno didn't correspond to any of the expected signals you would get on a non-blocking socket), and required forging new processor defines that matched up with the expected values.

With that fixed, HTML pages came up all smushed together, with no links. Gopher menus appeared fine, but anything that generated HTML from a Gopher menu (like a CSO search) also didn't work, as well as Lynx's internal interfaces for downloads and image helpers. Some flailing around in the CodeWarrior debugger traced the problem down to the binary search algorithm that finds SGML tags: it wasn't matching any, even though the tags were there and correct. The problem turned out to be Lynx's internal strcasecomp (not strcasecmp), which seemed to barf on the MSL's implementation of TO_LOWER() and was returning unexpected values. I rewrote it in an embarrassingly obvious form to the compiler and finally got sane results.

With the browser basically working again, I turned my attention to something which always used to annoy me in the day: if the pointer is over the MacLynx screen, it flickers. The issue here turned out to be how Olivier had originally merged the Mac event loop and Lynx's main loop. Lynx waits patiently for a key, but you have to keep spinning the Mac event loop to service the GUI and allow other apps to run. Thus, the Mac event loop, even if the user is doing nothing, goes through all the work to update the status line and screen as if the user had done something, and the Mac curses implementation is inefficient enough that this ends up requiring an entire screen refresh every. single. tick. of the main loop. That's why the pointer flickers, and why keystrokes lag.

To reduce the need for keystrokes, I expanded the mouse support (which right now is pretty much limited to selecting links and manipulating the pulldown menu) to make the previously unused scroll area live. You can now page back and forth in a document by clicking in the top or bottom halves, and then click on links, reducing waiting for the main loop to catch up with keying around from link to link. But the pointer still obnoxiously flickered, so I worked on several ways of reducing the screen update frequency. The Mac curses is rather dumb and I wasn't sure how effective making it do partial updates would be (an exploration for another day), but it seemed like I could short-circuit all that work if a key wasn't being pressed. This turned out not to be the case. Lynx desperately assumes you pressed something, and trying to skip whole portions of its event loop when you didn't would maddeningly cause it to drop or delay other keystrokes. Eventually I added a simple global hack to Mac curses that just turns refreshes on or off and I tried to batch them as much as possible, which besides improving responsiveness also cut out a lot of CPU load too. It might have been possible to extricate the two loops with a little thought using the Thread Manager and spinning both loops separately, but I didn't want to introduce a new dependency (I like it being able to run on very minimal systems, and Thread Manager wasn't a standard part of Mac OS until System 7.5), and I didn't want to add too much complexity by telescoping yet another event loop inside Lynx's wait-for-a-key routine, at least not this time around.

Now, after everything that's been added and updated, there is one thing that's been removed: the PowerPC native build. That might seem strange coming from a vehement pro-Power ISA bigot like me (typing this post on a Raptor Talos II POWER9 workstation, even), but the reason is simply because I don't want to deal with multiple versions of CodeWarrior on my MDD G4, so I'm building this on my Quadra 800 instead and only a 68K target is available. Doing so, however, also allows me to test it on the same machine in System 7.1, System 7.6, MacOS 8.1 and A/UX 3.1, and keeps any additional toolchain changes I may need to make localized. I may resurrect the PPC build in the future but Classilla with styles off is more functional than MacLynx PPC, and of course this 68K build will run just fine on a Power Mac anyway.

What's next? I thought about porting a later Lynx but later releases may incorporate changes I might not want, and I have no idea if they would even build. More to the point, I haven't even been able to enumerate all the other changes Olivier already made to get it working, all of which would probably have to be adapted or dragged forward. I think improving its handling of <div> tags, especially with respect to positioning, as well as tables and HTML entities will get us a large part of the way to parity without introducing other potential incompatibilities. There are unused menu resources in MacLynx; those should be hooked up. I did initial work on a dialogue box for entering URLs instead of using Lynx's prompt, which is much faster than typing into the curses-driven prompt and lets you use cut and paste and the mouse, but that isn't in this version yet. And, because Lynx does know how many pages long a document is, it should be possible to actually have a real scroll bar which works by sending events to Lynx instead of the up/down mouse shortcuts (for that matter, we also need a mouse hotspot for backing up -- which is actually in this release, but I don't know if I'll keep it in its present location).

Finally, there's the whole topic of TLS 1.2. MacLynx can only access TLS sites by using Crypto Ancienne as a proxy (or anything else that offers an HTTP-HTTPS proxy, but I haven't seen others). It might be possible to build this into Lynx, but also keep in mind that most 68K Macs aren't fast enough for modern servers. My Quadra 800 is clock-chipped to 40MHz and is able to keep up by running the proxy self-hosted in A/UX or MachTen, but in my experience systems start running into trouble below that speed: my 36MHz Solbourne S3000, a SPARC, times out on a couple sites, while my 25MHz '030 IIci with no cache card can sometimes take up to 20 seconds to do a transaction and failed on most of the sites I tried. With those numbers it's painfully obvious a 8MHz 68000 wouldn't have a chance. For these, it makes more sense to run Crypto Ancienne on some other system and have the Mac talk to it because only the fastest 68040s are swift enough to do so themselves. However, current versions of Lynx don't allow offloading TLS to another system using an HTTPS URL to an HTTP proxy anymore, another reason to stick with 2.7.1.

Anyway, visit the MacLynx page for updated builds and the source code, and post your comments. Watch this blog for beta 4 when I feel up to it.