MacLynx beta 5: UTF-8, pull-down menus and more dialogue boxes, oh my!

I've been working off and on doing further Mac-ification to my updated fork of MacLynx, the System 7-compatible port of the venerable text browser Lynx for classic 68K Macintoshes (and Power Macs) running A/UX 3.x or System 7.x and later. There's still more to do, but a lot has been worked in since I last dropped beta 4, so it's time for another save point. Meet MacLynx "beta 5."

As in previous releases, MacLynx is a classic Mac-native port of Lynx, and runs in a simulated terminal window like Lynx would on a regular host (or, for that matter, on modern macOS). However, Olivier Gutkneckt's original port added Mac-specific features, like integration with Internet Config and being able to click on links, drop URLs on it or even have it read web pages to you with MacInTalk. It supports everything Lynx 2.7 does, along with TLS 1.3 proxied with Crypto Ancienne, either as a self-hosted proxy on A/UX and MachTen or to another system on your local network.

These screen shots are mostly on A/UX 3.1 with System 7.0 on my clock-chipped 38MHz Quadra 800, running Crypto Ancienne for self-hosted TLS 1.3. In so-called "beta 2," the first release in this series after Olivier's last beta in 1997, I monkeypatched a long-standing bug by hacking the binary directly. For beta 3, however, I resurrected the CodeWarrior project, rehabilitated the source code and started building actual new binaries with some minor fixes. In beta 4, I added functional page-by-page scrollbars and more hotspots onscreen you could mouse to, as well as using dialogue boxes and native Mac text fields for entering search queries and URLs — that still responded to the same Lynx keystrokes.

Beta 5 is an even larger update. It's also a bit of a bigger application, as it now requires some components from Apple's MoreFiles. Those files and paths have been added to the CodeWarrior projects (debug and regular):

To reiterate earlier build instructions, MacLynx is built using CodeWarrior Pro 2, CWGUSI 1.8.0 (included with CWPro2), and Internet Config Programmer's Kit 1.4 (available here). For the enhanced filehandling routines in beta 5, I decided not to reinvent the wheel and went with Apple's own time-tested MoreFiles package, components of which are now compiled into the app (using 1.4.9, available here). Ensure that CWGUSI, MoreFiles and ICProgKit are unpacked to the parent folder of (and parallel in the hierarchy with) your MacLynx build tree. Both the debug and optimized build projects are updated.

We'll talk about the enhanced filehandling in a moment, but the most obvious new feature in beta 5 is pull-down menus. The guiding principle I have with adding UI updates to MacLynx is that everything should enhance the Lynx experience, not obliterate it. It's still Lynx, so you can still sit down and drive it entirely from the keyboard using the same keys you used before, but now you also have the choice of doing it the "Mac way" with mouseclicks, native widgets and Command-key combinations.

To that end, every option shown on the Lynx "novice" user level (that default row of hint keys and prompts you saw in the first screenshot, and standard in all versions of Lynx) is now also accessible from the pull-down menus, and then some. To wit, there is an expanded File menu, plus Edit and Navigate.

You can see that we've added Macintosh-typical Command-key alternatives like Command-L for opening a URL — the ƒ and π are Option-Command-F and -P respectively since I'm not using a custom MDEF — but the Lynx "g" and "G" keys (the latter being the exact equivalent) still work. And I've built on the text prompt dialogue boxes from beta 4, adding a proper Edit menu and supporting cut, copy and paste, but your Lynx muscle memory to cancel or send still works fine also. If you were used to pressing the G key, typing a URL and pressing RETURN or ENTER, you can still do that. But now you can also press Command-L, type a URL and press RETURN or click OK like in other Mac browsers too.

Lynx help is still where it always is on the keyboard, but this release also adds Help Manager integration and puts it in the Help menu (8.0+) and/or the Balloon Help menu (7.x). There aren't help balloons currently. (Should there be? How would those work with a text display?)

Now, credit where credit's due: this was always the plan even before I started hacking on it. Olivier had clearly intended to add menu support at some point, just with a different layout.

Olivier had some of the same ideas like a Navigate menu, and some similar expansions to the File menu (in the resource named File (New)). This was obviously a first draft he would have expanded upon later. I think the spread in beta 5 is in the same general spirit but handles all the initial functionality it needs to.

Now that everything's in the menus, should we default to the "expert" layout in future versions to get more screen real estate on Compact Macs? Or should it still ship with the "novice" view and let the experts do as they like? Post your thoughts on that and the Help menu in the comments.

The second big update is 8-bit character support. That means we now support the full extent of MacRoman, and now we also have support for UTF-8. Let's compare with beta 4. (I should note that this is Lynx 2.7.1: character support in later and current versions is greatly expanded, and what we talk about here does not in general apply to mainline Lynx.)

In all previous versions MacLynx was limited to 7-bit ASCII; the implementation of curses and MacCurses (i.e., Robert Zimmerman's "curses for the Mac") stores the screen as 16-bit shorts but uses the upper bits for terminal attributes. As a result, where there should be accented or special typographical characters, you'll see mojibake and inverse letters here instead. Here are some real submissions on Hacker News that exhibit the issue.

In MacCurses I noticed that there were still a couple bits free that weren't being used for other attribute flags, so I shifted the standout attribute that occupied the high bit of the low byte (where the character value is stored) into the upper byte and expanded the character bitmask to allow all eight bits. That allows us to use MacRoman as the character set instead of what this version of Lynx calls "7-bit approximations," meaning we can display at least the original 217 glyphs and later almost all 255 with subsequent Mac OS versions of the Monaco font. (What you'll see depends on what version of that font you have.) This is now the default in lynx.cfg, but if you use a custom one, change CHARACTER_SET to "Macintosh (8 bit)" (no quotes, don't forget the space) and restart.

The next step was to add UTF-8 support. Although many folks would have simply embedded iconv and called it a day, we really don't need all that overhead; all we need is a converter to emit MacRoman only for those UTF-8 sequences that can be represented in it, and then consume the rest as unrepresentable. We can make this really fast, too: we have separate frequency-ordered tables for two and three byte sequences which are checked as full unsigned shorts and integers respectively (not byte by byte), we instantly reject four-byte UTF-8 sequences because we know we don't support any, and we have quick checks for the subset we do support to rule in or out a possible sequence without having to do the full linear search. There are two converters currently in the source code, a streaming one to handle body text and a string-based one for title text, which I might refactor out into a sidecar in a future version.

The last part was hooking this up to the SGML parser. This version of Lynx has some basic understanding for converting numeric character references (i.e., HTML entities with Unicode character point values) to 7-bit, but largely with kludges or in-place swaps looking for specific codepoints. I added code that would convert all numerical references into UTF-8, which then falls through into the converters, as well as code to support hexadecimal character references; this neatly sidesteps the existing kludges and expands the set of characters that can render correctly.

Now, in beta 5, there's a proper accented character and a curled apostrophe on that very same page, exactly as they should appear. Just like falling off a lög.

This support isn't perfect, of course. The native document format is still 7-bit ASCII for speed, and a document that doesn't say it's UTF-8 won't be processed as UTF-8, even if there are valid UTF-8 sequences present — though if it contains any HTML entities that specify a character greater than 8-bit, the document will necessarily be auto-promoted because we'll generate UTF-8 to be processed as part of the conversion. Also, there's no ATSUI or Text Encoding Converter support in System 7 (both features introduced in Mac OS 8.5), so we would be limited to WorldScript for non-Latin fonts, and we don't support fonts other than Monaco right now anyway. Doing so would be a much bigger deal, assuming suitable monospaced/non-proportional equivalents exist for those different writing systems. And then there's RTL script. Oh well, something for later.

Oh, yeah, file handling. Previous versions of MacLynx worked just like regular Lynx: if you selected a link that went to a MIME type it couldn't display, it would offer to download it (or cancel), and then ask for a filename. Recall that classic Mac OS deals primarily in the FSSpec, specifying a volume reference, directory ID (the containing folder) and filename, and many of the operating system functions for manipulating, moving or copying them could not cross volumes. MacLynx downloaded the file to the special Temporary Items folder on the same volume it was running from and then moved it to the desired filename and hopefully place, but it had no special handling for doing that move, so it too was limited to the same volume the temporary file was stored on. With the majority of systems just having a Macintosh HD at the time, this may have been good enough for most people in 1997.

Well, not any more. Today we have BlueSCSIs with tons o' LUNs and Raspberry Pis serving AFP volumes, and people want to save straight to those. It's time for a better way.

So let's say you want to download a file. In beta 5, you can now do that directly by Command-clicking on a link.

The first part is largely cosmetic: instead of the MIME type and D)ownload, or C)ancel text prompt, now it's a proper Mac alert. But it still responds to the C and D keys as well as the mouse, ESC and RETURN.

Once you've confirmed you intend to save it to disk, instead of a filename, in beta 5 you get a proper requester from the Standard File package. This replaces not only entering the filename but also confirming if you want to replace an existing file by that name, because the Open dialogue gives us that for free. And because I revamped the download code using MoreFiles' copier utility, you can save it anywhere. If it matches a file association in Internet Config, that application will be opened for you afterwards (such as StuffIt Expander).

(I did do some thinking here about the flow. Another principle I try to adhere to is that Lynx should be at the core and the Mac UI on the edges such that for any given high-level operation you don't cross into the Mac UI, back into Lynx and again into the Mac for each of that operation's individual steps. However, that's exactly what we're doing here by going from a Mac alert to a Lynx page to a Mac modal. Perhaps the Save to disk page in Lynx should be turned into a drop-down/pop-up or something?)

The last major feature is additional work on the document scrollbar, which was introduced in beta 4 (beta 3 had hot spots for advancing the page, but no true scrollbar, and beta 2 and earlier versions only supported using the keyboard to paginate). In beta 4 I added a document scroller that used a native Mac scrollbar and advanced page by page by mapping it to Lynx's movement keys. If you clicked the up or down buttons, you moved one page, just like pressing the up or down keys. If you clicked above or below the thumb, or dragged it, you also moved one page (except if you dragged it to the beginning or end; it supported jumping directly to those locations, since there are Lynx movement commands for those too). It did not allow you to jump multiple pages if you held down the button. If you had a scrollwheel, you moved one page with every click.

This worked and was a definite improvement, but didn't seem completely Mac-like. Lynx does support finer scrolling by the half-page or by two lines, but the trick is how to inject those keys back into Lynx's event loop from the Toolbox function TrackControl, which handles thumb drags and is usually how you'd handle a long click on the scrollbar buttons. We want to freely scroll when the mouse button is down. In the general case, Lynx's event loop calls into one of our two custom Mac event loops to get the keys we're feeding it, but unfortunately when it's running TrackControl, that function takes over the Mac event loop completely and is unaware of our carefully written ballet. Having it call a custom function on a continuous mousedown to call into Lynx's event loop to do things which may call back into the Mac event loop seemed a potential recipe for infinite recursion.

Instead, we have a scheme of "held" events. If a mousedown is detected on the scrollbar buttons, the click handler tells our Mac event loop to keep a copy of the event. If no events are pending, this event is run again, and again, and again; otherwise, events are run until WaitNextEvent has no more in the queue, and then the "held" event gets repeatedly replayed from there. To give the Lynx event loop a chance to perform the scroll command and update the screen, the code that replays the "held" event only fires it every other time Lynx's main loop calls to see if a keystroke command is pending. When we get a mouseup, the held event is obliterated, and normal event handling resumes.

This system allows us to now map the buttons to fine scrolling. If you click and hold the up or down buttons, now these trigger Lynx's line scroll, and because the event is held will do so repeatedly, yielding an effect that's more like regular Mac scrolling. Clicking above or below the thumb, or dragging the thumb, work as before. Now you have almost the entire range of Lynx motion: lines, pages, and beginning or end.

MacLynx isn't just for 68K Macs, even though I dropped generating a fat binary in beta 3 for technical reasons; here it is running on Mac OS 9.2.2 on my hyped-up MDD Power Mac G4 with dual 1.8GHz Sonnet 7447A G4s, 2GB of RAM (1.5GB available to Mac OS 9) and a 1080p desktop. This was the system I used to develop Classilla on, and I still use it on, but MacLynx is gradually displacing it for convenience and speed reasons. In this screenshot we have the MacLynx window enlarged vertically and the self-hosted TLS 1.3 support being provided by Power MachTen instead (but still running Crypto Ancienne). Notice the scrollbar's enlarged thumb, which is implemented by the Smart Scroll extension. That "just worked."

What didn't "just work," though, was the scrollwheel. I use an old Microsoft Laser Mouse 6000 with a nice clicky scrollwheel supported by the excellent classic version of USB Overdrive (a proud original supporter). USB Overdrive implements a highly compatible scroll system that hooks into the Toolbox and makes just about anything scroll "the way you expect" — except our custom routine here. When I rolled the scrollwheel, it would scroll ... and not stop until I gave it a mouseup. The trick is that USB Overdrive, and possibly other scrollwheel drivers, doesn't provide mouseups between clicks of the scrollwheel because TrackControl doesn't seem to need them. But we do. The workaround is, in the code that replays the "held" event, to check the Toolbox function StillDown(), which returns true if the mouse button is down and there are no other mouse events in queue. This is the best test the Toolbox offers us to detect a long press on the button. Fine scrolling does not work in SheepShaver, but this is because SheepShaver sends cursor up and down keys on scrollwheel events and never passes the wheel events themselves, which still advance page by page as usual.

The screenshot also shows one other trick long supported in MacLynx which I've exposed in the menus as Picture Links (the Lynx key to toggle it is the asterisk, or press Option-Command-P; the menu correctly shows the current mode as with or without a checkmark). In this mode images get their own links. You can click on them and MacLynx will download and open them in the viewer of your choice, which you specify as the application's four-character signature under XLOADIMAGE_COMMAND in lynx.cfg. Olivier's preference was GKON (Graphic Converter) because for some reason the venerable and very fast JPEGView (JVWR) would not accept the Open event. I can reproduce that; it doesn't work for me either and it's not clear why. Here I've chosen QuickTime's PictureViewer (the very memorable ogle); substitute your own preferred app. You could even use Netscape or IE as a viewer, I suppose.

In bug fixes, this release also gets rid of the issue in both beta 3 and beta 4 where keystrokes could get randomly dropped, which is kind of a big deal with a keyboard-driven application. The problem turned out to be partially a bug I introduced and a bug that was already present. There are three event loops in MacLynx that run at different times: Lynx's event loop, which sits around waiting for keys and calls the Mac code to get them, and two separate Mac event loops that primarily handle non-keyboard (the "Mac Core" loop) and keyboard events (MacCurses) respectively, though both have to be able to at least partially handle either. For Lynx to get a key, it calls the Mac Core loop to run an event, which if there are keys pending will cause a call into MacCurses to retrieve it from its keyboard buffer. Both Mac loops necessarily call WaitNextEvent in the Toolbox, which is the system's central gatherer for draining the event queue.

In beta 2 and prior, the Mac Core loop would run a single event, but only if there were no keys pending. If there were no keys pending, it would take the next event off the queue and do it. I decided this was technically disgusting and wasn't servicing GUI events fast enough (it wasn't) and eliminated the condition on keys so GUI events would always be serviced. My bug was failing to notice that this would end up dropping keystrokes if it got them before MacCurses did since it doesn't know how to pass them along, but the preexisting bug was that the bitmask in its call to WaitNextEvent would accept any event, even the keydown events it wasn't supposed to ever handle. The solution was to turn the Mac Core loop into an actual loop instead of dripfeeding single GUI events, but only ask for non-keyboard events so that keystrokes wouldn't end up on the floor, and exit back to Lynx's event loop when there's nothing but keyboard events (or ones we're injecting into Lynx from the GUI) left to do. From Lynx's view this is better because it can more reliably get a keystroke when it's ready to, and from our view this is better because system and mouse events continue to have priority and let the application respond and multitask better.

Other minor features: you can also Option-click links to get information on them, and the About box is more informative. If you hold down Option while selecting the About box, Iris will purr. It seemed an appropriate memorial.

Still to think about:

• I didn't really do much with layout in this release. I'd like things like HN and Lobste.rs (here is Lobste.rs in a debug build) to render a little more sensibly. We'll never be Acid compliant but we should be functional, and that would start with enough understanding of the page to know if elements should be conjoined or not, particularly lists.
• In this version of Lynx, entity processing is ad-hoc and duplicated in several places. The body text and title text required separate converters.
• I need to add a requester to open local HTML files (and attach it to Command-O). Drag and drop already works.
• There are some occasional repaint bugs with the scrollbar and I need to do more testing with the different scrollbar configurations possible with the Appearance Manager.
• There are still many named HTML/SGML entities we do not support, even though now we are more likely to be able to actually display them correctly.
• There is an irregularity on A/UX where if you try to run MacLynx from your home directory (presumably on a UFS volume), it will see that you actually have an environment and a home directory, ignore the initial startup page we provide, and try to use the one specified in lynx.cfg (defaults to something like ~/index.html). This typically doesn't exist and ends up making the application unable to start. If you run it from an HFS volume, this problem doesn't happen and the default local page is loaded as usual. I'm not sure which is technically the bug since that's the file we actually specified, but the A/UX behaviour seems the more unexpected of the two.
• I would like to understand why JPEGView won't return my phone calls open my image files. They're not at all dodgy. Some are quite nice.
• The options, bookmarks and cookies pages are pretty much fossils and should be audited for functionality (and whatever portions of them can become conventional GUI controls). Once they're working, they should be added to the Navigate menu. I have a dream that I should be able to interact at least on some sites with MacLynx.
• There is no way to save runtime state like window size and so on and there should be.

Well, that's enough pontificating. You can get the source code and the binary from the MacLynx home page; don't forget to adjust the memory partition, as we still default to 1024K required and 2048K preferred for our smaller systems. Post your thoughts in the comments. MacLynx, like Lynx, is released under the GNU General Public License v2.