Tuesday, March 21, 2023

Printing real headline news on the Commodore 64 with The Newsroom's Wire Service

Besides other things I've written or supervised, so far in my time I've also edited three periodicals, and Springboard Software's The Newsroom is a big reason why. In today's article we'll not only look at the guts of this pioneering 1984 software package, but also solve a childhood mystery I've wondered about since the very first day I touched the program by MITMing an RS-232 connection to snoop on serial data — and then print out a brand new 2023 newspaper with 2023 headlines to prove we cracked the case. (Scroll to the end if you just want to see the finished product.)

But first, indulge me in some memories. The most recent time I wielded the editor's pen was for the university's monthly student newspaper (although I specifically attended the school of medicine, the university paper covered the entire campus). That was largely a solo effort with a small and dedicated team of volunteer writers done almost entirely in QuarkXPress (plus Photoshop for graphics). Initially I laboured on an overworked Power Macintosh 7100 with an alarming stack of aging external SCSI drives threatening to die on a daily basis, and in short order I persuaded the dean of students to pay for a brand spanking new "Gigabit" Power Mac G4/450 and get everything copied onto it. My proudest moments were an exposé on the Frogger-like at-grade pedestrian street crossing between the south parking lot and the main campus, which years later when I returned as an assistant clinical professor I was pleased to note was replaced with an actual bridge, and an interview of the new university chancellor. Every month I rendered the Quark files out to Encapsulated PostScript, slapped it on a Zip disk (FTP dropbox? ha ha ha), and drove 45 minutes across town in my dented 1995 Civic LX to the print shop where they did the colour separation and prepress work and set it up for offset printing. I came back, picked up a few hundred copies, filed a couple with the university library and scattered the rest in bundles around campus. The Quark files are presumably still on the Gigabit G4, which I imagine later got scrapped along with the moribund SCSI disks, but I was able to rescue the 7100 and that machine remains in my collection. I also kept a printed personal copy of each issue in my archives.

Prior to that I did copy and layout for the high school newspaper. This was the very early 1990s, so Apple IIs were still common in schools including ours, and I typed up copy in AppleWorks on a platinum IIe and printed off strips of text with a cantankerous ImageWriter II in Near Letter Quality mode (previous papers used a Qume daisywheel printer with the Prestige font installed). Paste-up was a literal process: the columns, drawings, etc. were glued or taped on a master layout, along with the banner, headers and footers, then photocopied in the school office. Quality wasn't great, but hey, such was the state of 1990s high school journalism. I can't find any copies of it in my high school memories folder, but I do still have a couple issues of the literary journal which was produced with a similar process (and that I also worked on and wrote for). None of this would have happened without my experience with The Newsroom (and, as my first experience as a periodical editor, what I think was Newsroom Pro), because it taught me the basic steps of how to manage a publication, much of which still applies even today.

Springboard Software was a 1982 developer of educational and, later, creative software, primarily for the Apple II and PC, but many were ported to the Commodore 64 and a few ran on the Atari 8-bits. Later they broadened to early Macintosh and briefly into the edutainment sector with innovative political title Hidden Agenda before being bought out by rival Spinnaker Software in 1990 (Spinnaker in turn was acquired by juggernaut The Learning Company in 1994, having already picked up Broderbund and MECC, which became the zombified face of SoftKey in 1995 after a hostile takeover, which was folded into Mattel Interactive in 1999 via a disastrous acquisition BusinessWeek notoriously called one of "the Worst Deals of All Time," and finally spun off hastily with other TLC assets in 2000 to Riverdeep Interactive Learning, now named Houghton Mifflin Harcourt Learning Technology). Their earlier titles came in distinctive hard plastic boxes, like this copy of Fraction Factory, which was the other Springboard title we had:
Although domain-specific software packages like typesetters and word processors have been part of professional journalism since at least the 1970s, the 1974 Xerox PARC application Gypsy was the first advanced word processor that could truly be considered full-stack desktop publishing, complete with a GUI and inline graphics layout and editing. That said, while the first commercially sold desktop publishing package remains under recurrent debate, The Newsroom is surely one of the earliest, dating from 1984 on the Apple II, its début platform. It was released for IBM PC and PCjr later that year and then ported to the Commodore 64 in 1985, which is where I used it (my parents bought it in 1986).
Because the software had to run on 6502-based systems limited to a 64K address space (Newsroom would run on an Apple II+, but required the Language Card for the full 64K), Springboard turned necessity into virtue by splitting the workflow along the lines of what a real publication would do and loading those tools as overlays. This meant specific applets serving as a virtual photo shop for ingest of clip art and basic graphic editing, a bespoke tool for constructing banners, and a copy desk to enter body text and add graphics ("photos," but really line art) from the photo shop. An additional set of dynamically loaded overlays ran on top of the primary subprograms to provide graphic tools and font selection. A simple cursor-driven set of icon and text-prompt menus provided the interface and worked with keyboard, joystick or Koala Pad. Dividing the program into explicit functional blocks also served a pedagogical purpose: it helped to make plainer the steps of how to ingest and transfer content from reporter to printer, which the comprehensive manual explained in detail.

The same limits of memory also mean The Newsroom has no concept of global reflow. Newsroom pages are divided into regions of fixed size, one double-sized region for the optional banner and, in two columns below, anywhere between six and ten equally sized panels depending on page size and banner presence. Panels are generated in the copy desk, which offers primitive word processing capabilities, a handful of proportional fonts (two small, three large), and the ability to flow text around graphics within the limits of the current panel being worked on. However, text that doesn't fit in a particular panel doesn't overflow into an adjacent panel; it just sits in memory, unused, until it can be used again in the same segment. A large article would thus need to be manually broken up into separate panels, a sometimes tedious process, and possibly multiple times if graphics had to be added later or the fonts changed. A separate layout applet selects from the available panels and banners for each of a page's fixed regions. Multi-page documents are handled as separate independent pages.

The combination of limited graphics tools, a small font library and Springboard restricting clip art to their own art packs (you could manually draw graphics, but only the Apple II version could import graphics from anywhere else) caused Newsroom-generated pages to have a characteristic stereotypical appearance about them, illustrated by these sample pages from the manual, and even by contemporary standards the program was not at all sufficient for professional work. Still, it was comparatively inexpensive (my sealed copy still has the original price tag on it, MSRP $49.95 [2023 about $140], sale price $39.95 [$110]) and came with a double-sided 600-item clip art disk and that excellent manual out of the box, so it was quick and complete enough to get started with it right away. The printing press applet worked with a wide variety of printers and it could use a second disk drive to cut down on constantly having to flip the disk. The only wrinkle was fairly authoritarian copy protection which repeatedly checked the master and clip art disks when switching them, something some cracked versions still don't correctly handle.

At the time I was in elementary school and messed around with printing my own stuff on our Okimate 10 printer, but a number of the local Commodore user groups generated their own regular newsletters with it and in my opinion today it was the most well-rounded Commodore desktop publishing tool for basic purposes for many years. Berkeley Softworks introduced geoPublish in 1986, which could export professional-quality PostScript and print directly to laser printers, but while it did a lot more it was also more expensive (not counting the cost of GEOS itself if you didn't have it), slower, and demanded extra hardware — in particular an REU or geoRAM — to be useful practically.

The Newsroom sold very well and was popular enough to justify a modestly upgraded version in 1986, the bigger and somewhat better Newsroom Pro, which was PC-only and sold for $79.95 (2023 around $215). It still had the same limitations on clip art and fonts, but instead of fixed panels had two fixed continuous columns that could flow content from one to the next, eliminating the need for the layout applet. I'm pretty sure this is what I used in junior high for my first journalistic experience, editing the summer "college for kids" newspaper (the other possibilities were The Learning Company's Children's Writing and Publishing Center or, least likely, the original DOS Newsroom itself), on an IBM PS/2 Model 25 with a single floppy disk drive. I ended up as editor in my group and was the only one of the class editors who thought to standardize the font between submitted articles and came up with a primitive sort of house style. I got an A.

Almost as an afterthought, the last Newsroom port came out in 1988 for the Atari 8-bit family, the result of a frenetic write-in campaign to Springboard instigated by Antic magazine in 1987. At $39.95, roughly a cool $100 today, it was cheaper too — and possibly because it was missing something:

Yes, the Atari version of Newsroom completely lacked its most interesting and mysterious feature, the Wire Service. And that's where my childhood mystery begins.
The Wire Service barely gets three pages in the otherwise copiously documented manual, which says it "allows you to use your modem, if you have one, to send or receive pages, panels, banners or photos that have been created in The Newsroom." Intriguingly, it adds that the Wire Service "allows for transmission to and from previously incompatible computers. For example, a photo might be created on an Apple IIe and then sent to an IBM PC where text could be added to create a panel. The IBM PC could then send the panel to a Commodore 64 where the panel could be added to a page. The Commodore 64 could then send the page to a PCjr for final editing. In every instance, regardless of the type of machine, the photos and text will look identical. All the computers understand each other!" (The Newsroom Pro, which predated Atari Newsroom, also has a Wire Service — though only between other instances of The Newsroom Pro, which was only ever produced for the IBM PC.)
But by 1987, other users of The Newsroom were likely running into what 1986-me had already discovered: there was nothing for the Wire Service to connect to. Most Newsroom users were just composing small self-contained projects, not the kind of big collabourative efforts the Wire Service apparently envisioned, and many didn't even have a modem anyway. I give huge props to Springboard for a forward thinking concept, and I won't ding them for not offering some sort of service of their own which would have been very expensive to run back then (to say nothing of what the user would be billed from per-minute phone charges plus any national packet-switched networks they'd have to use). Furthermore, given that it wasn't probably used much, it would likely have been mostly a source of bugs and user confusion and I can't blame Springboard for canning it in the "make them go away" Atari release. But on the fully functional Commodore version, no matter what I tried to connect to, I would only ever see this stark screen:
How did it work under the hood? What could you download? Who could you send to? This larval nerd demands to know!

Well, we're supposed to be in the networked age, darn it. Let's delight 1986-me thirty-seven years later. Let's create the real Wire Service that The Newsroom never had — and send it real headlines! As one of Newsroom's key limitations is the ability to reflow over multiple panels, let's implement something to automatically generate panels for us off a text article and even a few actual photos, and then use the Wire Service to push them to our Commodore 128DCR as a proof of concept.

To do that, let's start with how the Commodore 64 version stores its files. This may not apply at all to other versions of The Newsroom, but if we do the connector piece correctly, theoretically our implementation should be able to upload to the Apple and IBM versions also. Once we understand how panels and photos are encoded, we can snoop some actual transmissions to understand the handshaking and protocol and compare it with what actually goes over the wire.

The first thing to figure out with panels is the simple case of text. How are fonts encoded, how much can fit inside a panel, and how much can you get on a line before it has to wrap? Since we know that there must be exact conversions between the three main ports, our workspace would therefore be constrained by the lowest resolution of the three versions, which is the Apple II at 280x192 (the Commodore version uses the standard 320x200 VIC-II hi-res mode, and the IBM PC version uses CGA). The workspace is further reduced by the icon interface on the left. There are two "small" fonts, Serif and Sans Serif (approximately 8 point-ish things), and three "large" fonts, Serif, Sans Serif and English (approximately 16 point-ish things). Only one of each can be used in a panel; if you switch the large or small font, everything large or small changes to match. We want to find the combination that gives us the most space for text and then look at how the Commodore version represents that. In this case, we'll start with the best case first, which is Sans Serif for both.

All Newsroom fonts are true proportional fonts. Counting whitespace, the slimmest characters like !, i, I and ' are two pixels across. Most characters are around six. The fattest, oddly, is the underscore (on the Commodore, entered with the backarrow key), at eight. We are fortunate that the metrics for large Sans Serif appear to be exactly double that of small Sans Serif.
The worst case is English/Serif. Since the English font is really meant for banner usage, that's no great loss. The lines that entirely fit in our Sans Serif/Sans Serif panel grossly wrap here.
However, either size of Serif takes up too much space also.
Even Serif/Sans Serif is suboptimal, so we'll encode our panels for Sans Serif/Sans Serif. Ignoring the worst-case underscore, we can get at least 38 characters per small font line and 21 per large font, and 21 lines per panel (large font lines take two lines) since text can't go on "half lines," i.e., smaller than integral multiples of the 8-pixel line height of the small font. This is about as good as we can get for an individual panel.

The Commodore and Apple versions save panels with a PN. prefix. Here's PN.EXTENTS in a hex editor.

 00000000:  00 a0 97 01 01 01 d7 d7  00 d7 d7 d7 d7 d7 d7 d7  ................
 00000010:  d7 d7 d7 d7 d7 d7 d7 d7  d7 d7 d7 d7 8d df df df  ................
 00000020:  df df df df df df df df  df df df df 8d 69 69 69  .............iii
 00000030:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000040:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000050:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000060:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000070:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000080:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69  iiiiiiiiiiiiiiii
 00000090:  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 0d  iiiiiiiiiiiiiii.
 000000a0:  5c 5c 5c 5c 5c 5c 5c 5c  5c 5c 5c 5c 5c 5c 5c 5c  \\\\\\\\\\\\\\\\
 000000b0:  5c 5c 5c 5c 5c 5c 5c 5c  5c 5c 5c 5c 5c 5c 5c 5c  \\\\\\\\\\\\\\\\
 000000c0:  5c 5c 5c 5c 5c 5c 0d 5f  5f 5f 5f 5f 5f 5f 5f 5f  \\\\\\._________
 000000d0:  5f 5f 5f 5f 5f 5f 5f 5f  5f 5f 5f 5f 5f 5f 5f 5f  ________________
 000000e0:  5f 5f 5f 0d 4d 4d 4d 4d  4d 4d 4d 4d 4d 4d 4d 4d  ___.MMMMMMMMMMMM
 000000f0:  4d 4d 4d 4d 4d 4d 4d 4d  4d 4d 4d 4d 4d 4d 4d 4d  MMMMMMMMMMMMMMMM
 00000100:  4d 4d 4d 4d 4d 4d 4d 4d  4d 4d 8d cd cd cd cd cd  MMMMMMMMMM......
 00000110:  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd 8d e9  ................
 00000120:  e9 e9 e9 e9 e9 e9 e9 e9  e9 e9 e9 e9 e9 e9 e9 e9  ................
 00000130:  e9 e9 e9 e9 e9 e9 e9 e9  e9 e9 e9 e9 e9 e9 e9 e9  ................
 00000140:  e9 e9 e9 e9 e9 e9 e9 e9  e9 e9 e9 e9 e9 e9 e9 e9  ................
 00000150:  e9 e9 e9 e9 e9 e9 e9 e9  e9 e9 e9 e9 e9 e9 e9 e9  ................
 00000160:  e9 e9 e9 e9 e9 e9 e9 e9  e9 e9 e9 8d 50 30 4f 6f  ............P0Oo
 00000170:  5e 5e 5e 5e 65 73 34 33  32 37 39 2b 2d 2a 4c 4a  ^^^^es43279+-*LJ
 00000180:  58 4d 4d 4d 4d 4d 4d 4d  4d 4d 4d 4d 4d 4d 4d 4d  XMMMMMMMMMMMMMMM
 00000190:  4d 4d 0d 4d 4d 4d 4d 4d  4d 4d 4d 4d 4d 4d 4d 4d  MM.MMMMMMMMMMMMM
 000001a0:  4d 4d 4d 4d 4d 4d 4d 4d  4d 4d 4d 4d 4d 4d 4d 4d  MMMMMMMMMMMMMMMM
 000001b0:  4d 4d 4d 4d 4d 4d 4d 4d  4d 0d 30 30 30 30 30 30  MMMMMMMMM.000000
 000001c0:  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  0000000000000000
 000001d0:  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  0000000000000000
 000001e0:  0d 2e 2e 2e 2e 2e 2e 2e  2e 2e 2e 2e 2e 2e 2e 2e  ................
 000001f0:  2e 2e 2e 2e 2e 2e 2e 2e  2e 2e 2e 2e 2e 2e 2e 2e  ................
 00000200:  2e 2e 2e 2e 2e 2e 2e 2e  2e 2e 2e 2e 2e 2e 2e 2e  ................
 00000210:  2e 2e 2e 2e 2e 2e 2e 2e  2e 2e 2e 2e 2e 2e 2e 2e  ................
 00000220:  2e 2e 2e 2e 2e 2e 2e 2e  2e 2e 2e 2e 2e 0d 2c 2c  ..............,,
 00000230:  2c 2c 2c 2c 2c 2c 2c 2c  2c 2c 2c 2c 2c 2c 2c 2c  ,,,,,,,,,,,,,,,,
 00000240:  2c 2c 2c 2c 2c 2c 2c 2c  2c 2c 2c 2c 2c 2c 2c 2c  ,,,,,,,,,,,,,,,,
 00000250:  2c 2c 2c 2c 2c 2c 2c 2c  2c 2c 2c 2c 2c 2c 2c 2c  ,,,,,,,,,,,,,,,,
 00000260:  2c 2c 2c 2c 2c 2c 2c 2c  2c 2c 2c 2c 2c 2c 2c 2c  ,,,,,,,,,,,,,,,,
 00000270:  2c 2c 2c 2c 2c 2c 2c 2c  2c 2c 0d 3a 3a 3a 3a 3a  ,,,,,,,,,,.:::::
 00000280:  3a 3a 3a 3a 3a 3a 3a 3a  3a 3a 3a 3a 3a 3a 3a 3a  ::::::::::::::::
 00000290:  3a 3a 3a 3a 3a 3a 3a 3a  3a 3a 3a 3a 3a 3a 3a 3a  ::::::::::::::::
 000002a0:  3a 3a 3a 3a 3a 3a 3a 3a  3a 3a 3a 3a 3a 3a 3a 3a  ::::::::::::::::
 000002b0:  3a 3a 3a 3a 3a 3a 3a 3a  3a 3a 3a 3a 3a 3a 3a 3a  ::::::::::::::::
 000002c0:  3a 3a 3a 3a 3a 3a 3a 0d  21 22 22 22 22 22 22 22  :::::::.!"""""""
 000002d0:  22 22 22 22 22 22 22 22  22 22 22 22 22 22 22 22  """"""""""""""""
 000002e0:  22 22 22 22 22 22 22 22  22 22 22 22 22 22 22 22  """"""""""""""""
 000002f0:  22 22 22 22 22 22 22 22  22 22 22 22 22 22 22 22  """"""""""""""""
 00000300:  22 22 0d 27 27 27 27 27  27 27 27 27 27 27 27 27  "".'''''''''''''
 00000310:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000320:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000330:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000340:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000350:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000360:  27 27 27 27 27 27 27 27  27 27 27 27 27 27 27 27  ''''''''''''''''
 00000370:  27 27 27 27 27 27 0d 26  3f 23 24 40 51 57 45 52  ''''''.&?#$@QWER
 00000380:  54 59 55 4f 50 41 53 44  46 47 48 4a 4b 4c 5a 58  TYUOPASDFGHJKLZX
 00000390:  43 56 42 4e 4d 3c 3e 5b  5d 3d 28 29 28 29 28 7f  CVBNM<>[]=()()(.

The first surprise is that, for the text which is legible, it appears to be in true ASCII, not Commodore-typical PETSCII. For the text in the "large" font, we can see the low nybble appears to match the letter in question, so it's very trivially encoded by simply setting the high bit (128). It terminates with a hex 7f which seems to be an end-of-text marker. Lines that don't reflow end in carriage returns (i.e., 0d), which also have the high bit set for a run of large font text.

The initial bytes seem to represent a header. The first two are probably a dummy starting address given the Commodore stores this as a PRG file. If we assume the header ends where the text starts at byte offset 9, the second word (remember this is a 6502, so it's little-endian) is equal to the length of the data section (i.e., the file minus the header bytes), so this appears to be the length. The next two bytes are the fonts. If we compare the various versions above, they only differ in those bytes. 01 01 appears to be Sans Serif/Sans Serif. I couldn't figure out what the next two were, possibly leftover echoes of something in the panel body, though the 00 seems to delimit the start of text. Nevertheless, this is enough for us to build a first draft of the article paginator, but we'll come back to that.

For photos, we'd like real photos, not just The Newsroom's fun but limited oeuvre of cartoon line clip art. Again, our absolute theoretical maximum is capped by the limits of the Apple II's high-resolution screen at 280x192, and they'll of course be 1-bit images because the intended output medium would be the simple dot-matrix printers of the day. To get photos of enough resolution such that dithering doesn't turn them into a blurry mess, they would need to be as wide as the panel allows and as tall as we can get to still permit a simple caption and some vertical gutter space between the panels above and below it. We can measure the workspace in the screenshot and see that our widest extent is the odd measurement of 231 pixels across. After some repeated experimentation in the copy desk, it seems we can get away with a height of 148 pixels, offset two pixel lines from the top of the panel, and still have two full text lines at the bottom for a one line caption and one line of whitespace (remember no half-lines).

As such, the photo format is pretty simple. Here's what we get just creating a patterned halftone (black pixel followed by white) fill of that size, shown in this screenshot, and eliding the repeated remainder of the file. Apple and Commodore photo filenames start with the prefix PH. followed by the filename proper.

 00000000:  00 a0 c4 10 14 a7 08 ee  00 ff 55 55 55 55 55 55  ..........UUUUUU
 00000010:  55 55 55 55 55 55 55 55  55 55 55 55 55 55 55 55  UUUUUUUUUUUUUUUU
 00000020:  55 55 55 55 55 55 55 54  55 55 55 55 55 55 55 55  UUUUUUUUUUUUUUUU
 00000030:  55 55 55 55 55 55 55 55  55 55 55 55 55 55 55 55  UUUUUUUUUUUUUUUU
[...]

The file is 4302 bytes long. Again we see the starting address and the length of the file minus the header, which this time ends on an ff byte. I initially guessed the next two words were shorts with the height and width, but upon further thought that wouldn't make any sense. Instead, they're bytes; a7-14 = 147, and ee-08 = 230 (not sure why they don't start at zero). The 00 ff can't be part of the screen, as those would both be solids of one colour and then the other, so we'll call them a delimiter of some sort. (Later I found an article partially explaining the format which indicated these bytes delimited metadata for photos derived from clip art images, tagging the disk they were read from. Obviously not a concern here.) Following those, assuming eight screen pixels to the byte, there must be 29 bytes per row for 232 pixels across with the 232nd bit ignored. Multiply that by 148 and we get 4292, the same as the 10c4 in the header, and adding up with the length of the header (ten bytes) to account for all the bytes in the file.

The remaining questions are the bit order within each byte, and whether the Commodore format organizes it as a Commodore would do hi-res, i.e., in 8x8 cells analogous to characters. Some monkeypatching of the file and loading it back shows that the most significant bit is leftmost, i.e., since black-white appears here as 01010101, i.e., hex 55, then this means black is off and white is on, and entire adjacent rows are stored rather than breaking them up into cells, i.e., a full 29 bytes of row two follows row one and so forth. Pretty easy: that's almost exactly a Netpbm PBM (portable bitmap) file, and just about everything can read and write that. This Perl script (genpho, for "generate photo") shown in its entirety and available from Github will turn a PBM of those dimensions into a Newsroom photo using the header from our test file.

#!/usr/bin/perl

# turns 231x148 P4 pgm image into C64 Newsroom photo
#
# (c)2023 cameron kaiser. all rights reserved. bsd license.
#
# known bugs:
# - this is the only size supported

use bytes; # remove for 5.005
undef $/;
while(<>) { $buf .= $_; }

die("expected 231x148 P4 PGM\n") if(substr($buf, 0, 11) ne "P4\n231 148\n");
$buf = unpack("H*", substr($buf, 11)); # avoid any irregularities
die("insufficient bytes remain: need 4292\n") if length($buf) < 8584;
warn("more than 4292 bytes: remainder truncated\n") if length($buf) != 8584;

select(STDOUT); $|++;
print STDOUT pack("H*", "00a0c41014a708ee00ff"); # standard header

# read and convert bytes until no more to convert
$byc = 0;
for($i=0; $i<8584; $i+=2) {
        $v = hex(substr($buf, $i, 2));

        # invert: newsroom white is on, pbm black is on
        $v ^= 255; # invert

        # newsroom and pbm are both big endian left
        print STDOUT pack("C", $v);
}

Since the President of the United States might show up in headline news, I did a sample by reducing and cropping President Biden's presidental portrait to 231x148 and diddling a bit with the contrast, then dumped it to PBM in Krita and ran it through our converter. (We don't have the Wire Service up yet, so I just wrote it out to a .d64 and tested it under emulation in VICE.) It may be dithered but it's recognizably Joe Biden:

Lastly, photos embedded in panels.
Moving down the picture a couple lines and inserting a caption, the resulting panel file now looks like this:

 00000000:  00 a0 1d 00 01 01 42 69  01 00 02 95 08 ee 42 49  ......Bi......BI
 00000010:  44 45 4e 00 00 00 4a 6f  65 20 42 69 64 65 6e 20  DEN...Joe Biden 
 00000020:  6f 66 66 69 63 69 61 6c  20 70 6f 72 74 72 61 69  official portrai
 00000030:  74 2e 7f                                          t..

Our header has gotten larger, but the length bytes still just reflect the length of the text, and the two font bytes follow as usual. However, while our "echoed" garbage-y bytes are still present, the next byte is an 01. This likely represents the number of photos in the panel (here just one). A null follows, then our Y coordinates (this time we know where the 02 comes from: the photo is offset two pixel lines from the top) and our X coordinates, still inexplicably based at eight. These match 231x148 as expected. An eight-character filename follows that, padded by nulls, because this time the lowest common denominator is the IBM PC which uses eight character filenames. Finally, the caption text follows, terminated by the usual 7f end-of-text marker. We can generate photo panels using this as a template. Here's a Perl script (available as gencap) to do just that.

#!/usr/bin/perl

# emit a captioned photo panel for a photo generated by genpho
# image is fixed at top, 38 character caption below
#
# (C)2023 cameron kaiser. all rights reserved. bsd license.
#
# known bugs:
# - could get more text on the line
# - doesn't handle accented characters/high bit ISO-8859-1 or Unicode
# - doesn't handle embedded CR or LF
# - doesn't handle photos other than genpho 231x148 images

$photo = shift @ARGV;
$caption = shift @ARGV;

die("usage: $0 photo 'caption string'\n")
        if (!length($photo) || !length($caption));

die("caption limited to 38 characters\n") if (length($caption) > 38);

if (!open(X, "$photo")) {
        warn("photo $photo not readable, trying anyway\n");
} else {
        read(X, $buf, 10);
        close(X);
        warn("$photo doesn't look like a genpho 231x148 image, using anyway\n")
                if (unpack("H*", $buf) ne "00a0c41014a708ee00ff");
}

$photo =~ s#^.*/([^/]+)$#\1#;
$photo =~ s#^ph\.##;
$photo =~ s#\.prg$##;
die("photo filename may only be numbers and lower case\n")
        if ($photo =~ /[^a-z0-9]/);
die("photo filename must be less than 8 characters\n")
        if (length($photo) > 8);

$photo = uc($photo);
warn("photo filename canonicalized to $photo\n");

# standard header: sans serif fonts, single image, 2px from top, 231x148 photo
@bytes=map { hex } qw(00 a0 1d 00 01 01 42 69 01 00 02 95 08 ee);
push(@bytes, map { ord } split("", $photo));
push(@bytes, 0) while (scalar(@bytes) < 22);
push(@bytes, map { ord } split("", $caption));
push(@bytes, 127);
$bytes[2] = length($caption) + 1;

print STDOUT pack("C*", @bytes);

The panel generator for text (genart) is a little more complex and longer than I care to insert here, but you can find it and all the other files referenced in this blog post in the Github repo. In broad strokes what we'll do is take a text file, word wrap it according to our experimentally determined font metrics, and spread it over as many panels as necessary to fit.

Now, where are we going to get some news we can use? Everyone will have their opinion over what's fake news and what ain't, so for the purposes of this blog post we'll ignore political inclinations and just go with what's freely distributable and easy to get. Democracy Now! offers a headline news feed, here from March 17 (updated five days a week), and available under CC BY-NC-ND 3.0. I'm very familiar with it because there's already a text version that yours truly provides in Gopherspace. Our Perl script will interpret an article we provide from the Gopher news feed, turn anything "underlined" by dashes in the next line into a large font block, wrap everything to fit and span multiple panels as needed.

We can now generate a body of work to transmit, so it's time to work on the hardware. The News Service "requires" a modem, but it offers support both for the Commodore 1600 (better known as the VICMODEM) and a "manual" connection such as an acoustic coupler, meaning it can be configured to work with us establishing a manual connection for it (neither would have Hayes AT commands). We should be able to simulate such a connection between two Commodore computers using a regular null modem link and our trusty SX-64 will do nicely as a client for the Commodore 128DCR to talk to.

Since we have no idea what the actual protocol used is, we'll have to snoop the physical connection to see the bytes. There are cables you can build to watch the signals in realtime, as well as tapping the lines from any RS-232 breakout box. However, we're confronted by the fact that most computers don't have serial ports anymore (my Raptor Talos II only has a serial console port which doesn't appear as a TTY in Linux, and my Power Mac G5 doesn't have one at all). We'll have to hook up a USB serial port dongle anyway, so why not just run two serial ports, log the data we see on each port, and pass data back and forth as if they were directly connected? As an extra bonus we'll know unambiguously who's sending what to whom because we'll know what port it's coming in on.

The Newsroom's Wire Service is limited to 300 baud on the Commodore 64 (likely because of its use of the Kernal's bit-banged 6551 ACIA routines which have an incorrect bit rate for 1200 baud), though that's no great loss since many modems were also limited to 300 baud at the time. For that matter, even the PC and Apple II versions top out at 1200 baud. The Newsroom also predates any of the cartridge-based UART or ACIA solutions, so everything is user port or bust. Fortunately, rather than having to run two user port-to-RS-232 converters and turn those into USB, there are now user port connectors that have a direct USB serial link onboard like the GGLabs GLINKUSB-LT (not affilated, just satisfied).

The GLINKUSB-LT inverts the 5V (TTL-level) Commodore user port signals and wires them directly up to an on-board FTDI FT230X USB-serial converter. This chip has drivers for darn near everything and terminates in a standard USB Mini-B port. It supports up to 2400 baud for those applications that do as well as configuration jumpers for the accelerated UP9600 mode. It also has a reset button, which is a nice touch if you're not using a 128.
You don't need to do this for The Newsroom, but to eliminate it as a variable during testing I modified my pair of GLINKUSBs with a jumper wire between user port pin H (Carrier Detect) and the provided ground header. Loop the wire over the port lug instead of contacting the pin on the board side and you don't even need to solder anything. For programs expecting the modem to assert carrier on a connection, this asserts carrier all the time (remember that these and other user port signals are inverted, so to set it you pull the pin low by grounding it). It turns out The Newsroom doesn't check this but many programs do.
Just, you know, don't grab it hard with your fingers (tight fit in the port) or you'll wish you only just got a paper cut. Ow.

I threw together a program in C (intended for Linux, but should work most places) that opens a connection to /dev/ttyUSB[01] at 300 baud and passes data back and forth, logging what it sees on which port to the screen. The display groups by port. This program is also in the Github repo.

Out of the rat's nest of previously present serial cables between the SX-64 and the 128DCR, we now just have two connections going to a USB hub which goes to a port on this Raptor Talos II.
The 128DCR is set to transmit. (Cue Pink Floyd: is there anybody ... out there?) For our first test let's just transmit a dummy panel I put together for the occasion.
And, after a moment or two ... the SX-64 answers! And successfully receives the file!

Now that we know it works, let's have it transmit PN.EXTENTS, since I showed you the file in its entirety. I annotate the output from the serial snooper below.

-- port 2 ---------------------------------------------------------------------
 06 

Port 2 is the SX-64. Unlike some transport protocols, the receiver initiates: no data was transmitted on the ports until I told the SX-64 to receive. 06 is an ACKnowledge byte, same as you would see with XMODEM and subsequent related protocols, which are also receiver-initiated. This can't be a coincidence.

-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  02  c5  d8  d4  c5  ce  d4  d3  00  00  00  00  00  30  30  30 
 31  31  35  32  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  ff  01  80  04  a7  88 

This is the first packet (Wire Service calls them "blocks") to be sent. All bytes are hexadecimal. If you scan ahead, all of the blocks start with aa 55 00 00, and are all 134 bytes long, suggesting 128 bytes of data (sound familiar?) and six bytes of framing. There are no block numbers, so this sequence, i.e., alternating 1's and 0's followed by two nulls, seems to be a "valid packet" marker. 02 is likely the filetype. The filename follows with the high bit set, padded to twelve characters long with nulls, though it's probable this is never longer than eight characters due to the IBM PC's limitations.

Following that are seven ASCII characters 0001152. Scan ahead and you'll see that nine blocks follow this first one; 9 times 128 is 1152. Nulls follow until we get to an unexplained ff, an 01, a little-endian short 80 04 ($0480, or 1152 again), and two more unexplained bytes.

There are two must-be-trues about whatever we're looking at. First, this can't be a very complicated protocol: a 6502 is driving it, and this system was designed prior to 1984. However, there must also be checksum bytes, because the entire transmission was intended to go over the noisy POTS ("plain old telelphone system") landlines typical of the 1980s. ff 01 doesn't seem like a very likely checksum from any algorithm; alternatively ff could be a delimiter and 01 the number of files to send, since our screens both showed only one file being sent. If so, that leaves our last two bytes. They differ in every single block of the transmission, including this one, and that behaviour is how a checksum would act. So how is it computed?

After scratching my head over this for a bit, I eventually reasoned that the use of XMODEM ACK and 128-byte packets meant Springboard probably didn't reinvent the wheel in other places, so the checksum is almost certainly a pre-existing algorithm, and in the early 1980s there weren't many choices for 16-bit ones. If we look at the XMODEM/YMODEM Protocol Reference, we immediately see references for an XMODEM/CRC, a modification of the protocol which uses a 16-bit cyclic redundancy check instead of O.G. XMODEM's 8-bit checksum. While XMODEM didn't encode the filename and associated data as one of the packets, the later YMODEM — which also uses the same CRC-16 — does do so for batch transfers (see page 12 of the same document) in a manner similar to, though not exactly like, this one. This mode was derived from TeLink, FidoNet's extension to XMODEM by Tom Jennings, so the idea wasn't new either. As such we can guess that the checksum algorithm used here is probably also CRC-16. This is the CRC-16 algorithm on page 19, formally referred to CRC-16-CCITT or CRC-CCITT, used by XMODEM/CRC and YMODEM (by the way, this specific type of CRC still turns up lots of places today such as Bluetooth and Secure Digital cards):

/*
 * This	function calculates the	CRC used by the	XMODEM/CRC Protocol
 * The first argument is a pointer to the message block.
 * The second argument is the number of	bytes in the message block.
 * The function	returns	an integer which contains the CRC.
 * The low order 16 bits are the coefficients of the CRC.
 */
int calcrc(ptr,	count)
char *ptr;
int count;
{
    int	crc, i;

    crc	= 0;
    while (--count >= 0) {
	   crc = crc ^ (int)*ptr++ << 8;
	   for (i = 0; i < 8; ++i)
           if (crc & 0x8000)
               crc = crc << 1 ^ 0x1021;
           else
               crc = crc << 1;
	}
    return (crc & 0xFFFF);
}

It won't come as any surprise to you that when fed this packet, minus the four-byte invariant header, this code computes a 16-bit short that matches the last two bytes, though interestingly despite every other short being little-endian the checksum is stored big-endian — exactly the way it would be sent in XMODEM/CRC and YMODEM. Most likely the Wire Service is based on an XMODEM/CRC-derived protocol extended for The Newsroom's requirements but using the same basic notion for framing and packets.

-- port 2 ---------------------------------------------------------------------
 06  06 

The SX-64 responds with two ACK bytes. Since the receiver initiates the transmission, the actual protocol probably goes more like ACK packet ACK, ACK packet ACK, etc. Indeed, notice that our very last response from the SX-64 on successful receipt is a single ACK. I later started up the receiver without a transmitter, and the SX-64 kept sending ACKs every second or two while waiting for something to talk to, meaning the first ACK probably means "go ahead" and the second means "successful."

I should note I never got anything other than ACKs. I could try to foul the transmission and see if I get a NAK (hex 15) or CAN (hex 18), but that doesn't seem very useful or likely with a null modem connection. If our tools get anything other than ACK, it's considered an error and presently undefined behaviour. An exercise left for the (bored) reader.

-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  97  03  01  00  01  00  09  00  28  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  0a  d6 
-- port 2 ---------------------------------------------------------------------
 06  06 

The second packet, however, is unique to the Wire Service. Recall that we must pass all data necessary for the other side to reconstruct its own native format. We see the length bytes (97 03) and the font bytes, but expanded into 16-bit shorts. I don't know where 09 00 28 00 came from but these are in the apparent garbage byte section of the header and are likely also garbage bytes expanded into shorts too. Effectively the nine-byte header ends up getting a block all to itself which doesn't strike me as very efficient.

The remainder is the text of the extents test, and appears to be completely unmodified.

-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7  d7 
 d7  d7  d7  8d  df  df  df  df  df  df  df  df  df  df  df  df  df  df  df  8d 
 69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69 
 69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69 
 69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69 
 69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69 
 69  69  69  69  69  69  69  69  69  69  69  69  40  ed 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69  69 
 69  69  69  69  69  69  0d  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c 
 5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c  5c 
 5c  5c  5c  5c  5c  0d  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f 
 5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  5f  0d  4d  4d  4d  4d  4d 
 4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d 
 4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  7c  12 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  4d  8d  cd  cd  cd  cd  cd  cd  cd  cd  cd  cd  cd  cd  cd  cd 
 cd  cd  cd  cd  cd  8d  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9 
 e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9 
 e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9 
 e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9  e9 
 e9  e9  8d  50  30  4f  6f  5e  5e  5e  5e  65  73  34  33  32  37  39  2b  2d 
 2a  4c  4a  58  4d  4d  4d  4d  4d  4d  4d  4d  c3  3a 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  4d  4d  4d  4d  4d  4d  4d  4d  4d  0d  4d  4d  4d  4d  4d  4d 
 4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d 
 4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  4d  0d  30  30  30  30  30  30  30 
 30  30  30  30  30  30  30  30  30  30  30  30  30  30  30  30  30  30  30  30 
 30  30  30  30  30  30  30  30  30  30  30  0d  2e  2e  2e  2e  2e  2e  2e  2e 
 2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e 
 2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  ed  0b 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e 
 2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e  2e 
 0d  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c 
 2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c 
 2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c 
 2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  2c  0d  3a  3a 
 3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  00  a5 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a 
 3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a 
 3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a  3a 
 3a  3a  3a  3a  3a  3a  0d  21  22  22  22  22  22  22  22  22  22  22  22  22 
 22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22 
 22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22  22 
 22  22  22  22  22  0d  27  27  27  27  27  27  a6  21 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27 
 27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27 
 27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27 
 27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27 
 27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27  27 
 27  27  27  27  27  27  27  27  27  27  27  27  27  0d  26  3f  23  24  40  51 
 57  45  52  54  59  55  4f  50  41  53  44  46  e4  01 
-- port 2 ---------------------------------------------------------------------
 06  06 
-- port 1 ---------------------------------------------------------------------
 aa  55  00  00  47  48  4a  4b  4c  5a  58  43  56  42  4e  4d  3c  3e  5b  5d 
 3d  28  29  28  29  28  7f  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  8e  97 
-- port 2 ---------------------------------------------------------------------
 06 ^C

The final packet is padded by nulls, but since we transmitted the length of the text section in the second packet, we already know how exactly long the file should be. A final ACK and the transmission is complete.

We now know enough to create a simple "receiver" in C that emits ACKs and displays the packets it receives, so we no longer need the SX-64 for additional tests. Thanks, portable Commodore 64! But better still, we can now write a transmitter in C, too. Acid test! Let's send it a panel we generated.

/holds breath
Yay! We've got the basics down! Let's examine more files and flesh out our new Wire Service transmitter using the dummy receiver we threw together.

The second packet header comes into more play when there's actual data there, such as when we transmit our Biden photo-containing panel. If we do this, the 128DCR tries to send two files: the panel and the photo file it refers to. (The --- 06 --- is the ACK sent by the dummy receiver.)

--- 06 ---
 aa  55  00  00  02  c2  c9  c4  c5  ce  00  00  00  00  00  00  00  30  30  30 
 30  32  35  36  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  ff  02  00  01  6a  d9 
--- 06 ---
--- 06 ---
 aa  55  00  00  1d  00  01  00  01  00  42  00  69  00  01  00  00  00  02  00 
 95  00  07  00  ed  00  c2  c9  c4  c5  ce  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  11  ed 
--- 06 ---
--- 06 ---
 aa  55  00  00  4a  6f  65  20  42  69  64  65  6e  20  6f  66  66  69  63  69 
 61  6c  20  70  6f  72  74  72  61  69  74  2e  7f  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  99  81 
--- 06 ---

The first packet is self-explanatory and proves our theory was correct: the 01 after the ff was indeed the number of files to be sent, as here it's 02 (in fact, as an exceptional case, if you send a page from the layout applet the Wire Service will not only send the layout file itself but also everything it references and everything those files reference and so on, up to a full 255 files). The second packet, however, has a bit more data in it. The shorts for length, fonts and "garbage" bytes are present, but the bytes-converted-to-shorts corresponding to the photo information — Y-coordinates and X-coordinates, plus the filename — are transmitted in this packet, not in the text. That said, there's still a lot of wasted nulls which were probably left there for future expansion that never happened.

The photo follows that, which I will not reproduce in its full form, but notice that our files-to-send counter dropped to one, and our filetype is now 03 instead of 02.

--- 06 ---
 aa  55  00  00  03  c2  c9  c4  c5  ce  00  00  00  00  00  00  00  30  30  30 
 34  36  30  38  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  ff  01  00  12  75  e9 
--- 06 ---
--- 06 ---
 aa  55  00  00  c4  10  14  00  a7  00  08  00  ee  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  3f  f0 
--- 06 ---

But it starts to diverge with the third packet ...

--- 06 ---
 aa  55  00  00  ff  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00 
 00  00  00  00  00  00  00  00  00  00  00  00  93  c5 
--- 06 ---
--- 06 ---
 aa  55  00  00  80  0a  00  00  00  00  40  92  24  29  55  55  55  55  55  55 
 55  55  55  55  55  55  55  55  55  55  55  55  55  82  0a  00  00  00  00  94 
 92  24  a5  24  49  55  55  55  55  55  55  55  55  55  55  55  55  55  55  95 
 94  24  20  00  00  00  00  00  10  52  55  a5  a4  2a  49  52  55  55  55  55 
 55  55  55  55  55  55  95  94  54  55  05  80  00  80  00  00  00  51  25  49 
 94  aa  aa  aa  4a  52  4a  55  55  55  55  55  55  55  95  54  55  55  a9  0a 
 10  08  00  00  00  00  92  24  49  55  25  a9  d2  74 
[...]

... and the image data doesn't match at all. The third packet is most likely where the tag information for clip art would go. As for the image data, some of it looks like just a matter of flipping the bits around, but by the second row our bytes completely get out of whack and remain so for a long sequence afterward before briefly resynchronizing again.

I lost some hair over this for awhile before deciding to just encode the file as it was and send that to see what I'd get. The Wire Service accepted it, buuuuuut:

There are two defects here. First, the transmission is indeed sent least-significant-bit on the left, not most-significant-bit as it is stored, which you can see from the regular 8-bit-wide periodic distortion. The other problem is that every subsequent line is shifted one bit to the right, meaning we sent an extra bit. We have to strip that dangling 232nd bit on each row, which explains why the bytes seem to get out of sync (and why, after eight lines, the data will appear to resynchronize for one row because we're an integral byte less). That sort of bit-level manipulation is really obnoxious to do on an array of unsigned char. I wrote up an embarrasingly simpleminded algorithm to skip that bit which can almost certainly be improved upon. The only difference is I seem to handle the very last bit differently than the Wire Service does, but since it's never shown, it doesn't apparently matter.

Let's tie it all together. We're going to use the First Republic Bank failure and the United Nations Ukraine war crimes articles from Democracy Now! and pair them with Treasury Secretary Janet Yellen's official portrait and the Assembly Hall at the United Nations in Geneva, selected because I thought it looked nice. We will caption those images and include proper attributions, yielding two photos, two photo panels, and two articles of three and two panels respectively. That's nine files. Let's see if we can batch-send them to the 128, and if so, our Wire Service implementation is sufficient for primetime.

"The transmission has been completed." We can now send headline news articles to The Newsroom over the Wire Service. The world was waiting for this.

But wait, there's more! As promised way back at the beginning, we should be able to print a newspaper with these headlines too!

For this part I'm going to do the remaining work on the 128DCR for reasons I'll explain in a moment, then transfer everything over to VICE on the Raptor T2 to spit this out as a printable PDF. After all, I don't have many ribbons left for my Okimate and they ain't making any more of them, so let's see if we can generate Newsroom pages on a modern printer instead.

The first part is the banner. This is probably The Newsroom's weakest applet: it can't import photos (despite the banner being stored on disk almost identically to a photo as rows of bitmap data), only clip art, and everything else you have to labouriously draw. There's only one level of undo, so hope you're good (I did a lot of "OOPS" trying to get the text positioned precisely: use CONTROL plus the CRSR keys to move a pixel at a time, or grab a joystick in port 2). You'd want an Apple II around to import graphics and then Wire Service them over to something else, though I kind of get why Springboard didn't support this on the C64 and PC because those are intrinsically colour displays while the Apple II high resolution screen is fundamentally monochrome overlaid with Woz colourburst wizardry.

Additionally the 16-pixel large fonts really aren't big enough for a banner even though they're enlarged in the printed copy, and the Statue of Liberty image I took from Springboard's Clip Art Volume 1 can't even be properly scrolled off-screen to get the tip of her flame to be visible. Fortunately it looked good enough centered. Incidentally, importing that artwork caused this tag to be added to the banner header:

 00000000:  00 a0 60 09 2b 7a 08 f7  01 00 23 a7 08 32 01 02  ..`.+z....#..2..
 00000010:  12 01 00 00 00 00 01 43  4c 49 50 20 41 52 54 20  .......CLIP ART 
 00000020:  56 4f 4c 20 31 00 00 4e  45 57 53 52 4f 4f 4d 00  VOL 1..NEWSROOM.
 00000030:  00 00 00 00 00 00 ff ff  99 ff ff ff ff ff ff ff  ................

This is the same thing that would happen using clip art in the Photo Lab, which again reinforces my impression that banners are just a special case of photo files.

Now for layout, which is as simple as picking a page size with or without banner, and just plugging in which panel goes where. We'll do layouts for both U.S. letter size (8.5 by 11 inch) and U.S. legal size (8.5 by 14 inch).

Note that I didn't have quite enough panels to fill the legal size layout but you could put something like a table of contents there in a real publication. Saving the layouts to our disk with the nine files we autogenerated, we have a complete paper ready to print. I read off the disk as a .d64 with my ZoomFloppy and copied it into VICE for printing.

Now to find an emulator-friendly disk image. At this point I'm going to talk about the various cracks and hacks I've found. We'll assume you're like me and own a real, physical, legal copy of The Newsroom, and are merely finding disk images of your legally owned software for convenience with an emulator — just like using a backup copy on a real Commodore.

Besides bitwise .g64 and .d64+error information images, the most common version that circulates is the 2011 BYB patch which strips the copy protection and relaxes The Newsroom's disk test routines to accept emulated drives, including mounting the local filesystem (I usually have a local directory mounted as drive 9). That does indeed work, but the patches seem to be incomplete in that it gets very confused sometimes over what disk is what, and if it won't let you continue or cancel from there then you'll have to reboot the program from scratch. As a likely related phenomenon it wouldn't save to my emulated drives at all because it repeatedly insisted the program disk was still present, which is why I did the banner and layout work with my real Newsroom master disk on the real 128DCR instead. It works better if I don't use a fastloader cartridge (use Warp mode to load) and don't change disks at all while it's running, though this makes it hard to use clip art, and it still won't save in VICE. Also, while the BYB patch will run on a real Commodore too, it still can get confused over disks even with actual hardware and the Wire Service seems to have a bug where it will intermittently crash trying to transmit (receive works). This happened on both my NTSC 128DCR and NTSC SX-64, so I don't think that was a hardware defect.

An alternative is the 1985 Swiss Cracking Association release, which most assuredly does not run on my NTSC 128DCR or SX-64 (they crash as soon as the crack intro appears), but oddly works just fine in VICE, even in NTSC, at least using 64C ROMs. This version is limited to drive 8 even if you have other drives available, and also seems to have issues with things like emulated fastloaders, but otherwise operates, also lacks copy protection, and doesn't appear to have the BYB patch's various glitches. For printing we will use the SCA crack with the .d64 image we made of our content.

Unfortunately, regardless of what you use, anything more than basic printing is not something any Commodore emulator handles well and The Newsroom definitely is not basic printing. VICE offers Commodore MPS-803 emulation, which is a superset of the venerable Commodore 1525 and Commodore MPS-801 printer control character sequences (most Commodore direct-connect IEC serial printers support this including my Okimates and the Star NX-10C we used to have), and almost any program that prints can send 1525 control codes as a lowest common denominator. The 1525 turns on graphics mode with a 08 byte, after which it accepts 7 bits of data (the high bit must be set to identify it) corresponding to the seven vertical pins of the printer head. VICE offers a RAW mode where what's sent to the printer should be emitted exactly byte for byte to a file, but this file was consistently missing bytes (at least with VICE 3.6.1 in the Fedora package repository) and generated a recognizable but intermittently garbled image when I tried to process it. I left that program in the Github repo for when this gets fixed, since that's the ideal way to support this.

VICE also should be able to do the graphics conversion itself to an image when set to Graphics mode, but while it did write an image file, all I got was a blank page or pieces of it when I tried to print that way.

There's one more option for VICE graphics printing, which paradoxically is to set the output mode to text. The options I ended up using look like this:

In text mode, when graphics mode is detected, it simply prints a pattern of asterisks as dots with an effect not unlike putting a dot matrix page under a magnifying glass. We can mostly use this format, but even that isn't glitch-free. The converted output in the images below are intentionally unretouched, at the actual resolution rendered out (480 pixels wide and a variable height).
This was the initial output from another Perl script I wrote to take the exploded dot data and convert it to a Netpbm PBM file, which could then be converted to PDF or anything else. To my great frustration, as shown here on the American legal layout (the American letter layout was the same), while most of it came through beautifully there are regular areas where we get gaps and shifted lines wrecking its overall appearance.

The problem seems to be that VICE doesn't politely ignore the possibly spurious INST/DEL character (ASCII 20) that The Newsroom's Press applet inserts before each section. A real 1525-compatible printer would simply ignore this character at the left margin but VICE tries to do something with it and for some reason ends up shifting the line six dot positions right. The text output is fixed to 480 dots wide, so those overflowed dots get pushed onto a new set of seven lines thereafter, introducing the gap. I need to do some more experimentation with it to see if that's really the entire problem before filing a bug, but in the meantime we can compensate for this in the conversion to PBM: the data is preserved, just shifted, and because of the fixed position of each panel and the banner, we can predict every coordinate where this will occur and fix up the "dot matrix" on the fly.

Ta-daa! Now that we have a PBM file, we can turn it into a PDF. My printer uses U.S. letter paper and that page size is 612 points by 792 points (for A4, the dimensions are approximately 595x842). If we resize it to 520 pixels across, this should generate a decent set of margins but still take up most of the page. ImageMagick turned it into a PDF with convert output.pbm -resize 520 -gravity center -background white -extent 612x792 output.pdf and I sent it to the printer.
Our victory is complete. This workflow would also serve for doing any other kind of retro-styled newsletter: after all, if you want to make it look like it came from 1985, use something that came from 1985.

Is our Wire Service transmitter practical? Yes, but it's slow. Transmitting 30-odd blocks of photo data at 300bps is not a good time, especially when with my 1571 disk drive in burst mode the ZoomFloppy can transfer that file in a few seconds (and without the protocol's inefficiencies). It makes more sense to push the files to disk and work on them like that rather than push them over the modem connection since we're generating them in the Commodore native format, and for receiving them from The Newsroom, reading the disk directly is the only option since our dummy receiver doesn't even save to disk yet — let alone convert anything. Moreover, we don't yet support the banner or layout file formats (either generating or transmitting), and I have no idea whether this implementation of the Wire Service transmission side will successfully talk to an Apple II or IBM PC since I don't have The Newsroom for those platforms. I'll be interested to hear if it does.

But my childhood mystery has been solved and that's reward enough for me: the Wire Service is no longer a black box with no friends, and autogenerating and converting content for The Newsroom is now totally possible. Share my joy with the Perl scripts and C code we demonstrated, along with the files used for our example newspaper (available individually and in a .d64 image), in the Github repo. All code is under the 3-clause BSD license.

3 comments:

  1. Great article! Glad to see there is finally a way to get pictures / clip art into Newsroom.

    ReplyDelete
  2. Thank you so much for publishing this! I have fond memories of the old home published newsletters. The cliparts are so much fun and I regret that the younger generations won't get to experience the joy of crafting content this way. To add some variety to the board game Codenames I reverse engineered the old PrintMaster bitmap format and then printed out their cliparts as my deck of cards..

    ReplyDelete

Comments are subject to moderation. Be nice.