Sunday, January 22, 2023

Bringing TLS to the Magic Cap DataRover

Today we're adding TLS 1.3 to the one and only web browser on a 36MHz MIPS handheld running Magic Cap, the most unique mobile operating system from the most influential startup you never heard of. But before we do, a thank-you to Scott and Barbara Knaster:
Scott, who is of course well known for his Macintosh books and his work at Alphabet-Google, also worked at General Magic as a technical writer. Barbara, his wife, wrote this very complete third-party book on Presenting Magic Cap — apparently the very first! — and all the cool things users could do with it, from basic E-mail and contacts (there's a reason why there's a cloud downtown, long before the term was fashionable) all the way to building forms and interfaces with the Magic Hat in construction mode. (One demonstration is making an E-mail form for lunch orders. You send it to people, they fill it out and send it back. It's all supported by the operating system.) It also includes a look at tinker mode, letting you steal the painting from the hallway and put it in your office behind the desk, or change the pictures on the doors. All this was possible without writing a line of code: as she concluded at the end of the book, "Magic Cap has great potential to expand and enrich the way people communicate." It sure did. Thanks, Barbara and Scott!
In our preceding long tour of General Magic's Magic Cap, we not only looked at the history of the company and the influence of its former employees, but also at the first commercially available Magic Cap "Personal Intelligent Communicator" device, the Sony PIC-1000 Magic Link, and the last, the General Magic (later Icras) DataRover 840. Magic Cap was a unique operating system strongly based around an object and scene metaphor with settings like a desktop, a hallway with functional rooms and even a downtown area for remote service access. Messaging and remote code execution (using the Telescript language) were first class citizens, though unfortunately the latter's potential went largely unrealized. The PIC-1000 was based on a 16MHz Motorola 68349 running Magic Cap 1.0, the first release of Magic Cap, and the DataRover 840 ran Magic Cap 3.1, the final release (codenamed "Rosemary"), on a 36.864MHz Toshiba TMPR3902U derived from Toshiba's MIPS R3000-compatible R3900 core.
But Magic Cap isn't (just) an overgrown E-mail client: it's a true platform you can run apps on. Sadly, there were comparatively few applications available for the 68K-based Magic Cap 1.x, and very little for the MIPS-based Magic Cap 3.x (infamously, there was no Magic Cap 2). So for this entry we'll talk about how those development environments work and the architecture of the operating system, though I'm going to concentrate primarily on Magic Cap 3 since just about nothing out there talks about it, and use the Magic Cap 3 development tools to hack in Crypto Ancienne support for TLS support with the Magic Cap 3 Web Browser.
Magic Cap is pervasively object-oriented, which is to say, it has objects onscreen that the user interacts with that map more or less directly to objects the operating system maintains, along with a menagerie of unseen objects and items that are objects internally but don't appear as such to the user. Objects have a view hierarchy and can contain other objects, like the desktop scene containing your date book and phone, or the global tote bag schlepping items from another room or scene, or (in this case) one of your desk's drawers containing a calculator.

The "Dragon I" Motorola 68349 CPU in 68K PICs does not have an MMU in the standard sense and all code installed in the device runs where the OS places it in memory. Magic Cap also does not multitask, but packages always appear to be "running" because their data is typically loaded and saved transparently as the user switches between them. In Magic Cap 1.0, 512K was useable by the current running package for working memory ("transient") such as temporary objects, and the remainder (in the PIC-1000, it was a 1MB unit, so 512K also) was available to store data and packages ("permanent"), plus any objects or packages in ROM. Objects were never created on the stack, but a package could explicitly place an object within either heap; both transient and permanent heaps are periodically garbage-collected by the OS, though infrequently, as this could lead to sometimes obnoxious pauses on slow machines. Objects are secondarily grouped into clusters, which handle objects in a contiguous tract of memory of a particular type (be it RAM, ROM, or static RAM cards), and an object can belong to one and only one cluster. The 68349 has various system protection features which could contain the damage from a rogue package and/or force the unit to warm start, clearing transient memory and returning control to Magic Cap in ROM, though it was still possible to crash the system and require a power-off.

68K Magic Cap was developed almost entirely on Macintosh II systems, which is not surprising, given that General Magic was originally an Apple spinoff and leadership were former Apple employees (Bill Atkinson, Andy Hertzfeld and Marc Porat being the most notable of many). As such the development tools were built around the Macintosh development environment of the time, which was usually MPW (the Macintosh Programmer's Workshop) and later the Metrowerks CodeWarrior IDE. Code could be previewed in a simulator, which the screenshots on this page are actually generated from, and ran on the developer Mac natively. Regular tools like MacsBug thus could debug the program in simulation on the Mac before downloading it to a real device for validation. This required either a "Telebug box" or the Magic Xchange sync cable, the latter being far more commonplace.

The first commercial development tool release was Magic/MPW from Metrowerks in 1995. It required a 68K with an FPU (if you had a Power Mac, you had to install an FPU emulator, since the nanokernel 68K emulator only provides a virtual 68LC040). Programs were built in MPW but compiled and linked with Metrowerks tools. Documentation, the Simulator and sample applications were included. Later Metrowerks added explicit support for Magic Cap to the CodeWarrior IDE and it became part of their developer offerings through at least CodeWarrior Gold 10.

Despite being object-oriented, Magic Cap is programmed in C, not C++. Objects don't even work like C structs even though structs underlie object fields. Instead of a pointer to an object, you get an ObjectID.

This is the Inspector view within the Simulator (I've used the Rosemary simulator here for convenience but the Magic Cap 1.0 is much the same). Every object has an ObjectID, which is provided to the operating system as a reference if you want to do something with it, and the OS then does the actual look up and dispatch. The Inspector view allows you to see the properties and hierarchy of an object at runtime in its "native habitat" (you can pick a particular object by clicking the question mark, and then selecting the object: here we chose the desk phone). You can also dump this data to a text file to recreate it in your own package.
Normally, the Inspector shows symbolicated names so you don't go crazy, but clicking the "$" (presumably for hex) turns symbols back into raw hex values.
To prove this matches, we'll also pull up the phone's "tinker" view (a shortcut is clicking the "o"), showing the object ID that we are editing is the same.

But though they sure look like pointers, ObjectIDs are not pointers, and can't be turned into pointers and have methods called in them the Simula way. (Yes, it's C, you can still make pointers, but only to your own stuff; Magic Cap won't facilitate it for you.) This is where the Rustaceans in the audience start cheering, and I grudgingly admit this makes a whole class of errors impossible to even write, which helps to mitigate some of the absent memory protections we take for granted elsewhere. But the language feels somewhat like the C preprocessor and Smalltalk had a fling and kept the kid, then locked the kid in a room and only spoke Elizabethan English in their presence. By this I mean to say that you can get a general idea what's going on but the compiler responses seem somewhat quaint and often circuitous, and it sometimes felt like I had to write twice as much to do half as little.

Methods are actually direct calls to C functions overlaid with some preprocessor magic: a C++ statement like obj->method(parameter) is expressed in Magic Cap C more or less as Method(Obj, parameter) and becomes a call to, sort of, there are probably exceptions, Obj_Method(self, parameter), the function body for which starts out as Method Type Obj_Method(ObjectID self, Type parameter) { }. Private methods are possible too. The preprocessor handles much of this, even the name mangling, which means you also have to keep a define called CURRENTCLASS current with the, uh, current class you're writing methods for so that it does the right thing. In 68K Magic Cap, code also was divided into segments (like on a Mac) and direct cross-segment calls were not supported (public method calls were fine). Normally each class had its own segment, using another define called segment that also had to match the current class, but the cross-segment call limitation proved problematic if classes were calling private methods in other classes and could lead to crashes. In this case the programmer had no choice but to put everything in the same segment and hope it all fit (or pass -model farCode to a compiler that supported it; the Metrowerks ones did).

The actual classes themselves are not declared in C. Well, under the hood they are, but you don't write them as C headers; a separate tool called ObjectMaker does the generation for you off a definition file (.Def). Inheritance and overrides are supported, as well as object variables called fields. Because fields actually are underlaid by structs and can present the risk of an ABI mismatch, another layer called attributes allows fields to be treated by foreign objects as just an implementation detail with the attributes serving as the public interfaces to internal properties.

As an example, consider this trivial object from Barry Boone's Magic Cap Programmer's Cookbook, written for Magic Cap 1.0.

define class MyStamp;
    inherits from Stamp;
    
    overrides Tap;
end class;

This defines a new class derived from the system Stamp class, with one overridden method. New methods are defined as operations, with the operation keyword basically a synonym for "method" elsewhere. Stamps are utility objects in Magic Cap that contain visible items such as pictures and UI controls, and can even serve as the visual manifestation of a particular abstract property (we'll show an example later). They are so important to Magic Cap that the Stamper is one of the ever-present icon soft buttons, and when the title bar of the Stamper window is option-tapped turns it into the Magic Hat which allows you to drop almost any object into the current scene. In fact, doing so is how you build interfaces — you create them with the Magic Hat and position them, and then have the simulator dump out the definitions for your source code (or you can just write them by hand). Accordingly, an object for our class might look like this, again from Boone for Magic Cap 1.0:

instance MyStamp 'my name' 25:
           next: nilObject;
       previous: nilObject;
      superview: (Scene 24);
        subview: nilObject;
 relativeOrigin: <0.0,0.0>
    contentSize: <0.0,0.0>
      viewFlags: 0x50081200;
     labelStyle: {60,1};
          color: 0xFFFFFFFF;
       altColor: 0xFF000000;
         shadow: nilObject;
          sound: nilObject;
          image: {6,263};
end instance;

The optional text in single quotes gives it a user-exposed label, but on the backend this instance of MyStamp is referred to as number 25 within your package. The reference numbers in this snippet uniquely identify this particular definition of MyStamp and where in the environments (Scenes) it is located, here at a particular coordinate within scene 24 defined elsewhere in the same package, whatever that is. Reference numbers must be unique within your package, which was such an important constraint that later CodeWarrior IDEs supporting Magic Cap would search your project's entire stack of definitions to find the next free reference number for you. Again, you'll notice the format is pretty much the same thing you saw in the Inspector. If you've got an object set up the way you want it, the Simulator can just export the data as text that you can copypasta right into your definitions. As the definitions file was also pre-processed, you could include other definitions from other files or use macros.

However, reference numbers only specify a particular definition to the compiler; despite what the keyword instance may imply, they don't specify the particular object created according to that definition to the runtime (that's what ObjectIDs are for). Additionally, because reference numbers are only unique at the package level, global identifiers called indexicals are needed to point to specific object definitions in other packages or the system, or can simply be used for convenience or to refer to specific static objects. Two indexicals are in the object definition for labelStyle and image, set off with { } , though they can also have symbolic names specified by the preprocessor, such as #define ipPackageIndexical MakePackageIndexical(26,4). Some indexicals are provided by Magic Cap itself: the particular indexical used here for the image points to the built-in smiley face. As indexicals are functionally global variables, they have much the same advantages and disadvantages, so the convention is to use the i prefix for system-provided indexicals and ip for package-specific ones.

Finally, as we've overridden a method, let's write it (again Magic Cap 1.0):

#define CURRENTCLASS MyStamp
#define segment MyStamp

Method void
MyStamp_Tap(ObjectID self, ObjectID touchInput)
{
    InheritedTap(self, touchInput);
    HopToToteBag(self);
}

#undef CURRENTCLASS
#undef segment

Unlike C++, where you get an implicit this, the object is passed explicitly. In C++, the method might have been written as:

void
MyStamp::Tap(void *touchInput)
{
    ::Tap(touchInput);
    HopToToteBag(); // or this->HopToToteBag();
}

Operations can be called by number as well, which are Magic Cap's rough equivalent to function pointers, just safer because the OS does the lookup and dispatch for you also.

The next release of Magic Cap was much delayed, and it wasn't until December 1996 that General Magic brought developers together for a look at Rosemary. Rosemary was to use all new hardware, based on what General Magic called a "32-bit [MIPS] R3000-class processor," and was written in C++. Applications would have to be recompiled and in some cases modified. Despite that, the object model in Rosemary is still C and still uses the same basic concepts, even though new applications would be notionally written in C++ also, and the preprocessor still does much of the work. After all, an absolutely clean break requiring a total rewrite would have probably frightened developers off the platform completely.

On the other hand, Rosemary brought real improvements to programming on Magic Cap. MIPS didn't have to deal with code segments, so segmentation was no longer required (though the #define CURRENTCLASS dance still was). Second, class definitions were now split from object definitions into separate files (.cdef and .odef), helping to break up those sometimes massive .Def files and make finding errors in them more manageable, as well as potentially reducing compile times by narrowing dependencies. The third big change was that reference numbers were no longer required in object definitions: ObjectMaker could now take symbols instead. Our example above could thus be rewritten as instance MyStamp smileystamp 'my name'; (notice the transposition of the last two arguments). There was no longer any need to go through your source code to find an available number even though a reference number was still being generated for you under the hood. A reference symbol could be used anywhere a reference number was, so (Scene 24) might have also been rewritten as (Scene studio), assuming such an object definition existed in the package. Finally, the Magic Internet Kit, a separate add-on to Magic Cap 1.x, came built-in.

(Incidentally, I found mention of a fifth Rosemary device besides the DataRover 840 and the unreleased DataRover 440, Zodiac and Sputnik prototypes: a weird Portico-Magic Cap hybrid phone using a Dino MIPS CPU at "74MHz" [probably 73.728MHz, double the 36.864MHz in the 840] and Betty screen digitizer on a QVGA 320x240 display. No slots, so no Glaciers — scroll to our teardown of the DataRover to understand what these chips were. It was also only ever proffered as a prototype and was probably nothing more than an overclocked Zodiac in a desk phone form factor; unlike Andy Hertzfeld's handheld prototype, it was apparently never intended as a portable device. The DR840 remains the only Rosemary device you could have actually bought.)

This time, General Magic distributed the SDK to developers themselves; Rosemary development was never directly supported by Metrowerks. The Rosemary SDK required a 100MHz or better Power Mac with 80MB of RAM, Mac OS 7.5.5 or later and CodeWarrior Pro 1, from which it used the Metrowerks PowerPC compilers and its included version of MPW. Although it's possible it may work with later CodeWarriors (or MPWs), I haven't tried, and the Rosemary MPW integration seems pretty version-specific.

You can get the Rosemary SDK from Macintosh Garden, or if you just want to try the Simulator out and don't want to install the entire SDK, you can get just the Rosemary Simulator for Mac over Gopher. I installed the SDK on my Power Macintosh 7300 running Mac OS 9.1; you may wish to install it on a separate disk image or on a "clean" computer with no other development tools on it to avoid conflicts. Here's what it looks like installed (the aliases are shortcuts I added for convenience that we'll come back to):

But wait, I hear you cry. "PowerPC? I thought Rosemary devices were MIPS!" Well, they are. So how do you run the binaries?
For Rosemary, the Simulator was newly ported to PowerPC, and the Metrowerks compilers build PowerPC objects (General Magic provided a linker to turn the objects into a formal package the Simulator could run instead of a usual PEF binary). A native debugger is still used as before.

For a real device, however, General Magic provided a ported cross-compiling gcc 2.7.1 and associated bintool-based utilities as MPW tools, back when GNU libiberty still supported MPW; the same GNU support was used for the "native" classic Mac OS gcc. This gcc cross-compiler was what actually built the MIPS binary for running on physical hardware, connected to the Power Mac's serial port using a Magic Bus serial cable (the Magic Xchange cable doesn't fit the DataRover). Although officially the downloader required a test device that could be wiped and a memory card, there's a easier way to do this with Virtual PC that I'll show you a little later.

Parenthetically, Icras (the renamed DataRover Mobile Systems, spun off from General Magic) subsequently developed Windows build tools and ported the Simulator to x86, possibly using the work that already existed for the brief and sorrowful existence of Magic Cap for Windows. However, their internal package builder was actually a Silicon Graphics MIPS system running IRIX (makes sense), though it's not clear if they were using gcc also or SGI MIPSpro. Since I don't have access to either of those development environments, however, we'll just solely discuss how we'd develop on a Power Mac.

The official web browser for the DataRover was originally a free download you got via E-mail, though you can still get it as a package to transfer today. Icras was very proud of it and the message urging you to pick it up is the default message in your inbox when starting up for the first time. Our plan of attack will be to hack our new code into the browser's HTTP handler (short-circuiting the section that checks for HTTPS URLs and throws an error message), and expose new settings to control it in the UI. Unfortunately, though I always try to get the machine itself to do the crypto self-hosted, that's not really an option here: while Rosemary devices have more onboard RAM, the amount of working memory was increased to just 768K and that just isn't enough to do the crypto work and run the web browser, especially on an underclocked R3900 core with 4K/1K I/D caches. Instead, we're going to implement support for HTTPS-over-HTTP (as we did for Classilla) to talk to a Crypto Ancienne-compatible proxy, add HTTP proxy support as a bonus, and then create two rules to instantiate those settings in the user interface.

Rules are accessed from the magic lamp and are analogous to preferences in other user interfaces. Rules apply to a particular scene, but the Web Browser package just has one (the browser chrome), so the mapping is complete. A rule conventionally appears as a sentence which should explain what it does, and a rule currently in effect has a check in its box. Any parameters are underlined as if to suggest a blank you could fill in. When you tap a parameter to specify it, a new subform appears to enter the information.

Rules are effected through a particular action. We need a new type of rule action for our proxy settings that contains both a hostname and a port number. This isn't already built-in to the Web Browser, so we'll create a class to implement it. I chose to derive it from the existing class LocalRuleAction, which in retrospect made a few things harder, but it works fine. In the .cdef,

// Double parameter host and port rules.

define class HostPortRuleAction; // Cryanc
	inherits from LocalRuleAction;
	
	field valuePort: PortField;
	field portGetter: OperationNumber;
	field valueHost: HostField;
	field hostGetter: OperationNumber;
	field disabledPort: Unsigned;
	field disabledHost: Text;
	field rule: Rule;
	field templateText1: Text;
	field templateText2: Text;
	field templateText3: Text;
	field operationToExecute2: OperationNumber;
	
	overrides PerformRule;
	overrides ComputeRuleText;
end class;	

define class PortField; // Cryanc
	inherits from DigitField;
	
	field portFieldPort: Unsigned;
	attribute PortFieldPort: Unsigned; // also set by Deactivate
	
	overrides Deactivate;
end class;

define class HostField; // Cryanc
	inherits from TextField;
	
	field hostFieldHost: Text;
	attribute HostFieldHost: Text; // also set by Deactivate
	
	overrides Deactivate;
end class;

This provides classes for GUI fields, overriding their Deactivate methods so that everything is updated live, using public attributes to turn their fields into storable data (the compiler generates code for this as long as the names follow convention — notice the slight difference in capitalization, which is required). The action then refers to a hostname and port field, along with containing what the sentinel values should be if the rule is actually disabled, and operation numbers for value getters and the storage operation to perform. There are also fields for template text which is used to construct a sane-sounding sentence if there are values specified or not.

Next, we write the C for our text fields.

#define CURRENTCLASS PortField

Method void
PortField_Deactivate(Reference self)
	{
	Unsigned data = 0;
	InheritedDeactivate(self);

	(void)UnsignedFromNumeral(CopyPlainText(self), &data); // OOOH! POINTER!
	SetField(self, portFieldPort, data);
	}	
	
Method void
PortField_SetPortFieldPort(Reference self, Unsigned newPort)
	{
	SetField(self, portFieldPort, newPort);
	ReplaceText(self, DigitsOnly(Numeral(newPort)));
	}
	
Method Unsigned
PortField_PortFieldPort(Reference self)
	{
	return Field(self, portFieldPort);
	}

#undef CURRENTCLASS
#define CURRENTCLASS HostField

Method void
HostField_Deactivate(Reference self)
	{
	Reference nhost;
	InheritedDeactivate(self);
	
	nhost = CopyPlainText(self);
	ReplaceText(Field(self, hostFieldHost), nhost);
	}
	
Method void
HostField_SetHostFieldHost(Reference self, Reference newHost)
	{
	ReplaceText(Field(self, hostFieldHost), newHost);
	ReplaceText(self, newHost);
	}
	
Method Reference
HostField_HostFieldHost(Reference self)
	{
	return Field(self, hostFieldHost);
	}

#undef CURRENTCLASS

Notice the use of Reference. This is a generic handle to any object, including text. In this case, we extract the text portion of "ourselves" and push it into our field, sanitizing it as appropriate. We also provide setters, though these are private, and only the rule action methods will call them as a means of synchronization.

Now for the rule action.

#define CURRENTCLASS HostPortRuleAction

// Special rule setter for taking host and port at once. - Cameron Kaiser

Method void
HostPortRuleAction_PerformRule(Reference self, Reference UNUSED(contextObject), Reference rule)
	{
	Unsigned port = PortFieldPort(Field(self, valuePort));
	Reference host = HostFieldHost(Field(self, valueHost));
		
	if (Enabled(rule)) {
		if (port > 0 && port < 65536 && TextLength(host) > 0) {
			// Rule enabled, parameters are valid, set internal state
			OperationByNumberPassInteger(ActionObject(self), OperationToExecute(self), port);
			OperationByNumberPassObject(ActionObject(self), Field(self, operationToExecute2), host);
			return;
		}

		// Rule enabled but parameters are invalid, so turn rule off and warn user.
		// The default state of the rule is disabled, so this won't run initially.
		SetEnabled(rule, false);
		ReplaceText(iWarningAnnouncement, (TextLength(host) > 0) ? ipIllegalPortText : ipIllegalHostText);
		Announce(iWarningAnnouncement);
		// fall through
	}
	// Rule is or should be disabled, so load disabled values.
	OperationByNumberPassInteger(ActionObject(self), OperationToExecute(self), Field(self, disabledPort));
	OperationByNumberPassObject(ActionObject(self), Field(self, operationToExecute2), Field(self, disabledHost));
	}

// does the appropriate text substitution for the rule - see the Rule documentation

Method void
HostPortRuleAction_ComputeRuleText(Reference self, Reference rule, Reference mapping)
	{
	// Don't update live off the text fields. If the user cancels, the visible rule
	// will not only be defaced but it will also not properly reflect the internal state
	// of the application.
	Unsigned port = OperationByNumberReturnInteger(ActionObject(self), Field(self, portGetter));
	Reference host = OperationByNumberReturnObject(ActionObject(self), Field(self, hostGetter));

	if (port > 0 && port < 65536 && TextLength(host) > 0) {
		// Valid host and port, so display them using the template strings
		Reference tempText1 = CopyText(Field(self, templateText1));
		AppendLiteral(tempText1, " ");
		AppendText(tempText1, CopyText(Field(self, templateText2)));
		ReplaceLine(mapping, kRuleTextLineService, tempText1);
		
		Reference tempText2 = CopyText(Field(self, templateText3));
		Reference submapping = DirectID(iiHostPortTextMapping);
		ReplaceLine(submapping, kPortNameLine, DigitsOnly(Numeral(port)));
		ReplaceLine(submapping, kHostNameLine, host);
		MapText(submapping, tempText2, nil);
		ReplaceLine(mapping, kRuleTextLineAttribute, tempText2);
		
		if (Enabled(rule)) {
			// The rule can only be enabled if the inputs are valid. If the rule is enabled,
			// ensure that the edit fields reflect those current inputs (in case the user
			// cancelled out after editing them). If the rule is disabled, all bets are off,
			// so leave what they had there before so they can come back and fix it.
			SetPortFieldPort(Field(self, valuePort), port);
			SetHostFieldHost(Field(self, valueHost), host);
		}
	} else {
		// Invalid, so simply use the generic strings so the user knows what this does
		SetEnabled(rule, false);
		ReplaceLine(mapping, kRuleTextLineService, Field(self, templateText1));
		ReplaceLine(mapping, kRuleTextLineAttribute, Field(self, templateText2));
	}
	}
		
#undef CURRENTCLASS

This code is pretty hairy and took awhile to get right, so let me explain a few pieces. First, note the indexicals, which here we use to get references to our error strings as well as a reference to the system's alert box. The OperationBy*Return* family of functions take an object and an operation and call it, which we obtain from the definition of the rule action itself (we'll specify these in the .odef). This is how we call the correct getter, and when the rule is "performed" (its enabled state is toggled), the setters. If the parameters the user provided are bogus (zero length host, nonsense port number) and the user tries to enable it, PerformRule will cause the rule to automatically disable itself and display the alert with an appropriate message. There is only one scene to deal with, so we don't care about the context object.

To display the text of the rule, the rule definition has template strings. If the host and port are valid, it builds one sort of string from those templates, replacing "lines" (fields) in the string with the host and port. If the host and port are invalid, it ensures the rule remains disabled, and uses the templates to build a generic string to prompt the user to enter something. The k* constants are numbers I defined in a separate header that ReplaceLine uses.

Last but not least, we wire it into the browser's single scene. We add new fields and attributes, essentially this:

field proxyHttpsTcpPort: Unsigned, getter; // Cryanc
field proxyHttpsTcpHost: Text, getter; // Cryanc
field proxyHttpTcpPort: Unsigned, getter; // Cryanc
field proxyHttpTcpHost: Text, getter; // Cryanc
attribute ProxyHttpsTcpPort: Unsigned; // Cryanc
attribute ProxyHttpsTcpHost: Text; // Cryanc
attribute ProxyHttpTcpPort: Unsigned; // Cryanc
attribute ProxyHttpTcpHost: Text; // Cryanc

and link in these new setters (the compiler generated a getter for us).

Method void
WebScene_SetProxyHttpsTcpPort(Reference self, Unsigned port) // has autogetter
	{
	SetField(self, proxyHttpsTcpPort, port);
	}
	
Method void
WebScene_SetProxyHttpsTcpHost(Reference self, Reference field) // has autogetter
	{
	Reference newHost = CopyPlainText(field);

	ReplaceText(Field(self, proxyHttpsTcpHost), newHost);
	}

Method void
WebScene_SetProxyHttpTcpPort(Reference self, Unsigned port) // has autogetter
	{
	Log(("WebScene_SetProxyHttpTcpPort %d", port));
	SetField(self, proxyHttpTcpPort, port);
	}
	
Method void
WebScene_SetProxyHttpTcpHost(Reference self, Reference field) // has autogetter
	{
	Reference newHost = CopyPlainText(field);

	ReplaceText(Field(self, proxyHttpTcpHost), newHost);
	}

The object definitions for this are rather lengthy, so I'll just show the action, which is where everything comes together. Here's the core HTTPS rule action from the .odef.

indexical ipProxyHttpsRuleAction = (HostPortRuleAction proxyHttpsRuleAction);
instance HostPortRuleAction proxyHttpsRuleAction;
     actionType: 8; // actionCallOperationPassInteger
   actionObject: ipWebScene;
operationToExecute:	operation_SetProxyHttpsTcpPort;
      valueData: 0; // not used
ownsActionObject: false;
      valuePort: ipProxyHttpsRuleTcpPort; // TCP port control
     portGetter: operation_ProxyHttpsTcpPort;
      valueHost: ipProxyHttpsRuleTcpHost; // hostname control
     hostGetter: operation_ProxyHttpsTcpHost;
   disabledPort: 0; // disabled value
   disabledHost: '';
           rule: ipProxyHttpsRule;
  templateText1: ipHTTPSProxyIllegalSettingsText1;
  templateText2: ipHTTPSProxyIllegalSettingsText2;
  templateText3: ipHTTPHTTPSValidProxySettingsText;
operationToExecute2: operation_SetProxyHttpsTcpHost;
end instance;

We specified the browser scene as the action object, so our glue setters get called when the rule is updated. We also specify the controls (using indexicals so we get the exact control), the controlling rule (which lists this as its action), the getters, and the operations to execute (which are the setters). This is a subclass, so we have to follow the layout of the fields of the object we subclassed before we add our new ones. Everything is given as an indexical except oddly the actionType of the LocalRuleAction, which seems to need to be specified as a literal.

After all that, the proxy code itself is actually rather anticlimactic. We alter the protocol dispatch to allow HTTPS if the HTTPS rule is validly set and enabled. When an HTTP or HTTPS request comes through and the applicable proxy rule is set and enabled, we then change the host and port it connects to and have it alter the GET or POST it sends to include the original hostname and port. Again I won't show all the pieces here but here's where we plumbed the connection through from the rules:

// Create a TCP stream for this connection.
tcpStream = NewPreferredTCPStream(iTCPStreamSelector, nil);
Boolean proxyMode = false;

// Get proxy information, if enabled.
Reference webScene = DirectID(ipWebScene);
Reference scheme = CopyLine(ParseURL(theURL), kURLSchemeLine);
Reference proxyHost = nilObject;
if (TextMatchesLiteral(scheme, "https:", false)) {
	port = ProxyHttpsTcpPort(webScene);
	proxyHost = ProxyHttpsTcpHost(webScene);
} else {
	port = ProxyHttpTcpPort(webScene);
	proxyHost = ProxyHttpTcpHost(webScene);
}
if (port < 1 || port > 65535 || TextLength(proxyHost) < 1) {
	// Disabled or invalid
	SetRemoteIPAddress(tcpStream, MakeIPAddress(GetSeparatedHostAndPort(hostAndPort, &port)));
} else {
	// Enabled and valid
	proxyMode = true;
	Log(("proxy mode on"));
	SetRemoteIPAddress(tcpStream, MakeIPAddress(proxyHost));
}
SetRemotePort(tcpStream, port);

The networking code is provided by Magic Internet Kit (and is actually really easy to work with!). DirectID() turns an indexical into an ObjectID so that we can get the browser scene, then queries our autogenerated getters for the current values. If they are present and valid, we turn on a boolean saying we're in proxy mode. Ta-daa.

Let's build this sucker. I wrote a new MPW makefile from scratch so that all the binary pieces are properly assembled, though the Rosemary SDK does have some of its own idiosyncrases on top of MPW's attendant set. One that really drove me crazy was that the thePackage variable inside the MPW makefile Something.make is merely a convenience: the name of the package is in fact derived by the build system from the filename. When I tried to rename the package with an initial space to make it sort to the top, a common trick on Macs, the makefile wouldn't work at all because it couldn't find any targets (i.e., thePackage and the derived package name didn't match). It took almost half an hour to spot that extra space in the package name — and spaces count in MPW targets.

The SDK has three targets, but the Sputnik target is for a prototype that also never saw the light of day, so we're either compiling for Macintosh or Apollo (the DR840's code name). Although there is provision in the interface for using different compilers, in practice CodeWarrior is your only choice for a Power Mac build and gcc is your only choice for an Apollo.
A full set of build options are available. For the Mac test we'll do a debug build, though I'm such an awesome programmer I don't need the debugger. The ToolServer support is a nice touch.
This 7300 has a Sonnet 800MHz G4, but despite being a fast system with a fast compiler it still takes a couple minutes to build everything with CodeWarrior. We'll launch the Simulator with the new package (if it doesn't automatically load, press Command-O in the Simulator to open it). I prefer to run the Simulator with the emulated device configured with an extra 2MB of RAM because it makes things a bit zippier, but does have a consequence to be discussed when we run it on the real McCoy.

If you're running the Simulator by itself on a Power Mac without the full SDK, here's a PowerPC build of our modified web browser you can run as a package in the Simulator (see important disclaimer at the end of this post).

I should note at this point that I have a host on my household network set up with carl, the proxy/curl clone in Crypto Ancienne, listening via inetd (compile from Github). If you're doing something similar, you'll need to ensure it or a compatible HTTPS-via-HTTP proxy is running before you try the browser out.

My web browser hangs on the wall. Doesn't yours?
Here's the default appearance of the rules when nothing (or something invalid) is specified. Notice that it's all plain English.
If we tap on the HTTP rule, then we get the subform with the two text controls. We enter our hostname and port number.
The rule, being satisfied, enables itself and now displays a new string with the new settings. We can turn it off if we're not ready to use it; our code above will keep the settings "warm."
But if we try to turn on the HTTPS rule without specifying a host and port, the rule turns itself off and the alert appears. If we give it a host but not a port, then we get a different message complaining about that.
Better give it what it wants. Arguing with a DataRover is even more fruitless than arguing with a hyperactive toddler.
The HTTPS rule is now valid.
We tap "go to" and enter a URL and ... uh oh. How are we going to get this thing connected to the Internet?

Well, one way is the old fashioned way: you can plug a modem into your Mac with a phone line and pull up a PPP connection. Incredibly, that actually seems to work. I could even see AT commands when I snooped the connection with a null modem.

However, I think we'd rather use the Mac's own Ethernet connection and as long as you specify a dummy provider the Simulator will tunnel TCP connections through Mac OS Open Transport. Let's do that now. First go to the Simulator's Hardware menu and ensure that the simulated modem is disabled. Then go Downtown.

Passing the "cloud" as we go Downtown to the majestic Internet provider building, as you do (if you're new to Magic Cap, go to the Desk, then the Hallway, then out from there; or option-tap the desk soft button and choose Downtown directly).
To configure a new provider, we'll tap the Setup form as directed.
AT&T WorldNet, which was not a Telescript service like AT&T PersonaLink was, is the default. However, we'll just choose a generic type of connection.
Enter anything for the name of our virtual ISP and tap Next. When it goes on to ask your E-mail address, just close the popup; you don't have to fill in everything.
Not done yet. Even though we aren't actually dialing anywhere, we still need to provide part of the dialing instructions for our phony ISP (get it?).
Go back to the Desk and tap on the Phone.
Tap on Location. I mentioned that stamps are very important to the system; this is an example of the "visual manifestation of an abstract property" I was talking about. Open the stamper as directed from the soft buttons at the bottom.
The Phone has its own bespoke stamps for indicating the location you're in. You can pick any type of location but we'll just say we're "home."
976 numbers weren't area codes, but eh. Deep question: if you called a 1-900 976 number, would you get charged twice? Would you get twice the psychic predictions? Would they be twice as wrong?
We're connected to the Internet! Our proxy code was correctly called (I added an indicator such that it explicitly says when it's contacting a proxy).
And here's Hacker News over TLS 1.3 on our simulated DataRover. Though there's some mojibake, so let's turn on one more rule, this one built-in to the Browser.
UTF-8, please.
Job well done!

Unlike a real device the Simulator forgets everything when you quit. You can create a virtual memory card (as a file) and back up your emulated device to it, and then restore from that memory card file to bring back your settings, data and any installed apps. For our purposes here we'll just exit.

Now we'll build it for the DataRover. If you enter MagicCleanPackage as a command into the MPW worksheet, the object files will be removed so you can do a fresh build, or you can just remove the .o folder and the package and its debugging data from your project folder. We'll switch to the Apollo build (the compiler switches with it) and make a non-debug build while we're at it. We'll also uncheck the download option, since we don't have that set up yet (we have a better alternative).

The cross-compiling MPW gcc is much slower than Metrowerks C and took over twelve minutes to complete. It also has an impedance mismatch with C files: the build chain almost entirely assumes you're building C++, and the header files and dumper files linked in automatically assume it. The Metrowerks compiler just copes with that but I had to alter the MPW makefile to pass -x 'c++' to gcc for such C-not-C files, something like this (note that the GNU compiler is called Gc):

.c.o ƒ .c
    If "{C}" == "Gc"; ∂
        {C} -x 'c++' {DepDir}{Default}.c -o {Targ} {COptions} {MagicDump} {ExtraCOptions} {MIKOptions}; ∂
    Else; ∂
        {C} {DepDir}{Default}.c -o {Targ} {COptions} {MagicDump} {ExtraCOptions} {MIKOptions}; ∂
    End

With the package built, we'll exit MPW. If you installed the MPW from CW Pro 1 to an HFS+ volume, you'll get an error message from the Finder at this point saying MPW needs to be updated, which you can freely ignore. Let's plug in the DataRover and start Virtual PC.
Here's VPC 3 (I like it because it's very fast) running a vanilla installation of Windows 98. SoftWindows may also work for this but VPC seems to have fewer issues for my use cases. We've dragged the WebBrowser-MIPS-USA package to the Windows desktop so we can transmit it; VPC will use the Mac's serial ports as COM ports, so we have the DataRover sync cable connected to the 7300's modem port.

We'll next start a utility called WinPCLink, which Josh Carter also offers for download on his very complete Magic Cap site.

We connect the DataRover and go to the Storeroom from the Hallway, then double click the WinPCLink icon that appears in the middle of the desktop. "Magic" swirls above the hat which is our cue to tap the little PC icon on the table in the Storeroom.
If we did this right, Bowser comes out of the hat and we're connected.
Drag the package to WinPCLink and the transfer will begin.
I should add that while this method is easy to use and doesn't require you to set up your DataRover in any special fashion, it's also not very fast: it takes about five to six minutes to transfer the 452K package.
We have a new package!
Configuring our proxy rules (and turning UTF-8 on).
We showed you Hacker News as the first photograph of the screen, so this time we'll do Lobste.rs. However, this is where I need to point out a few things: we don't have the extra 2MB emulated RAM of the Simulator, and we're now running the browser on a CPU with a clock speed over 20 times slower and cache measured in single digit kilobytes.
Loading the page over the DataRover's EtherLink III PCMCIA card.
Rendering takes a little while, though slightly less if you also turn off tables in the browser rules. But at this point the device hangs up because it's out of memory.

We do have a cheat here by powering it off "twice" and powering it back on. This forces a warm start and also triggers a garbage collection, and because the canvas for the web page is actually in the persistent pool, we should come back to the same point with enough memory freed to pan around the page with the stylus. Lobste.rs generally works with that manoeuvre, but not Hacker News which sometimes works and sometimes doesn't, probably meaning whatever was on the front page at the time was right at the very edge of what could fit. You can see why I didn't even try adding onboard crypto, as well as why images don't load by default.

Because Web Browser renderings are persistent objects, you can discard individual pages from your history to the dump truck (trash), and your history can branch — though with the amount of overhead required, you wouldn't be branching much.

But smaller pages work fine. And here's more proof right onscreen that our altered browser was able to use the proxy to make a secure connection, so we've achieved our immediate goal.

Trying to make the browser support things like PNG images is likely futile given how constrained the environment is (though images render to the canvas, so once they're there, they're largely no additional penalty). But it should be possible to try to cut down the rendering if a lot of memory is being consumed so at least you can see some of the page, and additional HTML entities are easy enough to add to make the text look better.

Something I'd also like to hack into the browser is Gopher support, which would make a lot more sense on the DataRover than trying to view today's heavyweight web pages and bringing the poor thing to its knees (but it's not that the dog walked well, as they say). That would need some additional plumbing in the protocol handler but wouldn't need much exposed user interface and I think might be truly useful.

This browser package is available for MIPS and the PowerPC Simulator via Gopher download. You may want to remove any existing browser version (and/or JavaScript support — haven't tested with it, suspect it would make things worse) before installing this one. Also note my now almost boilerplate legal disclaimer when using software I've altered, as follows: this browser is a hacked version of an Icras package. It is issued without the permission of Intellectual Ventures Management LLC, the current owners of the intellectual property of the former General Magic and the Magic Cap operating system, who retain any residual unexpired copyright, trademarks and patents upon it. This package is offered in binary format only and its public availability in no way implies the tacit, implicit or explicit approval of Intellectual Ventures Management LLC of its use or distribution. Your use of this driver accepts all potential legal eventualities for doing so, however remote or unlikely, and that THERE IS NO WARRANTY, NOT EVEN AN IMPLIED WARRANTY, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. YOU USE THIS SOFTWARE AT YOUR OWN RISK.

But we'll be developing some other stuff for the DataRover too, including a new project (mostly) from scratch. It's got an Option button and we ought to be able to make a game out of that. Watch for it in a future entry.

No comments:

Post a Comment

Comments are subject to moderation. Be nice.