Standing in Line With Mr. Jimmy

Saturday, 15 July 2006

Tim Bray responded quite nicely to my “Why Apple Won’t Open Source Its Apps” piece a few weeks ago, persuasively arguing that open sourcing apps like Mail, iCal, and iChat would not adversely affect Mac OS X upgrade sales:

If there were an enhanced guerrilla version of, for example, Mail.app, that did message-selection correctly and had a next-unread button, both these fixes being provided by a community member, some things would happen. First, I’d use it. Second, the vast majority of Mac users wouldn’t. Third, when the Leopard release of Mail.app, presumably with new goodies, came out, it would be back-ported to Tiger. Fourth, I and most of the other people using the variant Mail.app would upgrade to Leopard anyhow.

If there’s a hole in this logic, I’d say it’s with the “vast majority of Mac users wouldn’t” part. (Justin Blanton makes a similar argument.) I agree that most Mac users would not install these enhanced guerrilla versions and that they probably wouldn’t even know they existed — but there are, what?, 10 to 20 million total Mac users? The “vast” majority could ignore them and it would still leave a sizable number of Mac users.

Many — not most, just many — Mac users occupy a slot in the computer user spectrum that is often overlooked: that of the non-technical enthusiast. These are people who don’t really understand how a computer works from a computer science perspective, but who do understand quite a bit about how a Macintosh works, from the perspective of the Mac OS UI metaphors. I.e. they’re Mac nerds but not computer nerds.

Back in 2002, responding to a Joel Spolsky piece regarding whether or not the Mac is a big enough market for commercial software developers, I wrote about this:

Sure, some Macs are used in corporate settings, and some are used at home by the “I just use the web and email” crowd. But Macs are much more likely to be used by computer enthusiasts. You pretty much have to go out of your way to end up with a Mac on your desk (or lap). If you walk into CompUSA and say, “I’d like to buy a computer”, you’re probably going to walk out with a Wintel box. If you do that at Wal-mart, you definitely will.

Let’s divide computer users into two groups — people who think computers are fun, and people who just happen to use a computer. Which group do you think buys more software (even including all the Linux nerds in the first group)? Which group do you think most Mac users are in?

Mac users may only constitute four percent of the total number of computer users; but they constitute a significantly higher percentage of the total number of users who do things like install and purchase third-party application software.

Considering that these users already do things like installing plug-ins and unsupported system extensions like APE haxies and input manager hacks, I find it hard to believe many of them wouldn’t install enhanced guerrilla versions of these apps, too, if Apple were to release them as open source. Again, not most Mac users, but, I think, more than Bray (or Blanton, or many of you who emailed me about this) suspect.

And thinking about the practical implications of this idea, there would be some tricky technical problems that would arise in Bray’s scenario. For example, what happens if someone releases a patched version of Mail that replaces the system’s official version from Apple? What happens then when the next software or security update from Apple contains an updated version of Mail from Apple? Collision.

So, one might argue, there’d be some sort of rule in place that open source guerrilla versions would not be allowed to replace Apple’s official versions in /Applications/. But how would such a rule be enforced? By stating it as a rule in the license? Good luck with that. Section 2c of the Mac OS X software license agreement, for example, explicitly forbids reverse engineering or disassembling the software, but that doesn’t seem to have kept anyone from using reverse engineering in order to create input manager hacks and haxies.

(Of course, it shouldn’t have kept anyone from reverse-engineering the system, because it’s not illegal to reverse engineer software just because it says you can’t in the EULA; software license agreements are full of bullshit restrictions that aren’t enforceable and are contrary to established copyright law. I’m just saying you can’t count on a rule being obeyed just because it’s stated in a license.)

And even if developers did release their guerrilla offshoots with unique names (and more importantly, unique bundle identifiers), you’d run into other problems. Like, say, if you decide to switch to an offshoot derived from Mail, which doesn’t replace Mail in your Applications folder because it’s named, say, “JoeMail”. But now what would happen when you clicked the email button in iPhoto? It would still fire up Mail, is what would happen.¹

Or if you were to switch to a modified offshoot of iCal but were still using Apple’s standard version of Mail, and then receive a message with an event invitation, and you click the button to agree to attend — Mail would send you to iCal, not FooCal or whatever the name is of your guerrilla version of iCal.

Point being that unless you switch from all of Apple’s official apps to an integrated and coordinated suite of offshoot derivations which have been rejiggered to interact with each other in all the various ways that Apple’s own apps do, there are going to be holes in the user experience. And so thus there’d be a strong temptation for developers to just produce apps that are meant to replace Apple’s. At which point you run into the aforementioned problem where Software Update is going to make things go haywire.

I Knew She Was Gonna Meet Her Connection

The question I’ve been pondering the last few weeks, though, is why I’m not as bothered by the status quo as Bray. I can write a little code, and my brain is chockablock with niggling little complaints and feature requests regarding the software I use. But I feel no strong urge to see any of the closed source apps I use released as open source. Why?

The other day, while tinkering with a small project in Script Debugger, the answer hit me — I already do hack together tiny little additions and modifications to the software I use, with scripting. And for more serious extensions, many of the apps I depend on have plug-in APIs.

Mac OS X is a very hackable platform; it just isn’t hackable by means of the raw source code to the software. This is one area where Mac OS X inherits good aspects from both sides of its family. From the classic Mac OS, it inherits AppleScript and a tradition of rich plug-in APIs for major applications. From Next, it inherits both the entire (highly scriptable) Unix layer and the eminently hackable and modular Objective-C runtime system. A big part of the magic in Cocoa, and a major aspect of its appeal to developers, is that it’s dynamic — Cocoa apps can be changed on-the-fly after they’re launched, in very significant ways. That’s how input manager hacks work, for example. Check out F-Script — a Smalltalk-based open source scripting layer for Cocoa — for an example of just how dynamic it really is.

Popular examples abound:

• The entire desktop publishing software industry revolves around the plug-in APIs provided by the 800-pound gorilla² applications: QuarkXPress, InDesign, Photoshop, and Illustrator. (My complete switch from QuarkXPress to InDesign, for example, was made possible only through the use of Em Software’s outstanding Xtags and InFlow plug-ins for InDesign. I’d rather switch back to Quark than use InDesign without Xtags.

• Kinkless GTD is Ethan Schoonover’s free set of AppleScripts and outliner templates for OmniOutliner Pro that turns it into a task management app for implementing David Allen’s mega-popular Getting Things Done system.

• Has there ever been a successful text editor that wasn’t richly scriptable? That’s certainly the key to BBEdit’s success, and it’s also the key reason why TextMate has emerged as a worthy arch-rival. Neither is open source; both are deeply extensible in numerous ways. (E.g.: BBEdit doesn’t support a built-in auto-complete feature, but Michael Tsai’s excellent BBAutoComplete gives you one that works via AppleScript. Plus, because it works via AppleScript, I can use it just about any properly scriptable editor, including Mailsmith, too.)

• Apps like Quicksilver and LaunchBar can radically alter the way you launch applications and open documents. Quicksilver, particularly — it’s more of a framework for plug-in interface hacks.

Some of Apple’s apps are in on it too: Frasier Speirs’s FlickrExport is a plug-in for iPhoto that adds Flickr integration; MailTags is a plug-in for Apple Mail that adds tagging and task management features. Safari doesn’t have a plug-in interface³ and I’ve been griping for over three years now about Safari’s shitty AppleScript support, but thanks to Cocoa’s dynamic runtime, there are a slew of Safari modification hacks.

Plus, it’s worth noting that WebKit is enabling developers to use HTML rendering views for purposes far beyond web surfing. Apps like NetNewsWire and Adium provide anyone who knows HTML and CSS the ability to design and customize portions of the application UI. And because they do so by means of simple bundles containing HTML and CSS files, their work is as easily shared as scripts or plug-ins.

I stated earlier that there’s a long history of this sort of scriptability, extensibility, and hackability on the Mac; but these are by no means Mac-specific tenets. The aforementioned desktop publishing apps are all cross-platform with Windows, as are their plug-in APIs. And even in the open source Unix world, there’s a high correlation between popularity and extensibility. Tim Bray’s own preferred text editor (as well as Mark Pilgrim’s, for that matter), Emacs, is richly scriptable via its internal Lisp implementation. Yes, Emacs is open source, but Emacs hackers don’t typically tweak their editor by writing C code and compiling their own custom emacs binary — no, they write Lisp extensions.

Same goes for Firefox. Open source, yes, but when users add new features to Firefox, they don’t download the source code and compile their own web browser. They write extensions. Or, more commonly, they install the meta-hack Greasemonkey extension and then write scripts in JavaScript that run through Greasemonkey. (Which brings us right back to Mark Pilgrim again.)

That’s not to say that being open source is irrelevant, of course. Occasionally people do compile their own Emacs binaries — e.g. the famous Emacs/XEmacs fork), and things like the Carbon Emacs port for Mac OS X. And with regard to Mozilla and Firefox, there are offshoots like Camino and Flock.

But those examples only help make my point: Camino and Flock aren’t just minor deviations from Firefox; they’re entirely separate and different applications. The same goes for Carbon Emacs: the whole point of it — offering some measure of integration with the standard Mac OS X GUI environment — is inherently beyond the scope of Emacs’s insular Lisp extension mechanism.

Opening the source code for an application makes it possible to modify or tweak the app. Creating a plug-in or scripting API not only makes it possible, but makes it relatively easy. Open source offshoots such as Flock and Camino are about radical changes to the architecture or user interface of their parent app, and aren’t the sort of thing Bray is talking about.

Plus, doing your guerrilla UI hacking by way of scripting and plug-ins solves all the forking issues mentioned earlier — you’re not modifying the application itself, on disk, so you won’t interfere with system updates or the interaction between the app and the rest of the system. And it’s easier to share your work with others — “try my script” is a lot more appealing than “try my custom binary of iCal”.

Singing, ‘We’re Gonna Vent Our Frustration’

And so now we’ve arrived at a way of looking at Bray’s gripe with Apple that I can sign onto — by restating it from “Apple should open source their apps” to “Apple should enable someone like me to make small tweaks to their apps by writing a few dozen lines of code”. Because for all of the good things about Mac OS X’s culture of scriptability and extensibility, there’s a lot of room for improvement. Apple’s support for AppleScript in its own applications is all over the map. I mentioned earlier that Safari’s AppleScript support is poor, but it’s certainly better than Preview, which doesn’t even have a scripting dictionary, so you can’t even do something as simple as ask Preview for the path to the document of its frontmost window.

I mentioned the MailTags plug-in for Mail earlier, but the plug-in API that it uses is completely undocumented and officially private-to-Apple. And input manager-based hacks for Safari are about as unsupported as a hack can get. But while there isn’t any documentation or sanctioned support for these techniques, I thoroughly doubt there’d be any documentation or support for the source code dropped out of Cupertino in Bray’s hypothetical open source scenario, either.

Scripting and (officially sanctioned) plug-in APIs are implicit commitments; once you release an API, the developer of the app is pretty much on the hook to continue supporting it going forward. Whereas if, say, the next version of Apple Mail completely breaks the APIs currently used by MailTags, tough noogies for MailTags (and its users).

That’s not necessarily bad, though. Apple has limited resources, and even if the entire company were fully committed to adding as many supported, documented scripting and plug-in APIs to its software as possible, there’s only so much time in the day. There are always going to be holes in the official, supported coverage. It’s better that some of those holes are accessible through unofficial, unsupported means than if they weren’t accessible at all.

Support for AppleScript at Apple seems to vary from team to team. Unlike with, say, visual user interfaces, there’s apparently no mandate for any sort of standards regarding minimal scriptability in Apple software. When I speak of “visual user interface standards” here, I’m not talking about visual consistency — which is obviously no longer a priority at Apple — but rather the fact that all of their teams apply time and effort to their software’s appearance. You might not like the user interface of every app from Apple, but I don’t think you can point to one and argue that no thought or effort has been applied to it at all. Whereas with AppleScript, there are numerous apps from Apple that don’t support AppleScript at all, but which clearly should.

One common argument from the pointy-haired-boss contingent against spending the time to design and develop good scripting interfaces is that most users never write scripts. That’s true, but it’s not a good argument against supporting scripting. The benefit of good scripting support isn’t derived only by those users who write scripts, but also by those who use them. It only took one Ethan Schoonover to write kGTD, but there are now thousands of OmniOutliner Pro users using it.

Even great scripting support isn’t a panacea — some seemingly minor tasks just aren’t possible, because no API can anticipate all possible needs. For example, one of Tim Bray’s gripes about Mail is about multiple-message selection via the keyboard: that if you use Shift-Down to select multiple messages in a mailbox but accidentally overshoot and go one too far, and so then hit Shift-Up to compensate for overshooting, what happens is that instead of shrinking the selection by one it instead grows the selection in the other direction. But this isn’t a beef with Mail; it’s actually a beef with the system’s standard list controls (Data Browser in Carbon, NSTableView in Cocoa). I agree with Bray that the behavior is both wrong and endlessly frustrating,⁴ but that’s how list item selection works in any app that uses the standard list control with default behavior (including the Finder, iCal, and the system-wide Open and Save dialogs, just to name a few instances). Could it be fixed with access to Mail’s source code? Of course — it’s all just typing. But I suspect it couldn’t be fixed easily, because it isn’t Mail that’s providing the list selection. In other words, this is a slightly harder problem to solve than it may seem, and I admit that a superior scripting interface wouldn’t help you change it. (Whether you can hack this via Mail’s undocumented plug-in interface, I don’t know.)

But as for Bray’s other complaint about Mail — that it should have a “Go to Next Unread Message” command — that ought to be possible with AppleScript. I don’t know that it can be done (because I don’t use Mail I haven’t bothered to investigate) but if it isn’t possible, it should be. (LazyWeb fame and fortune to the first DF reader to implement this for Mail. Update: Here’s a good implementation by Bruce Phillips and “Qwerty Denzel” on the MacScripter forum.)

But nothing is a panacea, including open source. Firefox would be a lot more hackable if its source code were closed but the extension interface remained than it would be if the source were kept open but the extension interface were removed. For Apple and Mac OS X, the sweet spot is providing simple, clear, consistent default features and behaviors for typical users, but providing as much extensibility as possible for hackers to play with.

Mac OS X has improved in this regard with each successive major release. And Apple’s unscriptable apps don’t always remain unscriptable; Apple Remote Desktop added a very decent scripting dictionary in the recent version 3.0 upgrade. (You know it’s good if John C. Welch likes it.) Who knows, maybe one of these years Safari will give us scripting access to tabs.

It behooves Apple to satisfy users who want to write small chunks of code to customize their environment, and who would be not just willing, but eager, to share their work with others. I don’t want the source code to their software; I just want to be able to script the hell out of it.