Unifying File Systems, Objects, and Resources

I often have this vague sense that the artefacts of computing that we see today are often small, fragmented facets of some deeper “truth”. One area that I keep coming back to is the similarities and overlaps between file systems, objects, and Web resources.

Files, organised into directories, appear in almost every operating system since Multics, and there appears to be something essential about them. Mobile operating systems tried to remove files from the user’s view, but they’re making a comeback. It appears as though you can’t avoid them.

— A General-Purpose File System For Secondary Storage

Wikipedia states: “A computer file is a computer resource for recording data discretely in a computer storage device.” This seems intuitively obvious — their origin is in the fact that RAM is volatile so information is lost when a machine loses power. That information should be persisted between restarts, and discrete sections of that information should be independently addressable.

It seems as though as an abstraction they’ve been massaged a little over the years, but their essential nature hasn’t changed — or has it? To the best of my knowledge, the notion that a “file” had to map to a disk region started to evaporate with the introduction of procfs in 1984 in Unix 8. It is convenient to represent run-time information in the same way as disk backed information because you immediately inherit all the tools and utilities that work with files and directories. Files have the benefit of having a name that can be passed around as a reference, structured in a way that can be “walked”.

Plan 9 takes things further and introduced Control Files, which allow a program to write to one area of the file and read a response from another. This starts to look a lot like methods on objects, something pointed out in Stephen Kell’s fascinating paper “The Operating System, Should There Be One?”, which also points out that this ad-hoc agglomeration of behaviour has left some semantic gaps:

As the filesystem’s use has expanded, its semantics have become less clear. What do the timestamps on a process represent? What about the size of a control file? Is a directory tree always finite in depth (hence recursable-down) or in breadth (hence readdir()-iterable)? Although some diversity was present even when limited to files and devices (is a file seekable? what ioctls does the device support?), semantic diversity inevitably strains a fixed abstraction.

The result is a system in which the likelihood of a client’s idea of “file” being different from the file server’s idea is ever-greater. It becomes ill-defined whether “the usual things” one can do with files will work. Can I use cp to take a snapshot of a process tree? It is hard to tell. The selection of what files to compose with what programs (and fixing up any differences in expected and provided be- haviour) becomes a task for a very careful user. Unlike in Smalltalk, semantic diversity is not accompanied with any meta-level descriptive facility analogous to classes.

— The Operating System, Should There Be One?, Stephen Kell

Yet another angle is taken by the Web, which names resources by URL, which, when fetched, return a representation. This could be a straightforward static read of bytes on a disk, but equally can be dynamically generated on demand, personalised for each client.

HTTP exposes a small set of verbs to operate over those resources, and HTML itself only really uses two (GET and POST), but really, the power of the Web model is that the behaviours exposed by the server are embedded in the resource itself. This enables a decentralised system, at a lack of meta-level standardisation.

There are clear parallels to the Unix file system structure in the design of URLs. Tim Berners-Lee wrote: “In many web browsers like the classic Apache, the URL space maps directly to chunks of the unix file system. This is deliberate as many good things come with the unix file system.” And the similarity to objects and message passing has been noted many times. What I hadn’t appreciated until recently is that Tim made the argument that resources subsumed not only files, but folders too:

It is crazy, if you think about it, that the whole screen is used to represent the information which happens to be on your local file system, using the metaphors of folders, while one window is used to represent the information in the rest of the world, using the metaphor of hypertext. What's the difference between hypertext and a desktop anyway? You can double click on things you find in either. Why can't I put folders into my hypertext documents? Why can't I write on the desk? Folders should be just another sort of document. My home page could be one, or it could be a hypertext document. The concepts of "folder" and "document" could be extended until they were the same, but I don't think that that would be necessarily a good idea. It's OK to have different forms of object for distinctly different uses.

— Consistent User Interface, Tim Berners-Lee

(As an aside, when I’m looking at open-source code, I often find myself preferring the Github file browser. My best guess as to why, is that it renders Readme documents at the same level as the folder contents, which makes navigating around large codebases, appropriately documented, much simpler.)

While there is a large overlap between objects, files, and resources, systems tend to treat them very differently; clients for file systems are focused around creating and managing files, whereas Web clients seem generally built around the assumption that users are browsing existing information, and objects are generally left to programming languages, or at best, programming environments like Smalltalk or Self.

As with the introduction of the proc filesystem, I think there’s a huge amount of power to be gained from providing a consistent model of interaction with addressable items in the system. Software could become simpler and more powerful by virtue of not being siloed by arbitrary type, and users would have a smaller mental burden. There are a couple of bright spots on the horizon: Google’s experimental operating system, Fuschia, seemingly takes inspiration from Plan 9 with its namespaces, and Upsin, a distributed file system for personal data, has APIs to encourage dynamic services being exposed as files.

I think that there could be some powerful, universal model of recording and accessing state in a system, and these 3 approaches are all converging on this point in the design space, albeit from very different angles. I believe that in the future it will be inevitable that these models are treated in a unified manner, and I think that computing systems that do will be conceptually much simpler than the fragmented systems of today.