===================================== LIST Implementations ===================================== The LIST specification can be implemented to varying degrees of "purity", depending on whether the system fully satisfies all architectural requirements or approximates them. We identify four primary implementation strategies: * **Native LIST Applications from Scratch:** Building software from the ground up to be natively LIST-compliant. Our LIST implementation as a virtual filesystem, called *LIST Filesystem Wrapper (LFW)*, is an example of this. * **Retrofitting LIST:** Adapting existing applications through plugins or middleware to intercept file operations and enforce immutable referencing. We denote this property as LIST-Compliancy (:math:`LC`), as seen in the adaptation of Zim Wiki into ZIM-LC [1]_. * A more surprising example of retrofitting LIST can be found in the chapter on :doc:`07_interesting_applications`, where we were able to make ``git`` a little bit more :math:`LC`. * **Ecosystem-Level Integration:** Implementing the specification within a controlled environment, such as an IDE (e.g., Emacs or VSCode). Here, the application acts as a surrogate OS, ensuring all internal tools and scripts share a unified referential and metadata logic. * **Holistic Pure LIST Systems:** A system-level integration where the OS kernel and all resident applications are natively compliant. This is, of course, the ideal dream, and it will resolve the *Hard Problem of Referencing* (:ref:`sec-hardness-of-referencing`) at all levels. .. LIST specification can be implemented as a filesystem, using databases, or even as a single document. Currently, we are providing a reference implementation as a virtual filesystem called LFW. For a database-based implementation, a document database appears the most natural approach (like MongoDB), but using PostGreSQL is perhaps better. For implementing it within a single document, we are currently implementing extensions and modules for ``LaTeX`` and Emacs org-mode files which will satisfy the LIST specification. In this chapter, however, we will be focusing our discussion on a reference implementation of LIST as a filesystem, called **LFW**. LFW: The Reference Implementation =================================== The **LIST Filesystem Wrapper (LFW)** is a virtual filesystem for Unix-based systems that implements the LIST specification. It employs a **"Laundry Basket" model**, allowing users to stash files without upfront organizational overhead. LFW is being developed in this repo: https://gitlab.com/thelistproject/lfw LFW transforms a traditional hierarchical filesystem to a laundry basket format that follows LIST specification. .. such as each entity has a unique UUID and is enriched with a strong typed metadata file, and so on. The following diagram shows how the traditional hierarchical filesystem is converted to LFW format. The files are first placed in a single directory and thus bringing a non-hierarchical form. During this conversion to non-hierarchical format, the files are enriched with metadata and immutable IDs. Further, the files are mounted to a mount directory to regain the hierarchical form for the user. .. Add link to list specification section here .. As seen in list specification section .. figure:: /images/lfw_architecture.png :width: 100% LFW Architecture - describing conversion of traditional hierarchical filesystem to LIST filesystem .. The workflow is organized into two primary phases: the conversion of traditional hierarchical data into a flattened storage format, and the construction of a graph-based indexing layer to facilitate the mounting of this structure as a virtual hierarchical file system. .. LIST specification can be implemented as a filesystem, using databases, or even as a single document. .. Currently, we are providing a reference implementation as a virtual filesystem called LFW. .. For a database-based implementation, a document database appears the most natural approach (like MongoDB), but using PostGreSQL is perhaps better. .. For implementing it within a single document, we are currently implementing extensions and modules for ``LaTeX`` and Emacs org-mode files which will satisfy the LIST specification. .. .. LFW: The Reference Implementation .. =================================== .. .. The **LIST Filesystem Wrapper (LFW)** is a virtual filesystem for Unix-based systems that implements the LIST specification. It employs a **"Laundry Basket" model**, allowing users to stash files without upfront organizational overhead. .. ------------------ Technical Workflow ------------------ The LFW is implemented into two following primary phases: * *Phase I*: The conversion of traditional hierarchical data into a flattened storage format (laundry basket model). * *Phase II*: The construction of a graph-based structure and mounting it to a virtual hierarchical file system. Let us look into these phases in detail. .. _Phase I Phase I: Conversion -------------------------------- We begin with converting the traditional hierarchical structure into a list format, such that all the content from our source directory is aggregated into a single root directory termed the **LFW root**. This conversion is executed exclusively during the initial system setup and is not repeated in subsequent runs. The process is systematic rather than a simple bulk copy. The conversion process initiates by verifying the existence of the LFW root directory, creating it if necessary. It then mandates that the LFW root be empty; this ensures that the resultiong LFW format contains only the intended files. Subsequently, the system prepares for migration by iterating through the source directory and performing the following operations. * **Entity Classification: Bundles and Singletons** This operation preserves internal dependencies during the flattening process by categorizing each entity in the source directory as either a **singleton** or a **bundle**. These classifications are defined as follows: * *Bundles*: Directories that must be preserved as a single unit to maintain internal dependencies - for example, Git repositories, Zim notebooks, or folders explicitly marked with a **.bundle** extension. * *Singletons*: Files that do not have strong dependencies and can be treated as independent entities. Note: Bundles are copied recursively to preserve their internal dependencies, whereas singletons are flattened directly into the LFW root as individual items. * **Unique ID Generation** This operation generates a unique name for the entity in each iteration. The LFW system replaces the file's original name with a UUID, while preserving the original filename in the corresponding metadata file to maintain traceability. This approach avoids naming collisions thus ensuring that multiple identically named files from the hierarchical folder structure can coexist in the LFW root. Moreover, the use of UUIDs ensures filename immutability, as the physical filename remains constant even if the logical metadata is modified. * **Metadata Association and Provenance** For every entity created in the LFW root - whether a **singleton** or a **bundle**, our LFW system generates a metadata file using the same unique identifier but with a **.md** extension. This file preserves the entity's original attributes, including its filename, absolute source path, creation timestamps, and user-defined tags. This metadata facilitates the linkability of files without altering the underlying data. Furthermore, this architecture supports a one-to-many mapping, allowing a single physical file to be logically associated with multiple directories. This is achieved by maintaining the list of folder paths within the metadata. .. * **Entity Classification:** Categorizes items as **Singletons** (independent files) or **Bundles** (directories with internal dependencies, e.g., Git repos). .. * **Unique ID Generation:** Replaces original filenames with UUIDs to ensure immutability. .. * **Metadata Association:** Creates a corresponding ``.md`` file for every entity to store original paths, timestamps, and tags. .. .. _Phase II Phase II: Mounting (Virtualization) ---------------------------------- In this phase, the LFW root is mounted to a user-specified location. To facilitate this, our LFW first builds a tree where all entries in the LFW root are represented as nodes. This graph-based architecture then allows the user to project the data into a hierarchical structure based on various criteria, such as original folder hierarchy or creation date, depending on the desired view. Unlike Phase I, the graph construction and subsequent mounting are performed for every mount request. In the following section, we discuss graph costruction and mounting it to a specified location. * **Building Graph based structure** In this section, we have detailed the process of building the graph based structure. To reproduce the original folder layout, the system first reads all metadata files in the LFW root to get the original filenames and folder paths. It then creates a series of nodes for the folders and sets the original filename as the leaf node of the tree. Each node stores key attributes, such as the file's UUID (used to locate the actual file in the LFW root), the node type (singleton or bundle), and any user-defined tags. The resulting graph consists of following nodes with key attributes attached to it: * *Root Node:* The entry point of the LFW filesystem. * *Intermediate Nodes:* Nodes representing directories in the classical hierarchical structure. * *Leaf Nodes:* Nodes representing singleton files or bundles. Note: Metadata files are not represented as nodes within the graph; instead, they are referenced via corresponding node attributes. * **Mounting as Virtual Filesystem** Once the graph is ready, the system uses the FUSE (Filesystem in Userspace) library to mount it as a virtual filesystem at a location chosen by the user. This turns the graph into a familiar folder structure that is easy to navigate. Beyond a fixed hierarchy, this graph-based architecture enables dynamic views organized by folder structure, creation date, or user-defined tags. Following the mount operation, the virtual filesystem supports standard POSIX operations such as open, read, edit, and rename, as well as the ability to view a file's metadata and all its logical locations. .. * **Graph Construction:** Builds a tree/graph from the LFW root using metadata. .. * **FUSE Integration:** Mounts this graph as a virtualed filesystem, allowing users to project the data into hierarchical views or dynamic clusters without altering the underlying UUIDs. .. ---------- Properties ---------- * **Reversibility:** LFW is designed to be reversible; users can revert to a standard hierarchical structure without loss of data or provenance. * **Decoupling:** The user interacts with a "view" while the system manages the "identity." .. rubric:: Footnotes .. [1] This is a work in progress and available in a public repository at https://github.com/shivams/zim-lc.