We need to go back to the drawing board and design a referencing system that’s humanistic, grounded in how the source files are organized in the project structure (rather than relative to other files), and decoupled from where the documentation site is published.

As writers, here’s our wishlist for a page referencing system:

What we’re looking for is a readable, reliable, and maintainable system for creating references between pages, regardless of where the pages are published.

A reference to a page should really work like a mobile phone number. When you call someone’s mobile number, as long as they have their phone on them, your call will find that person. It doesn’t matter where they are located physically. They could be at the grocery store, their mom’s house, or driving down the road. Wherever. It’s the phone you’re calling.

Isn’t that like a page’s URL?

On the web, each page has a URL. At first glance, that seems like a easy way to reference a page. But URLs in documentation are surprisingly unstable. That’s because they’re dependent on the publishing environment. Consider the case of the same page being published to a staging URL, a protected beta release URL, and a public production URL.

This would be like your friend’s mobile phone number changing whenever they left their home. You’d have to know that your friend was at the grocery store at the moment you called them, and you’d need to know their grocery-store-mobile-number.

URLs are also unreliable as references because they often change when your organization reorganizes their site structure, renames a product, or decides to make URLs prettier by dropping the .html extension.

So a URL isn’t a unique or stable reference to a documentation page. It just locates one of many possible and ephemeral published instances of it. What we need is a way to reference the source of a page that ultimately links to that page’s destination, protected from the vagaries of publishing environments and URL strategies. Like a mobile phone number, we want to get to the source.

In order to reference a page, that page needs a unique and stable identifier. That identifier doesn’t change, even if page gets published to different places. Like a mobile phone number, when that identifier is “called”, the page answers, no matter where it’s published.

We’ll call a page’s “number” a page identifier, or page ID for short. The page ID is constructed from the properties of the source document that make it unique within our documentation site.

In order to define a page ID, the first thing we need is the standard project structure we laid out in the previous article. Without a standard structure, we can’t navigate far beyond pages which are immediate neighbors because we have no way to describe pages that live anywhere else.

Organizing our files into documentation components ensures that each page lives in well-defined, describable location. From this structure, each location derives unique, reliable coordinates we can use to refer to it. For instance, we now know what component, version, and module each page belongs to. All we have to do, then, is determine how to arrange those coordinates and when we need to use them.

Let’s consider the “coordinates” that make a source document unique:

Using this information, we can construct a fully-qualified page ID as follows:

Here’s an example of a fully-qualified page ID:

We can repurpose the inter-document xref macro in AsciiDoc to convert this page ID into a link:

Here’s how that looks in practice:

We’ve avoided coupling to the published URL by using a source-to-source reference. (Notice the page coordinate ends with .adoc, the file extension of an AsciiDoc source file). Regardless of whether you’re deploying your site locally, to staging, or to production, or you change URL strategies, the page ID always remains the same. The reference locks on to the target page and produces a URL that points to it wherever it gets published.

We’ve avoided coupling to the filesystem by using a location based on the documentation component structure. The page ID describes the source file’s project (component / version) and where in that project the source file is located (module / page).

We’ve eliminated the relative path (../../) problem by specifying the page as a module-relative path. The path always starts from a module’s pages directory, even when the referencing page is located inside a topic folder. If you move or rename a page within a module, you don’t have to change any references to other pages.

Of course, inbound references to the page you move do have to be updated. To counter this, you could pin the page ID of the page you want to move, thus adding more stability. That way, references to the page don’t have to be updated even when it moves. Though, a little help from the text editor to “refactor” references could make this abstraction unnecessary.

We’ve made it possible to validate and update references by using a well-defined pattern that’s easy for a script to locate, parse, and rewrite.

This human-friendly referencing system saves you from having to do computations in your head while writing. You just specify the coordinates of the page you want to reference. There’s no need to worry about the source file physical location on disk or its published URL. All you need to know are the names of your components, versions, modules, and pages so you can fill in this information.

Specifying the component and version every time seems like overkill. Surely there must be some sort of shorthand, right?

The full page ID sure seems like a lot to type each time you need to create a page reference. It also increases coupling when linking between pages of the same module or component, something you’ll do often. The problem is, the page ID often includes more information than is necessary. What we want to do instead is trim it down to just the essential information. We can do that by taking into account one of the most important aspects of a page ID: its context.

Let’s continue with the phone analogy. Think about when you’re at the office. You probably don’t need to use a complete telephone number to call someone in your office. There’s a company directory with extension numbers for every phone. You just punch in a short extension for the person you want to reach, and the call goes through. You’re benefiting from the two phones sharing the same context.

It’s the same way for page IDs. If you want to reference a page stored in the same module as the current page, which is most often the case, you only need to specify the filename of the other page.

If you want to reference a page in a different module, you only need to specify the module and filename of the other page:

To reference another version of a page in the same module, just prepend the version number to the filename.

You only have to use the full page ID when you need to refer to a page that belongs to another component.

The great thing about contextual references is that they are not coupled to the current component version. That means you don’t need to update the contextual page references when creating a new version of the component. All you’ll need to do is change the version number in the antora.yml file after you create your new version branch.

You may have noticed we’ve been using the terms component and version instead of repository and branch when discussing page references. Why the distinction? Aren’t they the same?

There are two reasons for these abstractions:

First, let’s look at how this abstraction reduces coupling.

Page IDs provide an intuitive and reliable way to reference pages, whether within the same module, another module in the same component, or a completely different component and version. By extending the inter-document xref macro in Asciidoctor to support page IDs, writers can use page IDs to make links between pages. Transcribing the page ID into a reference for frequently cited pages should quickly become habit for writers. When they forget, the text editor can lend a helping hand.

Since page references can be used to make links to pages anywhere in the site, they’re also a perfect fit for the navigation system. The navigation system can be designed to read AsciiDoc files composed of page references organized in a nested list.

Page IDs could also be used for page routing. A page could claim multiple page IDs. These page aliases would then be translated into URL routes that redirect to the master page. Such aliases might come in handy when a page is moved or removed, or for making vanity URLs.

Page references are just one example of how the AsciiDoc syntax can be extended to simplify a common documentation task. In the next article, we’ll look deeper into how well suited AsciiDoc is for writing and producing documentation and at additional ways we could build on the syntax to address common writing requirements.

Special thanks to Sarah White and Lisa Ruff for their substantial revisions to this article. Additional thanks to Sarah White for creating the intuitive diagrams featured in this article.

A standard project structure (or project layout) is a practice commonly used in software development. In Java projects, Java code is organized under src/main/java and src/test/java. Android projects build on this structure by assuming the manifest file is located in src/main and the resource files are located under src/main/res. In Ruby projects, Ruby code resides in lib and test. And the list goes on.

So why is a standard project beneficial? When a software project uses a standard structure, a developer can come into the project and know right away where to find the source code, how to build it, and how to run it. The same goes for the tools the developer uses.

Wouldn’t that be a great practice to apply to a documentation project?

Writers are looking for a writing system that does not involve a steep learning curve. After all, their job is to write, not be detectives. Right now, the lack of a standard structure means writers must dedicate a lot of time learning the ropes of each documentation project they join.

If a documentation project had a standard structure, writers would know where to find files, from documents to supporting assets. Like their developer counterparts, they could immediately be productive upon onboarding. The structure would encourage more contributions since this instant productivity would extend to community members as well. And tools could be developed to work with this structure, further improving writers’ productivity and helping them to keep the content organized.

In the words of Michael Moore: that’s a good idea, so we should steal it.

Here’s a documentation project structure we’ve put together based on our experience working with a myriad of documentation projects. It includes several sample files thrown in for context.

The structure in this listing represents a documentation component. You can think of a documentation component as a documentation project.

We know the structure in the example above represents a documentation component because of the presence of an antora.yml file.

When we find an antora.yml file in a repository, we expect to find the standard structure of a documentation component below it. Thus, the documentation can live anywhere in the repository. It also means it can share the same repository as the software it documents. This project structure is then repeated in each branch of each repository that hosts a documentation component.

The antora.yml file contains information about the component, such as its name, title, version, and navigation data.

All the other files in the component reside in the modules folder. Let’s open that folder and have a look inside.

A compelling benefit of a standard structure is interoperability with tools. Software platforms which have established a standard project structure enjoy strong tooling. This is no accident.

Here are some examples of tools that could be created if we had a standard documentation structure:

The more that tools can assume about how the content is organized, the more those tools can help you. If a standard project structure for documentation is adopted, tooling efforts around documentation would surely emerge.

Another benefit of a standard structure hailing from the software world is that it provides a system for making references.

Several site generators support sections, otherwise known as collections, for partitioning the content. Although these mechanisms fill a role similar to modules, they have the drawback of cutting off references between pages. What we need is a contract with the site generator to track a reference to another source file all the way through to publishing. Wherever that other file ends up, you want the site generator to construct the correct link to it. And it should be possible to make a reference not only to a file in another module, but one in another version or even another component. The foundation of such a reference system is a standard, addressable structure.

By using a standard structure that consists of components and modules, files are in a reliable and, more important, referenceable location. That means we can use that structure with AsciiDoc and create input source to input source references that span modules or components. We’ll talk in depth about expanding AsciiDoc’s xref capabilities in a later article in this series.

A standard project structure would bring many of the same benefits to documentation as it has to software.

No longer would there be confusion about where to find or append documentation files. Instead, writers would recognize the familiar structure and immediately be able to get down to work.

By offering a logical place for different types of files, the self-describing structure would help writers navigate the hierarchy and keep files organized. Tools can then be developed to work with this structure to streamline daily tasks and automated processes. Among these tools is the multi-repository, CD-friendly site generator we’re introducing in this series, which would be able to discover documentation components and know how to classify the files contained within them.

Whether we’re talking about the writers or the tools, all parties benefit from being able to rely on a standard project structure. Now that’s an idea worth stealing.

Special thanks to Sarah White and Lisa Ruff for their substantial revisions to this article. Additional thanks to Sarah White for creating the diagram featured in this article.

Most static site generators insist on centralizing the site’s source content. The generator reads local files that originate from the current branch of the site repository, then use that content to make a site.

That means everyone in an organization—​writers, engineers, marketing, community contributors—​must write into the same branch of the same repository. The result is lots of content kings but zero sovereignty. Who owns the content? Who resolves conflicts? Who decides when to release? What we’ve observed is that contributors just end up stepping on each other’s toes and everyone’s left frustrated.

One branch, one repository. Sometimes even a single folder. That’s the first major obstacle we encounter when using a typical static site generator.

One workaround is to make a copy of the disparate content and import it all into the sole branch of the site repository. But not only does that spawn a content monolith, it also makes the content divergent! And it grows hairier with every new product and product version that engineering releases.

If we weren’t hamstrung by a generator’s one branch, one repository rule, where would we store our documentation?

We could store it alongside the source code it describes. We could store the documentation for distinct software components in their own individual repositories. Perhaps we’d do both!

We don’t want to hesitate to store the documentation in as few or as many repositories as needed so the most applicable workflow, policies, file organization, versioning scheme, and team permissions can be applied to each unique documentation component. What we’re talking about here is modular documentation.

So, if our documentation components could be set free from a site generator’s one branch, one repository mandate, how would all of the documentation components get incorporated into a single, unified site?

We’ve rebelled against the one branch, one repository rule and restored the sovereignty of our documentation components. But we still need to create a documentation site from our sovereign components. Who, or what, is responsible for gathering the content from these components?

To adapt a typical static site generator to this paradigm, we might add a shell script that clones a bunch of repositories and dumps the content into a source directory to feed the generator. Or, perhaps, we could use an extension for the site generator that hooks into its lifecycle and pulls down content at build time. Maybe we jerry-rig a few scripts, a custom build, and a dash of manual copy-paste together to grab the right content, at the right time, in the right order, and convince the generator to incorporate that content into the site. Maybe.

Or, we could rethink this whole process.

Let’s head back to the whiteboard and conduct a thought experiment. What do we want a static site generator for documentation to do? What scenarios do we want it to handle?

The start of this process might look something like this:

This process could have potential. But before we go any further, let’s address another major obstacle: versions.

When we think about software, what immediately comes to mind?

Versions.

Since software is versioned, so must the content that documents it.

If we don’t align the content with the software version it’s documenting, clarity is lost and confusion fills the void. The user won’t know which version of the software the documentation they’re reading is for. The writer could inadvertently drop critical information about older software versions when updating the documentation for the latest release.

But versioning documentation raises more questions. Where should these versions live? How do we keep documentation for different software versions separate? Do we append a version number to the filename? Or do we put the files inside version folders?

Let’s think about how versions are handled in software. In software, we use branches and tags to specify and manage versions. Employing the practice of docs as code, we would then use branches to manage versions of the software documentation as well.

We could even store the documentation files in the code branches, thus truly embracing the docs as code philosophy.

Think about if we didn’t use branches to specify versions, but instead used folders labeled with version numbers in a single branch. We’d have to duplicate all of the documentation for a software component every time there was a new version. And we’d have no easy way to compare, manage, and merge different instances of a document. So branches are clearly the way to go.

Let’s continue the thought experiment and update our new site generator’s workflow now that we’ve considered documentation versions.

Fetching all the branches of a repository could be problematic. It’s doubtful we want the generator to use every branch, or to publish the content from every documentation branch to the production site. Some branches may even contain unedited drafts or content for unreleased software. On the other hand, writers and engineers working on a beta release probably want to see how their new documentation is looking.

In order to reconcile this conflict, and to avoid having to generate the whole documentation site just to preview one component, we want to be able to select different branches for different circumstances.

But how does the site generator know what content to find and how much of that content to fetch?

We don’t always want to generate everything, every time. If we’re able to gather content from sovereign documentation repositories, each with a set of version branches, we need to decide how much content we’re going to take.

At the most basic level, we need to tell the generator what documentation components we want, what branches we want, and where they’re located. We need a way to communicate instructions to the generator. And if we can communicate these instructions without having three to five years of scripting experience, that’s even better.

Taking another page from software development, this time the configuration as code practice, we’ll call this set of instructions a playbook.

Here’s a fragment of a playbook that defines the content sources for a documentation site:

Now that we’ve figured out how the site generator knows which documentation components and branches to fetch, let’s update our thought experiment.

A site generator designed to take instructions this way could be very versatile. Depending on the documentation site we need to generate, we could select or exclude specific versions of a documentation component.

For example, we might want to include beta documentation in an early release site:

And, as the example above shows, the generator could also incorporate local content we’re working on so we can generate a site locally and preview our work.

Or, we might want to take the latest version of one documentation component and make a microsite for one software product while still including that content in our main documentation site.

Having the ability to tell a site generator what content it should use gives developers and non-developers alike tremendous control over documentation sites. Now we can create a wide variety of sites from the same content. That’s content reuse at its finest. That’s also something that’s difficult (if not impossible) to do when all the content lives in the same branch of the same repository.

Let’s review the new capabilities and process we’ve specced out for our improved documentation site generator so far.

Here’s the whole process starting with the playbook and ending with the content organized in a catalog:

We believe this process fits with the way documentation is managed in the real world.

Documentation is written and stored in different places, is organized in different ways, and is owned and managed by different teams. Documentation isn’t centralized, it’s distributed!

With a site generator that can find and fetch content, content can reside in multiple repositories and organizations, in different hosts, and have different permissions and visibility. Teams can manage their own content, in their own way, keeping it current and under version control. Content could even come from local directories not under version control, including the worktree of a cloned repository.

We’re not done yet. If we’re going to challenge and improve the typical documentation site generation capabilities and workflows, we’ve only scratched the surface.

Here’s a few of the other obstacles we’ll tackle in this series of articles:

Special thanks to Sarah White and Lisa Ruff for their substantial revisions to this article. Additional thanks to Sarah White for creating the spectacular diagrams featured in this article.