Converting hugo sites to Gopher Hole and/or to Gemini Capsule

📁 tools
🏷 hugo • Gopher • Gemini

Introduction

If you are implementing your web site using Hugo site, you may want to consider posting the same content as a Gopher hole and/or as a Gemini capsule. This post will describe a tool to do exactly that, called Hugo-2-Gopher-and-Gemini. We will build an hypothetical web site in Hugo using a the mini theme, and will generate a Gopher hole and as a Gemini capsule from the site.

In the old days of the Internet (circa 1990), when I started using it at work. We relied on email, mailing lists (and its faqs), Usenet and FTP. Then it came Gopher (RFC 1436) that was much better that FTP, because it can link multiple sites. But, then the World Wide Web took over. In any case, the Gopher protocol still in use today and there are some active sites (called Gopher holes).

A modern attempt to rescue some of the advantages of Gopher is the Gemini protocol described by the Gemini Project. It is described as heavier than Gopher but lighter than the web, but does not attempt to replace either of them. A Gemini site is called a capsule. The advantages of Gopher and Gemini is the lack of cookies, trackers and advertisement; which help with privacy.

There are few use cases in which you want to use Hugo-2-Gopher-and-Gemini, including:

  • A person with a current Hugo generated site that want to have presence in Gopher space or/and in Gemini space, while maintaining the original web site using Hugo. In this situation the person can use Hugo to maintain all the sites (the web site, the Gopher hole and the Gemini capsule). Most content will be common between them, but other content may be specific to one or two of the sites.
  • A person that wants to use Hugo to generate and maintain a Gemini capsule or Gopher hole.
  • A person that wants to migrate a current Hugo web site to a Gemini capsule or a Gopher hole.

Notation

To simplify the instructions, I will use the following notation:

Any time that I describe a command to type in the command line, I will use a $ as the prompt of the line. Before the $ will come the folder (directory) in which the command line is is open. Therefore, you only need to type the command after the $. For example:

~$ ls
~$ cd tmp
tmp$ ls

There are three commands here:

  1. ls that should be typed in the user folder.
  2. cd tmp that should also be typed in the user folder. In here we are asuming that there is tmp folder in user folder.
  3. ls that should be typed in the tmp folder (assuming it exits)

Creating a Hugo Web site

This is a very detailed section and therefore long, but good if you are not familiar with Hugo. The reason for this section is to have a sample web site to test the conversion process. This not intended to be a good Hugo introduction, but just to walk you to the creation of a very simple site. However if you are familiar with Hugo, you can safely skip it.

In here we will follow the instructions in hugo’s quick start and will use the mini theme. I assume that you have already installed Hugo and git. They are both required to work with Hugo to create a Web site.

First, open a command line terminal and navigate to a folder (directory) where you will develop your Web site. For this example, I will create a folder called source, and will work in that folder, as follows:

~$ mkdir source
~$ cd source

CLI

Then, we will create the site called nice-site using Hugo.

~$ hugo new site nice-site

That will create the site and provide some useful information. We will then, move into the site folder (nice-site), and will initialize git.

~$ cd nice-site
nice-site$ git init

Now we need to select a theme for our site. In this example, we will use the mini theme, and so we use the following git command:

nice-site$ git submodule add https://github.com/nodejh/hugo-theme-mini.git themes/mini

Note that we use git submodule and not git clone, because we want to place the mini theme in the theme folder, but we don’t want ownership of mini. In addition, we don' want to put a git project inside another git project. However, doing that means that if we want to upgrade the mini theme, we need to do it manually (using git submodule update --remote --merge). But, for now we are OK and need to work on the configuration of our site (file config.toml).

We need to add a line to the file config.toml located in the base folder (nice-site/config.toml). That file was created by Hugo when we created the new site nice-site above. Now, we need to tell Hugo which theme we are using (in other words the name of the folder under nice-site/themes/'). So you need to edit the file config.toml` and add the following line :

theme = "mini"

The location of the line does not matter, so just added as the last line of the file. For this example, we will just do that to the file. However, in a real situation, you will look into the config.toml file located in the theme and adjust the main config.toml file to customize it to your liking. The content of the file, will probably look something like this:

    baseURL = "http://example.org/"
    languageCode = "en-us"
    title = "My New Hugo Site"
    theme = "mini"

Now you can test the resulting basic web site. For that just execute hugo server from the command line, as follows:

nice-site$ hugo server

This will create the nice-site in memory and will be exposed in the local machine. Therefore, you want to use a browser and use the URL given by Hugo (probably http://localhost:1313/)to check that everything is working until here.

Adding content to the Hugo site

To understand how the tool (Hugo to Gopher and Gemini) works, we need to add some content to the site. We will add an about page and five posts, as follows:

nice-site$ hugo new about.md
nice-site$ hugo new posts/some-good-content.md
nice-site$ hugo new posts/good-gopher-content.md
nice-site$ hugo new posts/good-gemini-content.md
nice-site$ hugo new posts/good-web-content.md
nice-site$ hugo new posts/more-content.md

Note that this created the five posts under the folder nice-site/content/posts, and the About page (nice-site/content/about.md). At the beginning of each of those files there is front matter with some metadata about the post. They will look something like this:

    ---
    title: "Some Good Content"
    date: 2022-12-13T15:23:49-08:00
    draft: true
    ---

The draft: true metadata indicates that the post is not ready to be posted, and so will not appear in the site, because it is a draft. Therefore, if you execute hugo server, you will not see any of the files. Just give it a try.

Now let edit each of those files to do two things. First let change draft to false or delete the draft line, and second let add some content. On the about.md file, you may want to delete the draft and the date lines, leaving only the title. Then you can add content to the about page. For sake of space, I will just show you one of the files, with my modifications. But you should do similar changes to the other files. The resulting file looks like this:

    ---
    title: "Some Good Content"
    date: 2022-12-13T15:23:49-08:00
    draft: false
    ---

    # Introduction

    This is some good content. 

    This is a link to [my other site](https://www.mkamarin.com/).

    This is a ![image file](/image01.png)

    That is all.

Now we can again execute the web site in memory, using hugo server.

nice-site$ hugo server

Now you want to use a browser and the URL given by Hugo (probably http://localhost:1313/) to check that everything still working. Note that the public folder is empty. This is because we have only tested the site in memory. So, if you do an ls public there is no content.

nice-site$ ls public

When you want to host your web site, you need to generate it using hugo and then copy the public folder to the hosting machine. Let now generate the site and list the generated files, as follows:

nice-site$ hugo 
nice-site$ ls public

Now that you have a our web site working and some test data, we can move to generate a Gopher hole and a Gemini capsule.

Hugo-2-Gopher-and-Gemini

In this section, we continue using the sample web site, we created in the previous sections for our examples.

The tool Hugo-2-Gopher-and-Gemini is implemented as a Hugo theme. Therefore it needs to be installed as an additional theme, which is OK because Hugo will use the configuration file to know which theme to use. The design of Hugo allows you to have multiple themes for the same project.

The idea here is to have at least two themes. Your web site theme (which following our example is “mini”) and the Hugo-2-Gopher-and-Gemini theme. Your main theme will generate the site in the public folder, and the Hugo-2-Gopher-and-Gemini theme will generate the Gopher hole and/or the Gemini capsule under the public-gg folder. Therefore, this requires two distinct executions of hugo, namely one for your regular web site theme, and one for the Hugo-2-Gopher-and-Gemini theme. There will be no changes or side effects to your original Hugo site or files.

Installation

First, you need to add the Hugo-2-Gopher-and-Gemini theme to your current hugo site, just do:

nice-site$ git submodule add  https://github.com/mkamarin/Hugo-2-Gopher-and-Gemini.git themes/Hugo-2-Gopher-and-Gemini

Now, you will have at least two themes under the themes folder. In case that you want to update any of the submodules (meaning all the themes that you have installed using git add submodule), you should use git submodule update --remote --merge.

Second, you need to create an empty folder called layouts-gg. This is done to avoid Hugo from using the content in your main layouts folder.

nice-site$ mkdir layouts-gg

Third, you need to copy and edit the config-gg.toml file to the main folder to adjust it to your situation. You can just do:

nice-site$ cp themes/Hugo-2-Gopher-and-Gemini/config-gg.toml .

Now you need to edit the config-gg.toml file to customize it for your situation. The end result is that you have two distinct config files, namely one for your main theme (config.toml) and one for Hugo-2-Gopher-and-Gemini (config-gg.toml).

Using Hugo-2-Gopher-and-Gemini

The tool is implemented in python and it is located in themes/Hugo-2-Gopher-and-Gemini/src/hugo2gg.py. The output will be generated in a folder called public-gg. Assuming you generate both Gopher and Gemini, then the public-gg folder will have two sub-folders, one for the Gopher hole named public-gg/gopher and one for the Gemini capsule named public-gg/gemini.

You may want to start by executing the tool with the –help command to understand your options.

nice-site$ python3 themes/Hugo-2-Gopher-and-Gemini/src/hugo2gg.py --help

Note that python3 is optional if you make the script executable.

This tool relies on executing Hugo using config-gg.toml and the Hugo-2-Gopher-and-Gemini theme. The tool itself can invoke Hugo, but you have the alternative of executing Hugo yourself. Therefore, there are at least two alternatives to use the tool, as follows:

Alternative one is to just let the tool invoke Hugo. Let assume that you are OK with all the default options, so just do:

nice-site$ python3 themes/Hugo-2-Gopher-and-Gemini/src/hugo2gg.py --type all

Alternative two is to execute Hugo manually first and the the tool.

nice-site$ hugo --config config-gg.toml --destination public-gg --layoutDir layouts-gg --disableKinds sitemap
nice-site$ python3 themes/Hugo-2-Gopher-and-Gemini/src/hugo2gg.py --type all --no-hugo

Independent of the alternative you used, you will have the output in public-gg (unless you selected a different location).

Now you can check the generated files. By default, the Gopher hole will be in public-gg/gopher and the Gemini capsule in public-gg/gemini. You may want to browse those directories using the Gopher-and-Gemini-Walker tool before deploying them to your hosting environment.

Pages front matter

In some cases, you may want some pages to be treated different when generating them for Gopher or Gemini. In those cases, you can add to the page front matter some of the flags described in Hugo-2-Gopher-and-Gemini/archetypes/default.md. For example, you may have a post with the following front matter:

---
title: "Example post"
date: 2022-12-12T18:08:56-08:00
description: "some description"
draft: true
outputs: ["gemini","gopher"]
ggKeepRaw: false
ggRemoveExtras: false
ggCopyPage: false
ggIgnoreLinks: false
---

Those flags are:

outputs

Outputs is a set of outputs. Outputs is used to limit the generation of the page to one or more of “html”, “rss”, “gemini”, “gopher”. For example, if you want a page to be generated for gopher only, then use: ‘outputs: [“gopher”]’

The syntax is:

outputs: ["html", "rss", "gemini", "gopher"]

ggKeepRaw

Keep raw flag will just keep the raw page as it was written. In other words, no conversion process will be done in the page. Useful if the page was written specifically for Gopher or for Gemini and you use the corresponding outputs (above)

For example:

outputs: ["gopher"]
ggKeepRaw: true

ggRemoveExtras

The remove extras flag will not generate extras for this page. Extras are controlled in the config.gg.toml under [params.gopher] and [params.gemini]. Their names start with include. They are Menu, Categories, Social, Return home, and Author. This is useful if you wants a specific page to be different.

You can use it as:

ggRemoveExtras: true

ggCopyPage

The copy page flag will force hugo2gg.py to copy this page from the ‘last’ directory. This uses the hugo2gg.py --last option (default to public-gg-sav). Useful when you manually modify a generated page and want to keep your changes. In that case you will copy your modified site to a directory, and execute hugo2gg.py with the correct --last option.

You can use it as:

ggCopyPage: true

The ignore links flag will ignore the links in the page. Links will not be generated for that page. Useful for pages where the links are not that important, and you don’t want to include them in Gopher or Gemini.

You can use it as:


ggIgnoreLinks: true

Other tips for content creation

  1. If you plan to have content that is specific to Gopher and/or Gemini, you may want to use a naming convention for those posts. For example, you may name those posts <name>-gg.md and write them in a way that will look nice in Gopher and/or Gemini. In that case, you want to add the following line to your main config.toml file:
ignoreFiles =  ["-gg.md$"]

Then, when you want to create content that should only be in Gopher and/or Gemini, you will create the markdown file as follows:

~$ hugo new posts/name-gg.md
  1. You may find the Gopher-and-Gemini-Walker tool useful to verify the generated Gopher hole and/or the Gemini capsule before deploying them.