Version 0.2 of Gophermoon did send all output from stdout (standard out) and stderr (standard error) back to the socket. For error messages, that was not ideal, as Lua stack traces do not honor the Gopher protocol :)

In this version 0.3 I have added a logging channel for Gophermoon. Writing a log-file into the filesystem does not work, as Gophermoon runs inside an ephermeral container, and all changes to the file system are lost once the client request is done and the session terminates.

So logging is done via syslog. While it is possible to have a syslog-daemon (or systemd-journald) running inside the container, that would make the container more complex. My solution is to have the systemd unit catch all data from the stderr file descriptor and send that to syslog (and implicit to the systemd journal as well).

A small change in the Gophermoon systemd service unit (/etc/systemd/system/gophermoon@.service):

[Unit]
Description=GopherMoon Gopher Server in Lua

[Service]
ExecStart=/bin/systemd-nspawn -q -x --private-network -D /srv/gophermoon /bin/lua /bin/gophermoon
StandardInput=socket
StandardError=syslog

Standard input and standard output are still connected to the socket (from Systemd socket activation, see Part 1), but all data written into standard error will now end up in the Syslog/Journal.

Systemd needs to be made aware of the changes:

systemctl daemon-reload

The function write_log in Gophermoon is used to write a log entry:

--- write a log entry
function write_log(severity,msg)
  io.stderr:write(severity .. ": " .. msg .. "\n")
end

Example use from the source code:

[...]
write_log("INFO", "Selector is " .. selector)
[...]
   if isReadable(filepath) then
      send_textfile(filepath)
   else
      send_error("Selector [" .. selector .. "] not found")
      write_log("ERROR","Selector not found")
   end
[...]

The systemctl status command for the socket activation unit of Gophermoon gives a count of connections (successful and unsuccessful):

 * gophermoon.socket - GopherMoon - Gopher-Server in Lua
   Loaded: loaded (/etc/systemd/system/gophermoon.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Sa 2017-12-23 22:54:44 CET; 2 days ago
   Listen: [::]:70 (Stream)
   Accepted: 209; Connected: 0

Dez 23 22:54:44 gopher.defaultroutes.de systemd[1]: Stopping GopherMoon - Gopher-Server in Lua.
Dez 23 22:54:44 gopher.defaultroutes.de systemd[1]: Listening on GopherMoon - Gopher-Server in Lua.
Dez 23 22:54:44 gopher.defaultroutes.de systemd[1]: Starting GopherMoon - Gopher-Server in Lua.

Systemd creates a new service unit for each new connection, the unit name contains

• a counting number (number of connections since restart of the socket unit)
• local IP-Address and port of the current connection
• remote IP-Address and port of the current connection

Here is a Lua-Error message and stack trace from stderr in the Systemd journal:

# journalctl  -u "gophermoon@*"
Dez 26 19:21:28 gopher.defaultroutes.de systemd[1]: Started GopherMoon Gopher Server in Lua (185.22.143.172:49828).
Dez 26 19:21:28 gopher.defaultroutes.de systemd[1]: Starting GopherMoon Gopher Server in Lua (185.22.143.172:49828)...
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: Info: Selector is /0/filedoesnotexists
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: /bin/lua: /bin/gophermoon:37: attempt to index a nil value (global 'f')
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: stack traceback:
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: /bin/gophermoon:37: in function 'isDir'
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: /bin/gophermoon:134: in main chunk
Dez 26 19:21:29 gopher.defaultroutes.de systemd-nspawn[29973]: [C]: in ?
Dez 26 19:21:29 gopher.defaultroutes.de systemd[1]: gophermoon@172-5.45.107.88:70-185.22.143.172:49828.service: main process exited, code=exited, status=1/FAILURE
Dez 26 19:21:29 gopher.defaultroutes.de systemd[1]: Unit gophermoon@172-5.45.107.88:70-185.22.143.172:49828.service entered failed state.
Dez 26 19:21:29 gopher.defaultroutes.de systemd[1]: gophermoon@172-5.45.107.88:70-185.22.143.172:49828.service failed.

A list of all Gophermoon service names and connection information.

# journalctl -o verbose -u "gophermoon@*" | grep "_SYSTEMD_UNIT" | tail
    _SYSTEMD_UNIT=gophermoon@199-5.45.107.88:70-212.111.43.252:60215.service
    _SYSTEMD_UNIT=gophermoon@200-5.45.107.88:70-212.111.43.252:47726.service
    _SYSTEMD_UNIT=gophermoon@201-5.45.107.88:70-212.111.43.252:18139.service
    _SYSTEMD_UNIT=gophermoon@202-5.45.107.88:70-212.111.43.252:14576.service
    _SYSTEMD_UNIT=gophermoon@203-5.45.107.88:70-212.111.43.252:62963.service
    _SYSTEMD_UNIT=gophermoon@204-5.45.107.88:70-212.111.43.252:21729.service
    _SYSTEMD_UNIT=gophermoon@205-5.45.107.88:70-185.22.143.172:50432.service
    _SYSTEMD_UNIT=gophermoon@206-5.45.107.88:70-185.22.143.172:50433.service
    _SYSTEMD_UNIT=gophermoon@207-5.45.107.88:70-185.22.143.172:50434.service
    _SYSTEMD_UNIT=gophermoon@208-5.45.107.88:70-185.22.143.172:50436.service

In a later blog post I will show how to parse and aggrgate the connection information for some nice statistics (available via Gopher protocol of course).

Git commit for today: https://github.com/cstrotm/gophermoon/commit/ead8b366c639034b34884b17fb5c4fb28211c012

the sourcecode is now on Github @ https://github.com/cstrotm/gophermoon

With every blog post, I will post the link to the commit associated with the blog post.

Today Gophermoon learned how to serve static text files. The program first looks at the selector send by the client. If the selector is empty, and uses the root directory.

--- read the selector from the client
selector = io.read()

--- remove CR (13/$0D) from input
selector=selector:sub(1, -2)

--- if an empty selector has been send, use the root
if selector == '' then
   selector = "/"
end

--- make selector relative to the gopher root
filepath = gopherroot .. selector

Next, if the selector points to a directory, it looks if there is a file names gophermap in the direcory. A gophermap file is a Gopher-Document describing the resources available in this directory (as well as links outside the directory). Gophermoon uses the same gophermap format as Gophernicus (see https://github.com/prologic/gophernicus/blob/master/README.Gophermap)

Welcome to GopherMoon @ defaultroutes.de
----------------------------------------
0Gophermoon 0.1 - a Gopher-Server in Lua -- Part 1      /gophermoon01.txt       blog.defaultroutes.de   70
0Gophermoon 0.1 - a Gopher-Server in Lua -- Part 2      /gophermoon02.txt       blog.defaultroutes.de   70
0Gophermoon 0.2 - a Gopher-Server in Lua -- Part 3      /gophermoon03.txt       blog.defaultroutes.de   70
0Gophermoon Sourcecode  /gophermoon     blog.defaultroutes.de   70

Lines that do not contain a tabulator are formatted as "i" (Information) file type.

--- if the selector points to a directory,
--- do we have a "gophermap" file? If "yes",
--- send that file. Else, if the selector
--- points to a (text-) file, send it
--- if both are false, send an error message
if isDir(filepath) then
   gophermap = filepath .. "/gophermap"
   if isReadable(gophermap) then
      send_gophermap(gophermap)
   end
else
   if isReadable(filepath) then
      send_textfile(filepath)
   else
      send_info("Welcome to GopherMoon @ defaultroutes.de")
      send_info("----------------------------------------")
      send_info("Error: Selector not found")
      send_end()
   end
end

If the selector isn't a directory, the program checks the selector is a readable file, and if it is, it sends the file gopher-style (with CR/LF and terminating the file with ".").

--- send a Textfile Gopher-style
function send_textfile(filepath)
   local f=io.open(filepath,"r")
   if f~=nil then
      for line in io.lines(filepath) do
	 io.write(line .. "\r\n")
      end
      io.close(f)
   end
   send_end()
end

The commit from today is https://github.com/cstrotm/gophermoon/commit/f300e28677059c9364151a89b36afe552d858700

I will not change the (minimal) Gopher server (this time), but will prepare the execution environment for Gophermoon: the result will be a container containing, only one executeable and one Lua-Script besides the Gopher content.

The container will be isolated from the network and the filesystem and process-space of the host machine.

So any coding errors I will create while expanding the Gophermoon-Server will have limited security impact on the host system (unless there is a security issue in the container code of the Linux-Kernel, which there have been in the past. Nothing is fully secure).

I'm testing on a RedHat EL 7 machine, but the same result should be possible on other modern Linux systems with a container manager (systemd-nspawn, docker, rkt, LXC …).

One feature that makes the Go programming language popular among users of Linux-Containers is the fact that Go produces static binaries. Static binaries have no code runtime dependencies, they are self contained and do not need to be installed, but can just copied around and "just work".

Lua is written in C, but we can create static binaries in C too.

It is recommended to compile the static Lua binary on an development machine, not on the production Gopher server.

For Red Hat based systems, we need to install few build tools and the static library files for the GNU-Libc:

yum install make gcc wget glibc-static

Next, we're downloading the Lua sourcecode from https://www.lua.org/download.html

mkdir ~/src
cd ~/src
wget https://www.lua.org/ftp/lua-5.3.4.tar.gz
tar xfz  lua-5.3.4.tar.gz
cd lua-5.3.4/src

The option -static lets the gcc compiler and linker build a static binary. Change the MYCFLAGS and MYLDFLAGS lines in Makefile:

[...]
MYCFLAGS= -static
MYLDFLAGS= -static
[...]

Now we can build the new Lua interpreter. I'm building the posix flavor, which is more generic than the Linux flavor and links less external libraries:

make posix
strip lua

The resulting lua binary should be a static ELF executable:

# file lua
lua: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=0631bb0d50ad1209dbf55f2b1dd0eb60fc388df7, stripped
# ls -lh lua
-rwxr-xr-x. 1 root root 1,7M 23. Dez 23:04 lua

Next the container root directory is created and the static lua interpreter together with the gophermoon script is placed into that container directory:

mkdir -p /srv/gophermoon/bin
cp lua /srv/gophermoon/bin/
mv /usr/local/sbin/gophermoon /srv/gophermoon/bin/

Without timezone information files within the container directory, systemd-nspawn will complain about not being able to update the timezone in the container

Timezone <your-host-timezone> does not exist in container, not updating container timezone.

So far, I found no better way than to copy the timezone information into the container root (only one file is enough if you know the timezone of your host machine):

mkdir -p /srv/gophermoon/usr/share/zoneinfo
cp -a /usr/share/zoneinfo/* /srv/gophermoon/usr/share/zoneinfo/

A quick manual test starting the container from the commandline:

# /bin/systemd-nspawn -q --private-network -D /srv/gophermoon /bin/lua /bin/gophermoon

iWelcome to GopherMoon @ defaultroutes.de               defaultroutes.de        70
i----------------------------------------               defaultroutes.de        70
.

Here is the updated Systemd-Service-Unit (/etc/systemd/system/gophermoon@.service):

[Unit]
Description=GopherMoon Gopher Server in La

[Service]
ExecStart=/bin/systemd-nspawn -q -x --private-network -D /srv/gophermoon /bin/lua /bin/gophermoon
StandardInput=socket

The -x option creates an ephemeral container that is started from an BTRFS snapshot each time a new gopher connection comes in. The snapshot is destroyed as soon as the container terminates. An intruder will not be able to store new files into the container system.

At last we reload the new unit into Systemd and restart the Socket-Activation:

systemctl daemon-reload
systemctl restart gophermoon.socket

Now gophermoon runs in a (more) secure environment.

I'm writing a simple Gopher-Server in Lua. I want to play around with the Gopher Protocol, Systemd and Linux-Container and learn some Lua-Programming on the way.

Gopher is a document retrieval protocol that had been around the same time the world-wide-web was born, but it is much simpler that HTTP and HTML. It works with plain text and directories, and the protocol on the wire is ultra-simple.

The server will be named gophermoon.

Gopher (the protocol) is documented in RFC 1436 https://www.ietf.org/rfc/rfc1436.txt with some extension defined as Gopher+ in http://iubio.bio.indiana.edu/soft/util/gopher/Gopher+-spec.text.

In the beginning, I will use Systemd to do the network part of the protocol (listening on port 70). The Lua program will just read and write to standard-input and -output.

Here are the Service-Unit and the Socket-Activation-Unit for Systemd. The Gophermoon Service is in /etc/systemd/system/gophermoon@.service

[Unit]
Description=GopherMoon Gopher Server in La

[Service]
ExecStart=-/bin/lua /usr/local/sbin/gophermoon
StandardInput=socket

• and the Socket-Unit is stored in /etc/systemd/system/gophermoon.socket

[Unit]
Description=GopherMoon - Gopher-Server in Lua

[Socket]
ListenStream=70
Accept=yes

[Install]
WantedBy=sockets.target

Below is a very simple Lua-Script implementing a Hello-World Gopher-Service. It just emits the lines for a static welcome message.

A Gopher-Server waits for a connection and reads the path the client sends (ignored for now). Then it writes the Gopher-Menue out. Each Menu-Line has five fields. The first field is 1 character wide, the other fields are separated by the tabulator character (/t). Each line is terminated by a CRLF sequence.

A line with a single dot "." marks the end of the communication, server and client will close the connection.

--- Gopher Moon Version 0.1
--- a simple Gopher Server in Lua
--- with some help from Systemd socket activation

path = io.read()
io.write("iWelcome to GopherMoon @ defaultroutes.de\t\tblog.defaultroutes.de\t70\r\n")
io.write("i----------------------------------------\t\tblog.defaultroutes.de\t70\r\n")
io.write(".\r\n")

Systemd needs to know about the new unit-files, so we do a reload:

systemctl daemon-reload

Now we can enable and start the socket (no need to start/enable the service, as it will be started once a connection to Port 70 is made).

systemctl enable --now gophermoon.socket

The open socket on the Gopher-Port 70 now visible:

# lsof -i
COMMAND     PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
systemd       1    root   40u  IPv6 3989631      0t0  TCP *:gopher (LISTEN)
[...]

Gopher is simple, it can be tested with the telnet command:

# telnet blog.defaultroutes.de 70
Trying 5.45.107.88...
Connected to blog.defaultroutes.de.
Escape character is '^]'.

iWelcome to GopherMoon @ defaultroutes.de       blog.defaultroutes.de        70
i----------------------------------------       blog.defaultroutes.de        70
.
Connection closed by foreign host.

First we save the GUID (Globally Unique Identifier) Partition Table (GPT), so that we can restore the same partition table on the target machine. We use the commandline version of gdisk for this, named sgdisk

sgdisk --backup=/srv/macos/parttable.gpt.sgdisk /dev/sda

The content of the partitions can be cloned using the partclone tool. A default MacOS X installation has three partitions:

1. EFI system partiton
2. MacOS X (named "Customer")
3. Recovery partition

Model: ATA APPLE HDD HTS547 (scsi)
Disk /dev/sda: 976773168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start       End         Size        File system  Name                  Flags
 1      40s         409639s     409600s     fat32        EFI system partition  boot
 2      409640s     975503591s  975093952s  hfs+         Customer
 3      975503592s  976773127s  1269536s    hfs+         Recovery HD

First we create backup images of all partitions:

partclone.vfat -I -c -s /dev/sda1 | gzip > /srv/macos/sda1.partclone.gz
partclone.hfsplus -I -c -s /dev/sda2 | gzip > /srv/macos/sda2.partclone.gz
partclone.hfsplus -I -c -s /dev/sda3 | gzip > /srv/macos/sda3.partclone.gz

To restore the saved MacOS X installation, boot another Mac using Linux (a Knoppix live Linux DVD will work).

Next we restore the partition table (GPT) using sgdisk (/dev/sda is the target disk, all data on that disk will be erased, be warned!):

sgdisk --load-backup=/srv/macos/parttable.gpt.sgdisk /dev/sda
partprobe

as an alternative, the partitions can be created using parted and the GUID type codes set by sgdisk (important!):

sgdisk --zap /dev/sda
parted -s /dev/sda mklabel gpt
parted -s /dev/sda mkpart primary 40s 409639s
parted -s /dev/sda name 1 "'EFI system partition'"
parted -s /dev/sda set 1 boot on
parted -s /dev/sda mkpart primary 409640s 975503591s
parted -s /dev/sda name 2 "'MacOS X System'"
parted -s /dev/sda mkpart primary 975503592s 976773127s
parted -s /dev/sda name 3 "'Recovery HD'"
sgdisk -t 1:C12A7328-F81F-11D2-BA4B-00A0C93EC93B /dev/sda
sgdisk -t 2:48465300-0000-11AA-AA11-00306543ECAC /dev/sda
sgdisk -t 3:48465300-0000-11AA-AA11-00306543ECAC /dev/sda
partprobe

Restore the partition content:

zcat /srv/macos/sda1.partclone.gz | partclone.vfat -r -o /dev/sda1
zcat /srv/macos/sda2.partclone.gz | partclone.hfsplus -r -o /dev/sda2
zcat /srv/macos/sda3.partclone.gz | partclone.hfsplus -r -o /dev/sda3

If all went well, the new disk is ready to boot MacOS X.

For reference, here are the GUID details of a default MacOS X 10.8 MacBook Pro install:

root@(none):~# sgdisk -p /dev/sda
Disk /dev/sda: 976773168 sectors, 465.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 497736B0-7EA0-4C45-AB5F-8841CD773D24
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 8-sector boundaries
Total free space is 262157 sectors (128.0 MiB)

root@(none):~# sgdisk -i1 /dev/sda
Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI System)
Partition unique GUID: 6749C82F-02C9-4FF5-A889-F31A21726F8E
First sector: 40 (at 20.0 KiB)
Last sector: 409639 (at 200.0 MiB)
Partition size: 409600 sectors (200.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'EFI system partition'

root@(none):~# sgdisk -i2 /dev/sda
Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
Partition unique GUID: 60F7A3AA-69B3-4E59-87A0-3A47BB659255
First sector: 409640 (at 200.0 MiB)
Last sector: 975241447 (at 465.0 GiB)
Partition size: 974831808 sectors (464.8 GiB)
Attribute flags: 0000000000000000
Partition name: 'MacOS X System'

root@(none):~# sgdisk -i3 /dev/sda
Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
Partition unique GUID: FD007AA4-CF3A-42F6-BFC6-B3BC25521FC2
First sector: 975503592 (at 465.2 GiB)
Last sector: 976773127 (at 465.8 GiB)
Partition size: 1269536 sectors (619.9 MiB)
Attribute flags: 0000000000000000
Partition name: 'Recovery HD'