What is Serial-over-LAN?

Screenshot of terminal showing Serial over LAN session to a Linux system installer

On some of Privex's dedicated servers, instead of a graphical KVM (keyboard video mouse) for remote management outside of the OS (e.g. during OS installation, accessing the BIOS/UEFI, accessing a boot menu, etc.) - we often only have Serial-over-LAN.

Serial-over-LAN (or "SOL" for short) is effectively a virtual serial port, which works similarly to the classic RS232 COM ports. To the OS (e.g. Linux), the SOL controller will often show up as a normal serial port with console capabilities, on Linux it'd normally show up as /dev/ttyS0 (Serial Terminal 0).

Photo of an RS232 Serial/COM Port

The system and OS can then use the serial console as a terminal, outputting what's displayed on the screen as a series of characters and special control characters sent to the SOL serial port, which are then translated into TCP/UDP packets and sent over the IPMI's network connection to any connected clients (such as ipmitool).

On most systems, Serial-over-LAN is handled by their IPMI (IP Management Interface) controller, sometimes referred to as a BMC (Baseboard Management Controller),

What is an IPMI Controller / BMC?

What is IPMI / BMC?

An IPMI Controller (sometimes known as a BMC) is a special set of ICs on the motherboard or PCI card that can access and control various aspects of a system, and allows the system operator(s) some form of remote management, generally over a LAN or the internet - either via a WebUI, or the standard IPMI protocol (RCMP+) which operates over UDP.

What can an IPMI controller do?

Depending on the specific controller, a BMC could support a variety of features, such as:

• Power Control (turning the system on/off or resetting it)
• Fan readings / control
• Temperature monitoring
• Ability to configure various settings that would normally need to be configured in the BIOS, such as drive boot order, RAID controllers, or CPU profiles.
• Some IPMI controllers even have some form of built-in KVM, whether it's a full graphical KVM capable of emulating a mouse, keyboard, and monitor - or a simple Serial-over-LAN feature, allowing you to access the system over a virtual serial port.
• Many other useful features for remote management and monitoring, depending on the specific IPMI controller.

Common Configurations to enable Serial on bootloader's and OS's

Note about ttyS0,9600n8 serial config format

9600n8 is short for:

       Speed: 9600 bits per second
 Parity bits: n = no parity bit
   Data bits: 8 bits per character
   Stop bits: N/A (default: 1)
Flow control: N/A (default: No Flow Control)

On most kernels, the Stop Bits are assumed to be 1 and would be ignored if specified, thus most kernels expect <speed><parity><data_bits><flow_control>

Speed refers to the baud rate that your serial console runs at. Similar to the speed of your home internet, baud refers to how many individual data bits per second can flow over the serial connection at the most. With serial connections, it's very important that both the server (your OS / bootloader / BIOS) and client (e.g. ipmitool) are both configured with the same baud rate, otherwise things will go wrong.

1 baud = 1 bit per second, and generally 8 bits = 1 byte, thus 9600 baud = 9600 / 8 = 1200 bytes per second, or 1.2kbyte/s. Generally, 56k (57600 baud) is considered the fastest most serial connections can run at, though some systems may be capable of 115,200 baud. Most servers have their SOL baud rate default to 9600, thus if you're not sure what baud rate your system and SOL client expects, try 9600 first.

Parity bits are a form of error detection / correction. Using parity only mattered back when physical serial cables were used, and the cables were either very long and/or poor quality, and your usecase required the ability to detect potentially corrupt bytes at the serial layer. In the case of Serial-over-LAN, physical serial cables are no longer used, instead, the data would be transmitted over the network using robust RJ45 / SFP+ network cables, with very little risk of bytes getting corrupted along the way.

Using the 9600n8 format, parity can be set to either n (do not use parity bits), e (enable single parity bit, with positive parity bit meanining an even number of bits), or o (same as e but positive parity meaning odd number of bits instead).

Data bits define how many bits there will be per byte. This can be set to either 7 or 8. These days, there's very little reason to ever use anything other than 8 for data bits, especially since using 7 would result in breaking UTF-8 and non-english character sets.

Stop bits - the purpose and action that changing the number of stop bits actually does, has conflicting answers around the internet. Generally this can either be set to 1 (the default), or 2. According to tldp.org, using 2 stop bits can help improve reliability when using very long RS232 cables.

Flow control is rarely used on modern systems. When set to r instead of simply not adding it to the end, CTS/RTS flow control is used, which results in the "client" not sending data to the "modem" / "server" until the modem/server emits a "Clear to Send" signal - while the client may also emit "Ready to Send" to the modem/server to inform the other end that it's ready to receive data. The point of this, is to avoid either end running out of resources to handle the incoming data, something which is rarely an issue with modern technology, thus usually not used with things like Serial-over-LAN.

Breakdown of choices/ranges for each config type

Speed: 1 to 115200 - but usually a number dividable by 8, and/or a multiple of 1200
Parity bits: n (no parity) || e (even parity) || o (odd parity)
Data bits: 8 || 7
Stop bits: 1 || 2
Flow control: DISABLE BY NOT SPECIFYING || r (enable CTS/RTS flow control)

Linux and GRUB 2

To enable the use of Serial console (whether SOL or physical RS232) on Linux, you generally need to adjust the config for your boot loader (GRUB2 being the most common), as to both enable serial for your boot loader, and add a kernel flag to make your Linux install use ttyS0.

On most Linux distros, you shouldn't edit the Grub configuration directly, as it's automatically generated. Instead, you should edit /etc/default/grub to add / adjust the variables which control the use of a serial console for both GRUB (during the boot menu), and within the OS itself (using kernel flags).

Open up /etc/default/grub using an editor of your choice, such as nano or vim - make sure to do this as root:

sudo nano /etc/default/grub

Next, find the lines GRUB_CMDLINE_LINUX, GRUB_TERMINAL and GRUB_SERIAL_COMMAND - make sure they aren't commented out (starting with #). Edit them to match the below lines, or if they're not present, add them to the file.

GRUB Config (/etc/default/grub):

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

NOTE: If you know your serial operates at a different speed (9600 baud), word size (8), parity (n), stop bits, or flow control, then make sure to adjust both ttys0,9600n8 and the flags inside GRUB_SERIAL_COMMAND to match your SOL's serial configuration.

Save the file.

After updating the GRUB config, you'll want to run update-initramfs -u to ensure your system's initramfs is updated with the new serial configuration (on some systems, the serial configuration from /etc/default/grub may be embedded in the initramfs), and then update-grub to re-generate the real GRUB config in /boot with the updated serial config.

update-initramfs -u
update-grub

Raw GRUB 2 Config

If you're on a distro that doesn't use /etc/default/grub or /etc/grub for automatic menu generation, or you're writing a GRUB config by hand (e.g. for a live bootable system), then you'll need to edit grub.cfg by hand.

Below are the GRUB 2 configuration lines required to enable serial console with GRUB:

serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1
terminal_input serial
terminal_output serial

These lines should be placed towards the top of the config file, after any essential set / save commands, but before any menu setup (styles / colours etc.) or menu entries.

SYSLINUX / ISOLINUX / PXELINUX

Inside each SYS/ISO/PXELINUX config you manage, add the line SERIAL [port] [baud] [flow_control] to the top.

port is an integer number starting from 0, similar to the 0 at the end of ttyS0 within Linux, which selects the specific serial port to use for the console, where SERIAL 0 would mean "use the first available serial port as a console".

Note that the flow_control parameter is optional, but may need to be experimented with, to allow for maximum compatibility with different SOL / serial controllers.

In our case, we found 0x008 works best with our Foxconn dedicated servers' Serial-over-LAN systems.

SERIAL 0 9600 0x008
# search path for the c32 support libraries (libcom32, libutil etc.)
path sys/

UI menu.c32

You'll also want to make sure that you use a simple textual menu, such as menu.c32, rather than a graphical menu that may fail to work via serial.

Be aware that some serial controllers / BIOS's don't work properly with SYSLINUX menu's, for example, our Foxconn systems don't support interacting with PXELINUX menu's using the keyboard (but work just fine with GRUB and iPXE menus), however they can still boot using them, by configuring a default menu entry + auto boot timeout.

For more information on configuring the SERIAL line for SYS/ISO/PXELINUX, see this article from tldp.org, which explains the syntax in detail, and contains a table of hex codes for the flow_control argument.