Diskless network boot NFS root

From ArchWiki

Jump to: navigation, search

Contents

Root over NFS (diskless boot)

This HOWTO will explain the step-by-step method for booting an Arch Linux installation over an NFS share. The first half will primarily cover configuration of the server, and the second will cover client side configuration.


For more information on Network File Sharing, see Nfs.

Hardware Considerations

Server: Any standard computer should work, but for performance reasons it should have fast disk access and preferably gigabit ethernet.

Client: For a completely diskless boot, the client computer must have a network-bootable (PXE) network card. Wireless will not work, with exception of a wireless-to-ethernet external bridge. Many motherboards have a PXE-compatible ethernet card built in, but you will need to enable support in the BIOS.

If you don't have a PXE-compatible network card, you can setup a semi-diskless client. This involves building a flashdrive or CD-based boot partition with syslinux or grub. The configuration is the same, but skipping the tftp steps. Creating boot-CDs or flashdrives is beyond the scope of this article.

Server-Side Setup

The server will have to run a tftp daemon to share the kernel/bootsector along with an NFS share to share the root directory. PXE requires the DHCP server to announce the location of the tftp server to any clients that connect. Most routers will not support this, and this article will show how to modify an existing DNSMasq or DHCPD server. See NAT'ing_firewall_-_Share_your_broadband_connection for setting up a simple router replacement, including DHCP via DNSMasq, a good place to start for this project. If you do have a router that supports PXE, then you will need to tell it to send all tftp requests to the server.

At this point I will assume you have a server running with DNSMasq or DHCPd providing DHCP services, along with some space for the client's root.

First, install nfs-utils, syslinux (for the pxelinux bootloader, you can substitute for GRUB, see [1]).

pacman -S nfs-utils syslinux

Create Client Root Directory

First, create a directory that will hold the root filesystem, this can be anywhere. For this example, I will use /disklessroot.

mkdir /disklessroot

Create a directory for new pacman database,proc and dev mountpoints.

mkdir -p /disklessroot/var/lib/pacman
mkdir /disklessroot/proc
mkdir /disklessroot/dev
mkdir /disklessroot/sys

Mount the /proc and /sys filesystems to allow the installation to use the kernel-provided information within the chrooted environment, and then mount-bind the /dev filesystem.

mount -t proc none /disklessroot/proc
mount -t sysfs none /disklessroot/sys
mount -o bind /dev /disklessroot/dev

Now you need to populate that root with an installation of Arch Linux.

If you are running 64-bit server and your client is 32-bit, you want to change /etc/pacman.d/mirrorlist entry to point 32-bit architecture. Remember to switch it back after populating client's filesystem.

pacman -Sy --root /disklessroot --dbpath /disklessroot/var/lib/pacman base

Edit /disklessroot/etc/mkinitcpio.conf, add 'nfs' to the MODULE section, and add 'net' as the second entry under HOOKS..

MODULES="nfs"
HOOKS="base net udev autodetect pata scsi sata filesystems"

Now, chroot into /disklessroot and recreate the kernel image.

chroot /disklessroot
mkinitcpio -p kernel26
exit
umount /disklessroot/dev
umount /disklessroot/proc
umount /disklessroot/sys

You should be back to the normal root, the exit command should leave the chroot. Check before continuing.

PXE/TFTP Setup

Using DNSMasq

Now to setup tftp and PXE.

Within DNSMasq, there is an easy-to-use tftp server, so this article will use that. First, edit /etc/dnsmasq.conf and uncomment these lines, changing them to reflect your file paths.

dhcp-range=192.168.0.1,192.168.0.100,12h     #change ip addresses to your network
# Set the boot filename for BOOTP. You will only need 
# this is you want to boot machines over the network and you will need
# a TFTP server; either dnsmasq's built in TFTP server or an
# external one. (See below for how to enable the TFTP server.)
dhcp-boot=pxelinux.0  #change this if you are using GRUB.

# Enable dnsmasq's built-in TFTP server
enable-tftp

# Set the root directory for files available via FTP.
tftp-root=/disklessroot/boot/

#Send options to PXELinux. Note that we need to send the options even
#though they don't appear in the parameter request list, so we need
#to use dhcp-option-force here.
#See http://syslinux.zytor.com/pxe.php#special for details.
#Magic number - needed before anything else is recognised
dhcp-option-force=208,f1:00:74:7e

Configuration file name
dhcp-option-force=209,pxelinux.cfg/default

Path prefix
dhcp-option-force=210,/disklessroot/boot/

This has the additional feature of allowing you to update the kernel within the client OS.

Make sure the firewall on your server does not block UDP port 69 (UDP ports 67-68 and 4011 are also used, but should not need to be opened).

Using DHCP & XINETD

If you already have XINETD & DHCP running then you can use them to support net booting in place of the DNSMasq method above. The xinetd service will start the TFTP program on demand as needed.

First, edit /etc/dhcpd.conf to add the following statements (assuming you already have dhcpd working.)

#added for PXE net booting
allow booting;
allow bootp;
group {
	next-server 10.0.0.1; #change IP address to match your server
	# This is the pxe bootloader file
	filename "pxelinux.0";
	# One host block per client. This network only has one.
	host netboot1 {
		#change below to match MAC address of client network device
		hardware ethernet 08:00:27:d0:9b:97;
		#change below to IP address desired for client  
		fixed-address 10.0.0.10; 
		}
	}
#group

Second, create a file /etc/xinetd.d/tftp with the following lines. After the the xinetd service is restarted the tftpd program will start on demand.

service tftp
{
	per_source = 11
	socket_type = dgram
	protocol = udp
	user = root
	server = /usr/sbin/in.tftpd
	server_args = -s /disklessroot/boot
	wait = yes
	cps = 100 2
}

Be sure to restart the DHCP & XINETD services to activate the changes.

NFS Setup

Edit /etc/exports, add this line.

/disklessroot *(rw,fsid=0,no_root_squash,no_subtree_check)

If security is an issue, you can replace * with an IP address or IP range to restrict connections. This is semi-redundant with the /etc/hosts.allow configuration below.

Edit /etc/hosts.allow

nfsd: 10.0.0.
rpcbind: 10.0.0.
mountd: 10.0.0.

This partial IP will match everything in the 10.0.0.x range, you can specify a single address, IP range, or simply "ALL" (Insecure!).

Add "rpcbind nfs-common nfs-server" to the server's /etc/rc.conf under DAEMONS, in that specific order. (Assuming dnsmasq was already installed and on the daemons list.)

Refer to Nfs for any configuration/security issues that are not covered here.

Client Configuration

These files are to be edited from the server, as the client is not ready yet.

Boot Configuration

Copy pxelinux.0 to /disklessroot/boot/

cp /usr/lib/syslinux/pxelinux.0 /disklessroot/boot/
mkdir /disklessroot/boot/pxelinux.cfg

Create and edit /disklessroot/boot/pxelinux.cfg/default (Replacing IP addresses and file-paths as needed.)

default linux

label linux
kernel vmlinuz26
append initrd=kernel26.img rootfstype=nfs root=/dev/nfs nfsroot=10.0.0.1:/disklessroot,v3,rsize=16384,wsize=16384 ip=dhcp

The syntax for the pxelinux bootloader is the same as Syslinux, here[2] under "CONFIGURATION FILE". Remember that any filenames will be relative to the tftp server, check the setting in /etc/dnsmasq.conf named "tftp-root=". This example places tftp-root at /disklessroot/boot/. Thus the kernel should be at /disklessroot/boot/vmlinuz26 on the server, and will show up as /vmlinuz26 for the first stage of client boot. Exactly like a separate /boot partition.

As there is no entry for / in the clients /etc/fstab file, the client may display the error "mount: can't find / in /etc/fstab or /etc/mtab" on startup. This error can be safely ignored. However, if you want to suppress the error you can add a hack to /etc/fstab. This involves adding the following to /disklessroot/etc/fstab:

none / none

Note:If you want ssh into netbooteed machine,add this line to /disklessroot/etc/fstab:

none            /dev/pts        devpts        gid=5,mode=620    0 0

Without this,you will get an error that states: "Server refused to allocate pty". --edacval 16:54, 26 October 2008 (EDT)

DHCP/Network Daemon Workaround

This is to prevent the client from trying to reconnect the network and killing itself, also prevents the client from prematurely killing itself while shutting down. Any disconnect of the network and your client will freeze.

The following workaround involves running the dhcpcd DHCP client on startup with the -s option to use the existing kernel DHCP auto-configured IP address instead of requesting a new one. The -p option is used to prevent dhcpcd from deconfiguring eth0 when it exits (which causes problems when shutting down). Notes: This workaround assumes that eth0 is the network interface used for DHCP.

Create a new file /etc/rc.d/network-nfsroot (/disklessroot/etc/rc.d/network-nfsroot on the server) with the following contents:

#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions

case "$1" in
        start)
                if ! ck_daemon network-nfsroot; then
                        echo "Network is already running.  Try 'network restart'"
                        exit
                fi

                stat_busy "Starting Network"

                if dhcpcd -p -s $(ifconfig eth0 | grep -o '[0-9]*\.[0-9\.]*' | head -n1) eth0; then
                        add_daemon network-nfsroot
                        stat_done
                else
                        stat_fail
                fi
                ;;
        stop)
                stat_busy "Stopping Network"

                if [ -f /var/run/dhcpcd-eth0.pid ]; then
                        /bin/kill $(cat /var/run/dhcpcd-eth0.pid)
                        rm_daemon network-nfsroot
                        stat_done
                else
                        stat_fail
                fi
                ;;
        restart)
                $0 stop
                /bin/sleep 2
                $0 start
                ;;
        *)
                echo "usage: $0 {start|stop|restart}"  
esac

# vim: set ts=2 noet:

Edit the client's rc.conf and change to use network-nfsroot daemon instead of network daemon (/disklessroot/etc/rc.conf, or just /etc/rc.conf once you boot into the client).

Before:

DAEMONS=(syslog-ng network ...

After:

DAEMONS=(syslog-ng network-nfsroot ...

Another workaround is presented below which configures the DNS server manually.

DHCP/Network Daemon Workaround Number 2

Edit the client's rc.conf and disable the network daemon (/disklessroot/etc/rc.conf, or just /etc/rc.conf once you boot into the client).

DAEMONS=(syslog-ng !network ...

The main issue with this method is the fact that your resolv.conf will not auto-update (since this bypasses DHCP). To work around this, edit /etc/resolv.conf and add this line (/disklessroot/etc/resolv.conf on the server).

nameserver 10.0.0.1  #change this IP for whatever DHCP/DNS server you have, the server's IP usually works if you configured DNS correctly.


Testing/Debugging the Client

Now reboot the client computer, making sure the network card is first in the boot-order.

If all goes right, you should see the network card get an IP address from the server, then connect and boot the kernel. After the initial kernel messages, you should either see Arch Linux boot, or a "killed init" message.

If init was killed, check your configuration, it usually means it could not mount the NFS folder. Refer to Nfs for more help.

If it freezes during a mounting operation, check the client's /etc/fstab or the /boot/pxelinux.cfg/default settings.

If it freezes duing the "Network" section, either disable it using the previously mentioned workaround, or specify IP settings in /etc/rc.conf. (You can try setting eth0="dhcp")

Performance Notes

I am currently using a diskless client as my primary internet/development OS due to a hard-drive restriction on my PC. My server is an older machine is running Arch Linux with a software-based raid5 array. My client can read files from the NFS share almost as fast as my server can, but there is a limitation with latency and caching. It slows down during "pacman -Sy" and package installs that involve many small files. Otherwise, it runs fine. It also seems that NFS runs synchronously by default, meaning that all writes have to be committed before continuing, no caching in free memory like it would on a local hard drive. This is actually a good thing, from the data integrity standpoint, as fewer files are lost during a random power outage. I have not tried the "async" NFS option because the nfs man page does not suggest it will work for NFSv3.





Mostly Complete. Thank you, Net147, for your excellent edits. Please edit/update as needed.

Personal tools