2Gbps Link Aggregation on Turing Pi BMC with 802.3ad LACP

The Turing Pi 2.5 has two gigabit Ethernet ports (ge0 and ge1) on its BMC. By bonding these together with 802.3ad LACP, you can achieve true 2Gbps aggregate bandwidth for your cluster traffic. This post documents how I implemented this feature in custom firmware.

Why 2Gbps?

In a homelab cluster, the BMC handles all network traffic for up to 4 compute nodes. With multiple nodes running workloads that generate significant network I/O, a single gigabit link can become a bottleneck. Link aggregation doubles the available bandwidth and provides redundancy.

Prerequisites

• Turing Pi 2.5 board
• Custom firmware with bonding support (or build your own)
• Switch with LACP support (I used a UniFi USW Pro Max 24 PoE)
• Two Ethernet cables connected to both BMC ports

The Challenge

Getting 802.3ad LACP working on the Turing Pi BMC required solving several issues:

1. DSA Ports Share the Same MAC Address

The ge0 and ge1 interfaces are DSA (Distributed Switch Architecture) ports on a Realtek switch chip. By default, they share the same MAC address, which breaks bonding since Linux requires unique MACs for slave interfaces.

2. Hardcoded Bonding Mode

The stock firmware’s S00dsa init script loads the bonding module with a hardcoded mode:

modprobe bonding mode=balance-alb miimon=100

This prevents changing to 802.3ad mode at runtime.

3. Mode Changes Don’t Persist

Even if you manually configure 802.3ad, the changes are lost on reboot because the module is loaded with balance-alb mode before the network configuration runs.

The Solution

I created a proper bonding setup with these components:

Architecture

Web Browser → BMC-UI (http://turingpi) → REST API → bmcd daemon
    → network_config.rs → /etc/bonding.conf → S45bonding → bond0

Key Changes

1. Fixed S00dsa: Removed the hardcoded mode from modprobe:

   modprobe bonding  # No mode specified
   

2. Created S45bonding: A new init script that runs after network init and properly configures bonding:

   • Reads mode from /etc/bonding.conf
   • Sets unique MAC on ge1 before enslaving
   • Deletes and recreates bond0 with the correct mode
   • Adds bond0 to the br0 bridge

3. REST API Integration: Added network_config.rs module to bmcd for web UI control

Configuration Files

/etc/bonding.conf

Contains the bonding mode (one of the supported modes):

active-backup

The default is active-backup for safety - it works without any switch configuration. Change to 802.3ad for LACP once your switch is configured.

/etc/bonding.enabled

Empty marker file that enables bonding when present.

S45bonding Init Script

#!/bin/sh

BONDING_CONF="/etc/bonding.conf"
BONDING_ENABLED="/etc/bonding.enabled"

get_bonding_mode() {
    if [ -f "$BONDING_CONF" ]; then
        cat "$BONDING_CONF" 2>/dev/null | head -1 | tr -d '[:space:]'
    else
        echo "active-backup"
    fi
}

start_bonding() {
    if [ ! -f "$BONDING_ENABLED" ]; then
        return 0
    fi

    MODE=$(get_bonding_mode)

    # Remove interfaces from bridge
    ip link set ge0 nomaster 2>/dev/null
    ip link set ge1 nomaster 2>/dev/null
    ip link set ge0 down
    ip link set ge1 down

    # Set unique MAC on ge1 (DSA ports share same MAC by default)
    BASE_MAC=$(cat /sys/class/net/ge0/address 2>/dev/null)
    GE1_MAC=$(echo "$BASE_MAC" | awk -F: '{
        last = ("0x" $6) + 1;
        printf "%s:%s:%s:%s:%s:%02x", $1, $2, $3, $4, $5, last
    }')
    ip link set ge1 address "$GE1_MAC"

    # Delete existing bond and recreate with correct mode
    ip link del bond0 2>/dev/null
    ip link add bond0 type bond mode "$MODE" miimon 100

    # For 802.3ad, set LACP rate to fast
    [ "$MODE" = "802.3ad" ] && echo fast > /sys/class/net/bond0/bonding/lacp_rate

    # Enslave interfaces
    ip link set ge0 master bond0
    ip link set ge1 master bond0
    ip link set ge0 up
    ip link set ge1 up
    ip link set bond0 up
    ip link set bond0 master br0
}

Supported Bonding Modes

Switch Configuration

For 802.3ad LACP, your switch must be configured to expect LACP on those ports. On UniFi:

1. Go to Devices → Your Switch → Ports
2. Select the two ports connected to the Turing Pi
3. Create an Aggregate with LACP mode enabled

Verification

Check the bond status:

cat /proc/net/bonding/bond0

Expected output:

Ethernet Channel Bonding Driver: v6.8.12

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100

802.3ad info
LACP active: on
LACP rate: fast
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 2
    Partner Mac Address: 9c:05:d6:63:f9:bb

Slave Interface: ge0
MII Status: up
Speed: 1000 Mbps
Duplex: full

Slave Interface: ge1
MII Status: up
Speed: 1000 Mbps
Duplex: full

Key things to verify:

• Bonding Mode: IEEE 802.3ad Dynamic link aggregation
• Number of ports: 2
• Partner Mac Address shows your switch’s MAC (not 00:00:00:00:00:00)
• Both ge0 and ge1 show MII Status: up and Speed: 1000 Mbps

Web UI / API

The bonding configuration is also accessible via:

• Web UI: http://turingpi → System > Network Settings
• API: GET /api/network_config returns current status
• API: POST /api/network_config?enabled=1&mode=802.3ad&apply=1 to configure

Troubleshooting

LACP Not Negotiating (Partner MAC is 00:00:00:00:00:00)

• Verify LACP is enabled on the switch ports
• Check that both cables are connected
• Wait 30+ seconds for negotiation

Network Unreachable After Mode Change

If you change to 802.3ad but the switch isn’t configured for LACP:

1. Temporarily disable LACP on the switch
2. Access BMC and change mode to active-backup
3. Then configure switch for LACP and switch back to 802.3ad

Bonding Not Starting

• Verify /etc/bonding.enabled exists
• Check /etc/bonding.conf contains a valid mode
• Run /etc/init.d/S45bonding status to check bond state

Flashing to Internal Storage

To flash the custom firmware to internal storage without using an SD card:

1. Boot from SD card with working firmware
2. Attach UBI device: ubiattach -m 1 -d 0
3. Copy firmware: scp firmware.tpu root@turingpi:/tmp/
4. Write to UBI: ubiupdatevol /dev/ubi0_1 /tmp/firmware.tpu
5. Reboot without SD card

Links

• Pull Request
• Linux Bonding Documentation
• 802.3ad LACP Standard