commit b02183e180b2c7de03e98d72cab41b33a9d4a9d9 Author: Brad Anderson Date: Mon Jan 12 20:47:43 2026 -0800 Add first post diff --git a/.markdownlint.json b/.markdownlint.json new file mode 100644 index 0000000..5741242 --- /dev/null +++ b/.markdownlint.json @@ -0,0 +1,6 @@ +{ + "default": true, + "MD007": { "indent": 4 }, + "MD013": { "line_length": 120 } +} + diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..a294b03 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,14 @@ +repos: +- repo: https://github.com/pre-commit/pre-commit-hooks + rev: v3.2.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-added-large-files +- repo: https://github.com/igorshubovych/markdownlint-cli + rev: v0.47.0 + hooks: + - id: markdownlint + args: [ --fix ] + diff --git a/README.md b/README.md new file mode 100644 index 0000000..94dfb7e --- /dev/null +++ b/README.md @@ -0,0 +1,4 @@ +# branderson's blog + +## Stories +- [the time my wifi access point became a wifi dongle for my server rack](server-rack-wifi/index.md) diff --git a/server-rack-wifi/assets/actual-topology.png b/server-rack-wifi/assets/actual-topology.png new file mode 100644 index 0000000..dd55ed5 Binary files /dev/null and b/server-rack-wifi/assets/actual-topology.png differ diff --git a/server-rack-wifi/assets/intended-topology.png b/server-rack-wifi/assets/intended-topology.png new file mode 100644 index 0000000..832d3c6 Binary files /dev/null and b/server-rack-wifi/assets/intended-topology.png differ diff --git a/server-rack-wifi/assets/stp-blocked.png b/server-rack-wifi/assets/stp-blocked.png new file mode 100644 index 0000000..3b340d3 Binary files /dev/null and b/server-rack-wifi/assets/stp-blocked.png differ diff --git a/server-rack-wifi/index.md b/server-rack-wifi/index.md new file mode 100644 index 0000000..e418ceb --- /dev/null +++ b/server-rack-wifi/index.md @@ -0,0 +1,75 @@ +# The time my wifi access point became a wifi dongle for my server rack + +For the past several weeks, I've been dealing with significantly slow +connections to and from my hosted services. Accessing files from my NAS felt +like downloading them over a dial-up connection. Backups were hanging and +failing after running for hours and barely making progress. My photo library +service wasn't even able to finish scanning all of my photos before failing due +to timeouts. Something was wrong. + +This had escalated to the point that it was driving me crazy. I had tried +everything I could think of to try to diagnose the slowness and improve the +situation. I migrated from CIFS to NFS for network shares, having seen that CIFS +can be slow when handling large numbers of small files. I migrated services +which had been running in LXC to dedicated VMs. Assuming network congestion may +be an issue, I reduced the amount of internal traffic my services generate by +scaling back the rate of backups and file scans. I replaced cables in my server +rack. Almost nothing helped (I say almost because the migration from CIFS -> NFS +actually did significantly reduce IO pressure in my main server :frog-yay:). + +At this point, I decided to start more deeply investigating the state of the +network itself, having ruled out network congestion and service configuration as +likely culprits. I installed a tool called iperf3 which measures maximum network +bandwidth between hosts onto several hosts in my network. This yielded some +interesting results: +- connections between servers are fast (gigabit ethernet) +- connections between server VM hosts on shared linux bridges (e.g. vmbr1) on + the same server are SUPER FAST (10.9 Gbits/sec) +- connections between server VM hosts on different linux bridges (e.g. vmbr1 -> + vmbr3) on the same server are fast (gigabit ethernet) +- connections to server VM hosts from outside server rack are slow (10-20 + Mbits/sec) +- wifi has low bandwidth (1-4 Mbits/sec) + +It was clear at this point that the slowdown seemed to be introduced when +requests needed to enter or leave the server rack, as these connections were +capped at 10-20 Mbits/sec while connections within the server rack were capped +at the max rate for gigabit ethernet of around 943 Mbits/sec. I decided to take +a closer look at the network configuration itself. + +The intended ethernet route from my router to my server rack looks something +like this: +- [Intended Topology](./assets/intended-topology.png) +- Ethernet is run from my router, to a network closet switch, then through my + walls to my living room switch, then into my garage to the server rack switch, + where my servers and NAS are connected. + +Looking at the port configuration for my living room switch showed something +unexpected: +- [STP blocked](./assets/stp-blocked.png) +- The switch port handling the ethernet connection between my network closet + switch and living room switch had been disabled by the network due to a + spanning tree protocol (STP) issue! + +It turns out that around the time the slowness began, the living room switch had +performed a factory reset and set its STP priority to a very large number (and +thus low priority). This caused the network to configure the uplink for the +living room switch to not be the ethernet connection to my network closet switch, +but instead use the wireless access point Mr. Stabby which was meant to provide +wifi to my living room, disabling the intended ethernet uplink as an STP loop! + +In turn, Mr. Stabby was no longer receiving ethernet from the living room switch, +but instead meshing with another access point named Frisbee, located across my +house. + +This resulted in my network route from my router to my server rack being +configured something like this: +- [Actual Topology](./assets/actual-topology.png) +- Instead of functioning as a wifi access point, Mr. Stabby had instead become a + giant wifi dongle for my entire server rack! I would not have assumed this was + possible. + +Fixing the STP priority for my living room switch to the correct (lower) value +allowed it to take priority over Mr. Stabby for STP, reconfiguring the network +back to the intended layout, and finally fixed the network slowness issues and +my sanity. :phew: