What's the advantage of doing this over plugging multiple gigabit adapters into a linux machine and adding them all to a bridge?
I'm guessing performance might be better with the hardware, but I don't know --has anyone done tests to show the difference?
What ever happened to Openflow? Wouldn't this be a perfect device for it? Setting up a lab for it was almost impossible thats why I lost interest.
OpenFlow was kind of the wrong solution to the problem. (The right solution is switchdev or SAI.)
OpenFlow is a protocol, switchdev is a driver API for exposing access to essentially the same operation model as OpenFlow targets. You don't replace OpenFlow with switchdev, you might use OpenFlow to integrate a switchdev with an SDN
You want to write code that runs on the switch, whether it's a routing protocol or an intent reconciliation loop or whatever. You don't want to send low-level OpenFlow commands over the network because it ends up being slow and chatty.
A considerable portion of SDN craze involved pretty much this. Even systems that didn't use OpenFlow were incredibly chatty, creating complex chains of NextHop routes from centralized or at least semi-centralized servers to the actual switches.
Even in more sane setups, it's not actually uncommon to have control plane talk over network to actual forwarding plane - IIRC Cisco Nexus operates this way, with actual forwarding engines talking with control plane over Ethernet link.
I like the code to run on the switches directly, but SDNs very often eschewed that for dumber forwarding engines controlled OpenFlow way, even if actual protocols used weren't OpenFlow.
SAI being what?
Switch Abstraction Interface
I'll look them up. Cheers.
AFAIK OpenFlow tried to be far too generic, resulting in it not actually mapping onto any actual hardware without lots of abstraction leakage.
Ironically it could also be said OpenFlow was not generic enough. More and more networking hardware continues to move towards having a fully programmable pipeline but OpenFlow isn't the most flexible way to define a completely arbitrary pipeline.
https://opennetworking.org/news-and-events/blog/clarifying-t...
It mapped very well to HW, but only to dumbest part of the hw.
at least from pov of someone dabbling in SDNs around 2014-2017 including deep guts implementation details of one commercial one...
OpenFlow encoded the approach of dumb ethernet switches, combined with Cisco Express Forwarding - which is a common aspect of many switching ASICs, but not entirety of them, even for Cisco.
It never abstracted that historic aspect, if anything the leaking was the internal model of OpenFlow (editing forwarding table) rather than internals HW (which sometimes was a systolic array of CPUs that exposed "tables" as detail of interaction but nothing actually fundamental).
All was fine so long, like CEF, you dumped everything to CPU/northbound API outside "IP with this address should go there, and TCP flow with those parameters go there", in case of early openflow bejbf essentially "packet with dstMAC shall go to port X".
This does not work well with acceleration engines, or in beefier chips the aforementioned systolic arrays or similar arrangements, outside of hardcoding idea of specific accelerators and their interface tables.
"Network switches are simple devices, packets go in, packets go out. Luckily people have figured out how to make it complicated instead and invented managed switches."
Expensive switches involve some pretty fancy ASICs. For example I have a pair of fairly elderly Dell OS9 switches with 48 x 10Gb/s ports and four x 40Gb/s QSFP+ fibre ports. These are "old skool" stacked jobbies. Each switch can shuffle up to 1.28Tb/s (1). That's quite a lot. You can get those for £1800 including VAT (2) these days and they will last nearly forever.
I love to see efforts like this but do bear in mind that say Netgear will do a eight port 1Gb switch with Power over Ethernet on all ports for about £125.
If you cost your time at somewhere between £20-50 per hour when evaluating whether a project is financially viable, then an off the shelf box might be indicated. However, if the actual purpose is the project itself then sod the price!
(1) https://i.dell.com/sites/doccontent/shared-content/data-shee... (2) https://www.etb-tech.com/dell-force10-s4820t-10gbe-switch-os...
I worked with one of these ASICs, from Broadcom. Not 40, but something like 4x10+24x1Gbps + PCIe to CPU. The ASIC cost as much as you would expect (I don't know the actual number - a couple hundred bucks?). The software interface to it was very poorly documented, and was a library that supported all Broadcom switch ASICs, so was a few hundred megabytes .a file and was full of functions that would just return "not supported on this device" errors, which you wouldn't know until you tried it.
Not Broadcom, but Microchip has list prices straight on their website (how refreshing!), and yeah few hundred bucks sounds about right; for example this random 128Gbps switch chip is about $120 in single quantities, $80 in volume
https://www.microchip.com/en-us/product/vsc7552#purchase-fro...
It feels bit bonkers that a actual switch based on that probably costs thousands.
The software side is crap as usual. They sell their Linux-based package at $75000, with maintenance (basically updates) at $17500.
So even though the chip is cheap, you need to burn six figures to do anything with them :(
Their Sparx-5 line is supported by stock upstream Linux switchdev.
Intrigued by that, I went looking. First commit of support was 2019, for Linux5.2. Yet, 5 years later, there is not one example (that i could find) of OpenWrt running on these Chips. If the platform/FW/BSP package were any good, I'd expect there to be chat about it in the forums, and/or attempts to port to such devices from SMBStaX (The Microchip BSP Linux dist).
I suspect it's not worth the effort for most people. People who want to DIY a switch are likely content to brute force it with a decent cpu, multiport network cards, and lots of PCIe lanes. While others will be happy with a used enterprise OEM unit.
Does make me wonder what asics Microtik is using under the hood though...
A lot of them are just networking SoCs with built in switching fabrics or the lower end ones have the low end router SoC from various manufacturers.
Mikrotik is one of the few brands that actually list this in the spec sheets.
You are right, the situation is better that what I thought, the commercial smbstax SDK might not be absolutely required.
They even have some BSP stuff freely available which is really positive sign. https://github.com/microchip-ung/bsp-doc
This would be such an enticing project to jump into, there are just enough bits to make it seem doable, even if the realist in me knows its not really feasible.
That's a very cheap price for enterprise layer 3 switching chip. Thanks for the info, how this switch chip has escaped me is very perplexing.
Can you elaborate on their Linux-based package, is the package critical for the development and using the chip with embedded Linux. Can we just use it as switching fabric and use Intel CPU for example N100 as the main controller running Linux?
The sibling comment also mentioned that Sparx-5 line is supported by stock upstream Linux switchdev, and this the same chip as in your link.
Ignorance is bliss.
Interesting project.
For simpler use you can get an OpenWRT capable router which in most cases uses a managed switch chip. OpenWRT provides a nice interface to configure VLANs and other options.
OpenWRT these days can also be installed on some switches eg the Zyxel GS1900 series (though support for things like PoE and 10Gb/SFP+ ports might be limited, I’m not sure where things stand there).
WTF, i had no idea that was possible. I have that exact switch (GS1900-24E)... I need to look into that
Install mosquitto afterwards, add a small script, and the ports all show up in HomeAssistant as power-monitoring ON/OFF switches. :)
Jesus christ that’s awesome!
Looking at the OpenWrt forum, work is coming along nicely w.r.t supporting FasterThan1G on RTL9X. Many/Most configurations seem to be working.
RTL8X is AFAIC done, feature-complete. https://svanheule.net/switches/
POE on those devices is mostly two types: Broadcom (well supported), and Realteks inhouse solution, which uses a 'dialect' of the Broadcom protocol. There is a git branch/PR, where the 'dialect' differences have been moved to individual modules. But it's not released yet. https://github.com/Hurricos/realtek-poe/pull/35
Unfortunately, there seem to be hard problems migrating up from Linux 5.15 to 6.1 or 6.6.
I just bought one of the zyxel 24-port poe switches for use with openwrt.
I've been using openwrt on mikrotik switches, but max was 10-ports. the extra ports and poe will be nice.
I think the zyxel gs1900 48-port switches are also supported.
Does it support rSTP?
I don't think this support it, only regular STP and apparently MSTP
and apparently MSTP
Where did you find that?
STP is "slow" STP, and RSTP is a degenerate case of MSTP!
MSTP is Multiple spanning trees ie you can group VLANs and prefer paths for those groups of VLANs. That means if you have say two links between two bridges (switches) you can prefer some to use one link and the rest to use the other, that means you are not "wasting" a standby link. They will fail over to the surviving link on failure. STP and RSTP will only consider one link as a whole, so two ports are "wasted" when not in use: in the case of a two bridge, two links example.
Old school STP without the Rapid part hasn't really been a thing for several decades. I can't think of why you wouldn't use RSTP in general but if you need to make best use of your forwarding capacity then a 50/50 MSTP may be indicated. That's where you look at your traffic flows across VLANs and try to bundle them up into a 50/50% collection. One lot prefers link A and the rest get link B. Obviously you can get really creative as the number of VLANs and links mount up. Bear in mind that dot1Q is a simple version of QinQ!
Sorry, got a bit carried away there.
For nearly all intents and purposes, RSTP is STP. If you plug in a network cable between two devices and it does not start working within say five seconds then you are living in the 1990s.
I meant, where did you find that this specific Linux-managed network switch supports MSTP? Because the Linux bridge networking code, last time I looked, only supported STP; you had to run a separate daemon to even get RSTP.
If you plug in a network cable between two devices and it does not start working within say five seconds then you are living in the 1990s.
a living in a hub...
:)
No offense intended, I may have misunderstood something here.
You use a routerboard block diagram as your model to demonstrate how a hardware switch is connected to the rest of the system. But then you go on to claim that they are impossible/difficult to work with.
Did you source a routerboard at any point here? Fairly sure that OpenWRT can be built for most routerboards, and the 2011 should be a fairly common device on the second hand market.
Maybe this is a better question. Was the goal always to build from scratch? Or did you discard the concept of using someone elses hardware for a particular reason?
I have a stack of them here, they are great to work with for their intended purpose but they are not that great if you want to run custom software.
I also grabbed a RB2011 diagram because it's a simple diagram that explains it well I think, the RB1100AHx4 is a better example technically since it's using the same switch chip but it's more confusing because they use the two CPU ports together and claim it's a 2.5Gbps link while it's two 1.25Gbps link and they ignore the encoding overhead.
The reason why I build this from scratch is that it that the expense is reasonable and this should end up in the FOSDEM video recording boxes and it has to fix a few issues specific to that design like it needing to expose 4 network ports to the front panel of the case but also be connected to the internal SBC. There's simply not that much space in the case to put passthroughs in the case to a switch to not have an external loop cable for the SBC and with a dumb switch the system can't be monitored anymore. Since quite a few of these boxes are built this becomes a reasonable solution (if you ignore design time, but this is volunteer work)
Awesome that makes sense.
The RouterBoard RB2011 is stuck on an ancient release version of OpenWrt, 19.07, see here: https://openwrt.org/toh/mikrotik/rb2011
The problem seems to be somewhat related to the NAND, which is IIRC somehow different from the supported RouterBoards. Somebody proposed a new solution to this problem, but that hasn't landed yet: https://forum.openwrt.org/t/wiki-cleanup-for-mikrotik-rb2011... and other threads.
Nice article, Thank you for the write-up.
First time I saw somebody 'creatively using' an RTL83something switch was https://spritesmods.com/?art=rtl8366sb, and there were others since then, but yours was the first 'build my own managed switch', instead of 'adding an external brain to an unmanaged switch'
Ah yes the legendary sprite_tm, I've come across this while writing my own raspberry pi firmware to manage the switch over USB.
From what I've read, it's very hard to make a 10 Gbps switch without relying on some blobs since those accelerator chip makers don't have any open drivers support.
And you supposedly need those chips since switching it on the CPU is very taxing.
The first paragraph reads as if it's taken directly from The Hitchhikers Guide To The Galaxy.
For such a low speed and small number of ports using a hardware switch circuit is not necessary.
Nevertheless, if the designing and building effort is neglected, I assume that the total cost of the hardware might be under $100, which is less than a computer with multiple interfaces would cost.
Still, 1 Gb/s networks are rather obsolete. One could make a managed network switch that is bigger and faster by using only off-the-shelf components for slightly more than $200, e.g. a 6-port 2.5 Gb/s switch can be made with an Odroid H4+ having 2 ports, together with its add-on M.2 card with 4 extra Ethernet ports. Another variant is to use a small computer with an N100 CPU and 4 2.5 Gb/s ports, which can be bought in this price range from various Chinese companies. Similar small computers with 6 2.5 Gb/s ports are a little more expensive, perhaps slightly over $300.
Unfortunately gigabit ethernet is far from obsolete.
Yes, there's 2.5 gigabit on some consumer hardware, but it's still kind of rare.
Also who is excited about a 2.5x speedup after 20 years? Nobody cares until we need 10 gigabit internet access (which will probably never happen).
For what it's worth, my ISP gives me 5Gbps down/700Mbps up for ~40 euros a month which includes a bunch of TV channels and discounts for Netflix and Disney+.
They also have an 8Gbps down / 8Gbps up plan for ~60 euros, also including a ton of extra things.
Wow, that's pretty cool. Where is this offered?
But have you invested in 5Gbps+ networking gear to actually take advantage of the offer? 10 Gbps Nics have become super affordable (~20$) by the way.
Cat7 cabling is something I encountered at discount stores now
I wouldn't trust a Cat7 cable from a discount store to adhere to the spec though.
I wouldn't use copper for 10G anyway, use a fibre sfp, far more power efficient.
Doesn't that depend on the run length? Surely copper is more efficient for just a few meters.
My 10g copper SFPs are far hotter than my 10g fibre ones
It depends.
In my case, all rooms are cabled with a weird electrical standard that should get me 10G Ethernet (and does 2.5G without any issue). I'm not going to drill holes to pull my own fibre all around the place when I have perfectly good Ethernet connectivity.
Also, while 10G SFP+ NICs are vastly more available than Ethernet 10G ones, switches seem to be cheaper with 2.5/5/10G Ethernet ports than full on SFP+, unless you buy recycled Enterprise gear which would blow your power efficiency argument out of the water.
Used it more as illustration of market penetration - just like with HDMI cables, you're either ripped off on markup, sold under-spec cabling, or both.
Free in France.
The 8/8 plan comes with a router that has WiFi 7 (theoretical max 46Gbps, but who knows in reality, it's not really out yet), an SFP+ port, and 4x2.5Gbps Ethernet ports. The one I have only has one 2.5Gbps Ethernet, so that's the Internet speed at which my home network caps out at.
If you're interested in more details, I recently finished writing an article about my home network: https://atodorov.me/2024/07/03/running-a-multi-gig-home-netw...
And yeah, 10G equipment (in terms of NICs and cables) is quite affordable, but switches still aren't really super affordable.
The main reason for 2.5g is for digital video (2.5gbit will allow do 2 1080i 2110 streams), and especially the increasing numbers of wifi6e APs that do >1gbit but nowhere near 10g
Are they? Most consumer, even office gear still is maximum 1000M/port - your average USB-C network/multi-port laptop dongle, most USB-C/TB monitors (and shamefully, Apple's Studio Display which only has USB-C/TB ports, no network), VoIP phones (hell these are usually 10/100 only, with 1000 being reserved for top models), printers, virtually all entry to mid range NAS systems... the list is endless.
Options for more than that tend to be really niche, expensive, or are bring-your-own-module-SFP.
Besides, 1000M is way more than enough for almost all consumer and office needs. Only exception is heavy video and photo editing, if these workplaces don't already use direct Thunderbolt/FC attach.
No they aren't.
You failed to mention the big one: most people by very far have 1 Gb/s network switches to... Connect their machines to this thing called the Internet. Some may have heard of it.
And most people also have not more than 1 Gbit/s up/down.
Do I have switches (with a 's') with 10 Gbit/s SFP+ at home? Yup. Is 1 Gbit/s obsolete: definitely not.
1gig is starting to become a problem - we have increasing deployment of 2gig and 8gig internet to homes in Poland for example, figuring out how to deal with that when most computer gear still comes with 1gig is becoming an issue
They definitely aren't, no idea what parent is talking about. They are the standard unless you're talking about a SAN or something. If I had to guess, 92% of the switches in a typical office are 1000M, 5% are 100M, and the rest are > 1000M for switch interconnects or HA server interconnects/SANs.
A raspberry pi 400 can handle around 875 MBit when bridging two interfaces. So it’s not even fast enough for two ports let alone full duplex. I doubt a n100 can handle more than three.
An Intel Celeron N5105, on one core, does ~28Gbps locally between two bridged interfaces with iperf.
A Raspberry Pi 4 has a single PCIe lane with a total of 4Gbps bandwith for everything (all USB ports and Ethernet).
Edit: shameless plug, but I just finished writing an article on my new home network, including the router with the above spec/results: https://atodorov.me/2024/07/03/running-a-multi-gig-home-netw...
The CPU is the bottleneck. All switching is done in software with off the shelf NICs.
Yes, my Intel Celeron N5105 had a full core pegged at 100% while doing that iperf and getting those numbers.
But there are 4 whole cores, meaning I can get far more traffic switched and routed with the CPU capacity than all the ports combined can sustain.
$100? I paid roughly $80 for two of these and the price per boards will rapidly converge to ~$17 when increasing quantity.
The PINE A64-LTS alone is $40 + shipping + taxes, so you cannot have a $17 total cost.
I assume that you mean that the card with the switch circuit alone could reach $17, if made in a large quantity.
However, you cannot make a managed switch with that card alone.
If your current cost has been $40 for a card with the switch circuit, adding the PINE A64-LTS + whatever you have paid for shipping and taxes must make the total around $100, exactly like I have said.
I meant the cost of just the switch obviously... It can hook up to whatever Linux machine you have running and I happened to have this one in the drawer.
The point of this was not to make the fastest or cheapest or most featureful switch available, it just had to fit in a case and none of the options have one port facing backwards so it doesn't require an ugly loop cable on the front of the final case.
It's also possible to make a managed switch with that card alone, there is a footprint for a NOR flash chip, if you load the netgear GS105E firmware on that (which is available on the netgear website) then it will be just a GS105E without leds and one port on the back.
1000M networks are THE standard, no idea where you got the idea they were obsolete.
It'll be a combination of:
1. Throughput - say you use usb adapters, in a lot of ways usb is a shared bus so you'll run into max bandwidth quickly. This is especially because data will have to go in and then out, all the way to the cpu
2. Latency - because you're using software to do switching, it'll add time to process each packet and send it back out tne right place. You've also go any other interface latency adding to it
3. Power usage - eacj adapter will have it's own full network phy and hardware, which will increase the power draw. Combined with all the extra processing above and now your power usage is even higher. This means you also loae out on hardware offloading and other performance enhancememts that generally reduce power usage because less of the system is involved in mocing packets around
4. Features (potentially) - this will depend a lot on the hardware you choose, some of those cheap gigabit usb adapters i've tried didn't work with vlans and other features properly. But if you say load up a bunch of nice pcie cards with 1 or more ports thst support everything (never had issues with pcie ones) then you can now actually get a lot of features that are otherwise difficult or impossible on simpler hardware (though at that point you're doing routing more tham switching, but thay flexibility is why you'd potentially do this).
We're only talking 1 gbit ethernet here, so you can have multiple of those ports on PCIe.
I have a PCIe card here with two 2.5 gbit on it (don't remember exactly how much it was on Ali but between 20 and 40 EUR) and I can saturate both with iperf3. Since the example only uses 4 ports, it should be easy to make a simple router with just two PCIe cards. But there's probably 4x 1 gbit PCIe out there, too. And if you use 1 gbit fiber, that wouldn't cost much power nor would it need much speed. If your uplink is DSL, you could use a VigorNIC 132.
As everyone else is saying, power usage should be less this way. Probably less latency to traverse the switch than a software bridge, too. Switching should continue to function if the host OS crashes, combined with a watchdog and recovery, you could have a more available system where maybe some things don't work for a brief interval, but much better than a software bridge (assuming the switch chip doesn't crash or get stuck, anyway).
It depends on what your goals are though. If you want to inspect all traffic passing through the switch, having 4 interfaces is clearly better. If your host based switch is also doing a lot of communication, 4 interfaces gives you 4gbps from the hkst rather than 1gbps. Etc.
Ex-enterprise quad 1G cards are $15 or less on ebay. I'm partial to silicom quad bypass 1g_ PEG4BPI-SD; the bypass feature can be fun, and they're cheaper cause they're weird (you can mostly configure them to be 'standard nics' once and then plug them into anything without much fuss, but getting there can be challenging. Early ones come with pci ids set to silicom as the vendor and subvendor which makes them harder to use; the -SD cards have intel vendor id and silicom subvendor, so the normal driver will attach.
4x10g ports would be more to manage, and you might not have enough throughput for software bridging, depending on the host system. And quad port 10g cards are harder to find. 2x10G is reasonably priced though, if you're patient.
Your CPU will be in the data path there. Switches do their packet switching on ASIC rather than CPU. So it depends how good your CPU is, and generally that's not an effective use of whatever compute power you've got.
This is more efficient if most of the traffic will get switched, seperate adapters is more efficient if most of it needs to be routed.