IPMI

A few notes because there are a few pieces of information which aren't a 100% up to date:

As most people know, Intel have lost the plot and are behind AMD on everything CPU and GPU related (with the tiny exception of the N100 series CPUs from Intel which are perfect for low power and even fanless applications). As such, their CPUs are mostly bought by organisations updating previous environments with like for like replacements for whatever reason (like vSphere's EVC which allows you to run newer processors from the same manufacturer as if they were an older model, and enable hot migration between CPU architectures and thus minimum disruption on hardware refreshes). Pretty much everyone else is getting the better and cheaper (per processing power) AMD CPUs.

Intel NICs are pretty good overall (the X710 which spent a year+ with broken drivers causing silent network failures/downright crashes while still being in the hardware compatibility list of VMware and other "enterprise" vendors notwithstanding), and they're getting more popular for consumer devices too. An Intel CPU on a non-low cost PC/laptop almost always means an Intel NIC (WiFi and/or Ethernet).

For many organisations buying servers, Supermicro* is probably their best option. They're cheaper, have much higher flexibility in terms of form factors, chassis, components, number of whatever slots and mostly reliable. However their support is less reliable than the theoretical Dell/HPE support, so it works best for redundant setups.

Also, the IPMI specification is being replaced by Redfish which is much more complete, secure, normalised and presents normal usable APIs. Any mainline server from a few years ago should have Redfish alongside IPMI.

* There's recent research from Hindenburg, a short selling research firm, that exposes some shady stuff from Supermicro - but their hardware is still top notch and used by the hyperscalers. https://hindenburgresearch.com/smci/

If anyone wants to add remote access capability to machines without IPMI you can try with something like the $30 RISC-V NanoKVM[1][2]. It provides HDMI capture (and encoding), Ethernet/Wifi, ATX power control, and runs a normal linux distribution.

[1] https://www.aliexpress.com/item/1005007369816019.html [2] https://github.com/sipeed/NanoKVM

I really like IPMI, but what I don't like about it for homelab use cases is the additional ~5W of idle power consumption. Using the Gigabyte MC12-LE0 Board with Ryzen Pro 5650 for a home server should be a no brainer costing ~50 bucks, but the higher power consumption is something I'm not totally happy with.

On older devices (like Dell T20/T30) there is Intel AMT which is way less powerful and has security flaws, but at least SOME way to do remote administration when used with MeshCommander (unfortunately discontinued and releases deleted from everywhere - but I luckily saved the MSI and Node-Package on my server).

I'm planning to try out PiKVM (V2) with a Raspberry 4 and a simple 8 bucks USB-HDMI-Capture-Card (https://docs.pikvm.org/v2/). Besides the lack of some features, it looks promising and more universal for devices that do not support remote management in any form.

IPMI and other solutions are nice, but what I would like to have is a standard serial interface to an UEFI shell running at all times. How I access that serial port should be my problem.

IPMI is very common in enterprise servers but very rare elsewhere, much to the consternation of people like me that don't have the space or noise tolerance for a 1U pizzabox in their homes. If you are trying to stick to compact or low-power computers, you'll pretty much have to go without.

One option: you can get IPMI on an Atom-based Supermicro MicroATX board that can be cooled quietly with a small Noctua fan, in a short-depth 1U chassis.

I have an older one at home. The quietness and small size made it much more attractive than the Dell R2x0 models I saw. And it has server features like IPMI and ECC RAM that made it more attractive than a mini-PC, and it was also more reliable than a flimsy RasPi. (I'd be happy with a Dell quiet short-depth 1U with comparable characteristics, but I'm not aware of one.)

Personally, I wouldn't plug the IPMI port into my main LAN, but it's come in handy isolated from that, and was also a little fun to play with.

Can an AST2600 BMC be used as a graphics card by Ubuntu Desktop? I've read that it has a "2D Video Graphic Adapter with PCIe bus interface", which makes it sound like it is exposed to the OS like any other graphics card.

The desktop would be there only for troubleshooting, not for normal stuff which would usually require an iGPU or better.

If yes, would it work out of the box with Ubuntu?

Was IPMI useful in solving the CrowdStrike boondoggle?

Not a fan of the web based OOB management, it's convenient for now but this morning I had a right faff trying to connect to an old iLO3 service, which is new enough to force https but old enough that all it's protocols are deprecated on modern machines. I've used vPro and found it almost deliberately difficult to use, and poorly documented. It could be so easy but just isn't. Like so many things... shakes fist at sky.

A big question with IPMI is what the default keys are. If someone manages to install some extra IPMI keys somewhere in the supply chain, or there's a default key, they can remote admin the computer.[1]

[1] https://www.rapid7.com/blog/post/2013/07/02/a-penetration-te...

Dell is working pretty hard to replace IPMI with Redfish these days. Redfish is an out of band management protocol like IPMI, but using an HTTP/JSON REST API that doesn't assume a secure management network.

https://www.dmtf.org/standards/redfish

Does anyone know about AMD ST? It is the first time I've seen it mentioned. I tried to google it, but without result.

Seen devices with IPMI that had by design unauthenticated admin login to the IPMI from the host side that was not removable. They also could flash IPMI firmware from the host. So if your server with such an IPMI is infected you can't trust reimaging it via IPMI because that can be hijacked as well.

Does anyone else see a pixel or so of mismatch between the background and the text when scrolling?

Intelligent Platform Management Interface

https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...

This is not to make excuses for Intel ME, which is entirely unauditable by third parties and has harbored significant security vulnerabilities in the past. But, remember, we all use one processor architecture from one of two vendors, so Intel doesn't have a whole lot of motivation to do better. Lest you respond that ARM is the way, remember that modern ARM SOCs used in consumer devices have pretty much identical capabilities.

I actually took advantage of one such security vulnerability to unlock my old Moto X's bootloader! The details of the exploit are quite interesting:

https://bits-please.blogspot.com/2015/03/getting-arbitrary-c...

https://bits-please.blogspot.com/2015/08/exploring-qualcomms...

https://bits-please.blogspot.com/2015/08/full-trustzone-expl...

https://bits-please.blogspot.com/2016/02/unlocking-motorola-...

Personally, for smaller deployments (I.e. about 10k cores) I would just self deploy out from some integrator. A Gigabyte/AsRock Rack Motherboard + Epyc 9003 series + some 384 G of RAM with the usual dual PSUs and all that stuff will be $7k/node and can be very power efficient.

The integrated IPMI is pretty good on these and plays well with ipmitool. They'll usually have some Redfish stuff going on.

My takeaways:

"Out-of-band management, sometimes also called lights-out management, identifies a capability that is almost unheard of in clients. A separate, smaller management computer allows for remote access to a server even when it is, say, powered off. The terms out-of-band and in-band in this context emerge from their customery uses in networking and telecom, meaning that out of band management is performed without the use of the standard (we might say "data plane") network connection to a machine. But in practice they have drifted in meaning, and it is probably better to think of out-of-band management as meaning that the operating system and general-purpose components are not required. This might be made clearer by comparison: a very standard example of in-band management would be SSH, a service provided by the software on a computer that allows you to interact with it. Out-of-band management, by contrast, is provided by a dedicated hardware and software stack and does not require the operating system or, traditionally, even the CPU to cooperate."

[...]

"The IPMI software runs on a processor conventionally called the baseboard management controller or BMC, and the terms IPMI and BMC are sometimes used interchangeably. Lights-out management or LOM is mostly an obsolete term but sticks around because HP(E) is a fan of it and continues to call their IPMI implementation Integrated Lights-Out. The BMC should not be confused with the System Management Controller or SMC"

[...]

"IPMI offers "sideband" networking. With sideband management, the BMC communicates directly with the same NIC that the operating system uses. The implementation is a little bit weird: the NIC will pretend to be two different interfaces, mixing IPMI traffic into the same packet stream as host traffic but using a different MAC address."

[...]

"Details of network features vary between IPMI implementations, but there is a standard interface on UDP 623 that can be used for discovery and basic commands. There's often SSH and a web interface, and VNC is pretty common for remote console."

This is arguably the best educational article on IPMI and related subsystems (I did not know about 90% of this!) that I've read for a long time.

Well worth a re-read in the future...

IPMI has its place, but it clearly shows how none of us can trust commercial entities to properly support hardware long term. The OS that runs on IPMI is usually relatively secure when the system is new, but as soon as there's a new CPU socket, the manufacturer starts caring less and less about keeping the older systems up to date, even if the same IPMI hardware is on both old and new boards.

If we could load our own OS on to the IPMI hardware, it's be worlds more useful, because we could safely connect it straight to the Internet. As it is, we need sideband communications (VPN, port forward over ssh, separate network segment, whatever), which makes it so that we end up with lots of extra hardware and configurations just to support IPMI. For large installations, the extra cost is better aggregated, but for small installations, it's a rather large burden. Heck - for colocating a single machine, it's not even worth it.

Since it's not possible to run IPMI directly on the Internet securely, I've ended up pairing machines with Pis of some sort. Doing this makes it just as easy - heck, easier - to use a serial port. That means we're back to the standard serial port control we've had since the days of VAXen and Sun and Alpha systems, which, the more I think about it, made much more sense than a network interface that's insecure.

Hands up if you used a real weasel! Cute advertising imagery.

I depend on idrac. The newer one is bearable. The old Java was torrid.

As for the Intel ME and AMD PSP: these things have a huge role in getting the CPU to state that looks like x86 PC so that the host firmware can actually run. The complexity of the initialization got so high, that doing that in SW on a separate embedded core that is “well-behaved” and can be programmed in C makes more sense than trying to write all that logic in weirdly limited assembly that is inherent in traditional early stage BIOS startup code.

I believe that at least on some HPE ProLiants some part of this low-level initialization is in fact done by iLo, which at that boot stage also directly controls the framebuffer (on G10 you can even briefly see a flash of message like “handing over the console to host”, after which the display reinitializes and starts showing the function key legends). I assume that Dell does something similar, but in the early stages you simply get “Please wait” and giant throbber (and not any kind of progress indication).

Conspiracy theory: I wonder if the ____KVM people are betting that most people won't flash their own firmware, and just keep using what was provided, which may contain malware, both for accessing the network, and controlling the connected hardware.

IPMI is such an improvement on the old ISA-style iochips, and not even sure they are more expensive these days! It's not just the remote/web access which is great, plain 'introspection' with tools like ipmiutil and ipmitool are super useful, even if you don't have a huge rack 'enterprise' installation.

ipmitool has dug me out of many sticky situations in the past, although it needs to be handled as proper oob, in terms of network isolation.

I have found HP iLO very useful on my MicroServer Gen8 with UnRAID.

In the late 90s I installed several Intel-produced servers (i.e. Intel reference platforms shipped as "bare bones" computers w/ RAM and storage to be provided by the integrator) that used the LANDesk Server Manager Pro[0] software and "Emergency Management Card" (EMC) for "lights out" management. (I'm talking about boxes like the AP450GX, BB440FX, RC440FX-- Pentium Pro to early Pentium II timeframe.)

Given how reference code for the x86 platform never dies I've often wondered how much of the current IPMI edifice dates back to this hardware and software. As a point of note, the default password for the Intel LANDesk Emergency Management Card was "calvin". If you worked with earlier versions of the Dell iDRAC this will be a familiar password. I assume it's not coincidence. (As an aside: I was told by an Intel employee that the code name for the EMC was "Hobbes" but I can't find that documented anywhere.)

You see versions of the EMC periodically on eBay.[1] There were both ISA and PCI versions. They're an x86 PC on a card. Some (all?) of them have an integral UPS. They had a PCMCIA slot to attach out-of-band management, an external power supply, and connected to the server motherboard through both the card's host bus interface and via proprietary connectors.

I've downloaded firmware for some of the EMC versions and looked it over. It looks like some of the EMC versions an embedded DOS machine. (This has been one of my idle "when I get to it" projects and I haven't actually tried to reverse-engineer any of the code or get it running under qemu, though I'd like to.)

Third-party companies who sold the Intel reference platforms (Unisys, Fujitsu, ALR/Gateway, NCR) offered these cards, too. It's a good giveaway that they're using an Intel reference platform when you see the card on-offer. References to "LDSM" are a giveaway.

It'd be really wild if anybody on here knows anything about the heritage.

[0] https://www.intel.com/pressroom/archive/releases/1998/ld1030...

[1] https://web.archive.org/web/20240903131630/https://www.ebay....