In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat.
On the destination laptop:
$ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M
$ nc x.x.x.x 1234 </dev/nvme0nX
That said I didn't know about that Linux kernel module nvme-tcp. We learn new things every day :) I see that its utility is more for mounting a filesystem over a remote NVMe, rather than accessing it raw with dd.
Edit: on Linux the maximum pipe buffer size is 64kB so the dd bs=X argument doesn't technically need to be larger than that. But bs=1M doesn't hurt (it buffers the 64kB reads until 1MB has been received) and it's future-proof if the pipe sizes is ever increased :) Some versions of netcat have options to control the input and output block size which would alleviate the need to use dd bs=X but on rescue discs the netcat binary is usually a version without these options.
As a sysadmin, I'd rather use NVMe TCP or Clonezilla to do a slow write rather than trying to go 5% faster with more moving parts and chance to corrupt my drive in the process.
Plus, a it'd be well deserved coffee break.
Considering I'd be going at GigE speeds at best, I'd add "oflag=direct" to bypass caching on the target. A bog standard NVMe can write >300MBps unhindered, so trying to cache is moot.
Lastly, parted can do partition resizing, but given the user is not a power user to begin with, it's just me nitpicking. Nice post otherwise.
NVMe/TCP or Clonezilla are vastly more moving parts and chances to mess up the options, compared to dd. In fact, the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) By comparison, the dd on the source is read-only, and the dd on the destination only accepts the first connection (yours) and no one else on the network can write to the disk.
I strongly recommend against oflag=direct as in this specific use case it will always degrade performance. Read the O_DIRECT section in open(2). Or try it. Basically using oflag=direct locks the buffer so dd will have to wait for the block to be written by the kernel to disk until it can start reading the data again to fill the buffer with the next block, thereby reducing performance.
I won't be bothered in a home network.
...and one of these moving parts is image integrity and write integrity verification, allowing byte-by-byte integrity during imaging and after write.
Unless you're getting a bottom of the barrel NVMe, all of them have DRAM caches and do their own write caching independent of O_DIRECT, which only bypasses OS caches. Unless the pipe you have has higher throughput than your drive, caching in the storage device's controller ensures optimal write speeds.
I can hit theoretical maximum write speeds of all my SSDs (internal or external) with O_DIRECT. When the pipe is fatter or the device can't sustain that speeds, things go south, but this is why we have knobs.
When you don't use O_DIRECT in these cases, you see initial speed surge maybe, but total time doesn't reduce.
TL;DR: When you're getting your data at 100MBps at most, using O_DIRECT on an SSD with 1GBps write speeds doesn't affect anything. You're not saturating anything on the pipe.
Just did a small test:
Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.If the drive didn't have its own cache, caching on the OS side would have more profound effect since I can queue more writes to device and pool them at RAM.
Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s.
This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be able to keep up with O_DIRECT.
Another thing you are doing wrong: benchmarking with /dev/zero. Many NVMe do transparent compression so writing zeroes is faster than writing random data and thus not a realistic benchmark.
PS: to clarify I am very well aware that not using O_DIRECT gives the impression initial writes are faster as they just fill the buffer cache. I am taking about sustained I/O performance over minutes as measured with, for example, iostat. You are talking to someone who has been doing Linux sysadmin and perf optimizations for 25 years :)
PPS: verifying data integrity is easy with the dd solution. I usually run "sha1sum /dev/nvme0nX" on both source and destination.
PPPS: I don't think Clonezilla is even capable of doing something similar (copying a remote disk to local disk without storing an intermediate disk image).
I noted that the bus I connected the device has 500MBps bandwidth theoretical, no?
To cite myself:
Yes USB3.0 is 500 MB/s but are you sure your bus is 3.0? It would imply your machine is 10+ years old. Most likely it's 3.1 or newer which is 1000 MB/s. And again, benchmarking /dev/zero is invalid anyway as I explained (transparent compression)
TIL they have been sneaking versions of USB in while I haven't been paying attention. Even on hardware I own. Thanks for that.
No, it wouldn't imply the machine is 10+ years old. Even a state-of-the-art motherboard like the Gigabyte Z790 D AX (which became available in my country today) has more USB 3 gen1 (5Gbps) ports than gen2 (10Gbps).
The 5Gbps ports are just marketed as "USB 3.1" instead of "USB 3.0" these days, because USB naming is confusing and the important part is the "gen x".
I wonder how using tee to compute the hash in parallel would affect the overall performance.
dd followed by sha1sum on each end is still very few moving parts and should still be quite fast.
Yes, in the laptop and one-off case, that's true.
In a data center it's not (this is when I use clonezilla 99.9% of the time, tbf).
I don't see how you can consider the nvme over tcp version less moving parts.
dd is installed on every system, and if you don't have nc you can still use ssh and sacrifice a bit of performance.
NVMe over TCP encapsulates and shows me the remote device as is. Just a block device.
I just copy that block device with "dd", that's all. It's just a dumb pipe encapsulated with TCP, which is already battle tested enough.
Moreover, if I have fatter pipe, I can tune dd for better performance with a single command.
netcat encapsulates data just the same (although in a different manner), and it's even more battle-tested. NVMe over TCP use case is to actually use the remote disk over the network as it were local. If you just need to dump a whole disk like in the article, dd+netcat (or even just netcat, as someone pointed out) will work just the same.
This use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hurt as well. I prefer to just nc -l -p 1234 > /dev/nvme0nX.
Isn't `nc -l -p 1234 > /dev/nvme0nX` working by accident (relying on that netcat is buffering its output in multiples of disk block size)?
No — the kernel buffers non-O_DIRECT writes to block devices to ensure correctness.
Larger writes will be more efficient, however, if only due to reduced system call overhead.
While not necessary when writing an image with the correct block size for the target device, even partial block overwrites work fine:
Partial block overwrites may (= will, unless the block to be overwritten is in the kernel's buffer cache) require a read/modify/write operation, but this is transparent to the application.Finally, note that this applies to most block devices, but tape devices work differently: partial overwrites are not supported, and, in variable block mode, the size of individual write calls determines the resulting tape block sizes.
How about `truncate -s 512 foo`?
Somehow I had thought even in buffered mode the kernel would only accept block-aligned and sized I/O. TIL.
Your exact command works reliably but is inefficient. And it works by design, not accident. For starters, the default block size in most netcat implementations is tiny like 4 kB or less. So there is a higher CPU and I/O overhead. And if netcat does a partial or small read less than 4 kB, when it writes the partial block to the nvme disk, the kernel would take care of reading a full 4kB block from the nvme disk, updating it with the partial data block, and rewriting the full 4kB block to the disk, which is what makes it work, albeit inefficiently.
According to the documentation of dd, "iflag=fullblock" is required only when dd is used with the "count=" option.
Otherwise, i.e. when dd has to read the entire input file because there is no "count=" option, "iflag=fullblock" does not have any documented effect.
From "info dd":
"If short reads occur, as could be the case when reading from a pipe for example, ‘iflag=fullblock’ ensures that ‘count=’ counts complete input blocks rather than input read operations."
Thank you for the correction -- it is likely that I did use count= when I ran into this some 10 years ago (and been paranoid about ever since). I thought a chunk of data was missing in the middle of the output file, causing everything after that to be shifted over, but I'm probably misremembering.
Partial reads won't corrupt the data. Dd will issue other read() until 1MB of data is buffered. The iflag=fullblock is only useful when counting or skipping bytes or doing direct I/O. See line 1647: https://github.com/coreutils/coreutils/blob/master/src/dd.c#...
I would include bs=1M and oflag=direct for some extra speed.
I guess most people don't have faster local network than an SSD can transfer.
I wonder though, for those people who do, does a concurrent I/O block device replicator tool exist?
Btw, you might want also use pv in the pipeline to see an ETA, although it might have a small impact on performance.
I doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access.
Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection.
If we're looking at state-of-the-art consumer tech, the fastest you're getting is a USB4 40Gbit machine-to-machine transfer - but at that point you probably have something like the Crucial T700, which has a sequential write speed of 11.800 MByte/s.
The enterprise world probably doesn't look too different. You'd need a 100Gbit NIC to saturate even a single modern SSD, but any machine with such a NIC is more likely to have closer to half a dozen SSDs. At that point you're starting to be more worried about things like memory bandwidth instead. [0]
[0]: http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...
Can't find corroboration for the assertion 'SSDs are an awful lot better at sequential writes than random writes'.
Doesn't make sense at first glance. There's no head to move, as in an old-style hard drive. What else could make random write take longer on an SSD?
The key aspect is that such memory generally works on a "block" level so making any smaller-than-block write on a SSD requires reading a whole block (which can be quite large), erasing that whole block and then writing back the whole modified block; as you physically can't toggle a bit without erasing the whole block first.
So if large sequential writes mean that you only write full whole blocks, that can be done much faster than writing the same data in random order.
In practice, flash based SSDs basically never do a full read-modify-write cycle to do an in-place update of an entire erase block. They just write the new data elsewhere and keep track of the fragmentation (consequently, sequential reads of data that wasn't written sequentially may not be as fast as sequential reads of data that was written sequentially).
RMW cycles (though not in-place) are common for writes smaller than a NAND page (eg. 16kB) and basically unavoidable for writes smaller than the FTL's 4kB granularity
The main problem is that random writes tend to be smaller than the NAND flash erase block size, which is in the range of several MB.
You can check literally any SSD benchmark that tests both random and sequential IO. They're both vastly better than a mechanical hard drive, but sequential IO is still faster than random IO.
You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get expensive but for the "home lab" crowd since most of these cards are dual port a three-node mesh can be had for just a few hundred bucks. I've had a 40G link to my home server for a few years now mostly just because I could do it for less than the cost of a single hard drive.
In EC2, most of the "storage optimized" instances (which have the largest/fastest SSDs) generally have more advertised network throughput than SSD throughput, by a factor usually in the range of 1 to 2 (though it depends on exactly how you count it, e.g., how you normalize for the full-duplex nature of network speeds and same for SSD).
dd has status=progress to show bytes read/written now, I just use that
Aside, I guess nvme-tcp would result in less writes as you only copy files in stead of writing the whole disk over?
Not if you use it with dd, which will copy the blank space too
Not if you use it with dd, which will copy the blank space too
Seems awesome. Can you please tell us how to use gzip or lz4 to do the imaging?
If you search for “dd gzip” or “dd lz4” you can find several ways to do this. In general, interpose a gzip compression command between the sending dd and netcat, and a corresponding decompression command between the receiving netcat and dd.
For example: https://unix.stackexchange.com/questions/632267
Would be much better to hook this up to dump and restore. It'll only copy used data and you can do it while the source system is online.
For compression the rule is that you don't do it if the CPU can't compress faster than the network.
Note that you can increase pipe buffer, I think default maximum size is usually around 1MB. A bit tricky to do from command line, one possible implementation being https://unix.stackexchange.com/a/328364
This is exactly that I usually do, it works like a charm
Agree, but I'd suggest zstd instead of gzip (or lz4 is fine).
Yep I've done this and it works in a pinch. 1Gb/s is also a reasonable fraction of SATA speeds too.
I am not often speechless, but this hit the spot. Well done Sir!
Where does one learn this black art?
Just cat the blockdev to a bash socket
Sir, i dont upvote much, but your post deserves a double up, at least
I came here to suggest similar. I usually go with
(with some switches to select better block size and tell mbuffer to send/receive from a TCP socket)If it's on a system with a fast enough processor I can save considerable time by compressing the stream over the network connection. This is particularly true when sending a relatively fresh installation where lots of the space on the source is zeroes.
It's a little grimy, but if you use `pv` instead of `dd` on both ends you don't have to worry about specifying a sensible block size and it'll give you nice progression graphs too.