Accelerating Hetzner HDDs with Bcache and NVMe

Introduction#

When doing large downloads and reads recently, write speed often became unstable due to HDD write bottlenecks. Throughput could drop straight from 100 MB/s to B/s — a.k.a. the dreaded I/O stall.

Bcache Overview#

1
bcache 是一个 Linux 内核块层超速缓存。它允许使用一个或多个高速磁盘驱动器（例如 SSD）作为一个或多个速度低得多的硬盘的超速缓存。bcache 支持直写和写回，不受所用文件系统的约束。
2

3
主要功能：
4
1，可以使用单个超速缓存设备来超速缓存任意数量的后备设备。在运行时可以挂接和分离已装入及使用中的后备设备。
5
2，在非正常关机后恢复 - 只有在超速缓存与后备设备一致后才完成写入。
6
3，SSD 拥塞时限制传至 SSD 的流量。
7
4，高效的写回实施方案。脏数据始终按排序顺序写出。
8
5，稳定可靠，可在生产环境中使用。
9

10
以下教程基于Debian12

Bcache is a Linux kernel block-layer cache. It lets you use one or more high-speed disks (for example SSDs or NVMes) as a cache for one or more much slower hard drives. Bcache supports write-through and write-back modes and is independent of the filesystem you put on top.

Key features:

Use a single cache device to accelerate any number of backing devices. Backing devices currently in use can be attached or detached at runtime.
Recovery after unclean shutdowns – writes are only considered complete once cache and backing device are consistent.
Throttles traffic to the SSD when it becomes congested.
Efficient write-back implementation – dirty data is always flushed out in sorted order.
Stable and reliable, suitable for production use.

The steps below are based on Debian 12.

Prerequisites#

A RAID array built from fourteen 22 TB HDDs
nvme0n1 as the system disk (7.68 TB)
nvme1n1 as the cache disk (7.68 TB)
HDDs and the NVMe cache device already partitioned/formatted as you need

Below is what the final setup will look like:

1
root@Debian ~ # lsblk
2
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
3
sda           8:0    1    20T  0 disk
4
└─md127       9:127  0 280.1T  0 raid0
5
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
6
sdb           8:16   1    20T  0 disk
7
└─md127       9:127  0 280.1T  0 raid0
8
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
9
sdc           8:32   1    20T  0 disk
10
└─md127       9:127  0 280.1T  0 raid0
11
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
12
sdd           8:48   1    20T  0 disk
13
└─md127       9:127  0 280.1T  0 raid0
14
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
15
sde           8:64   1    20T  0 disk
16
└─md127       9:127  0 280.1T  0 raid0
17
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
18
sdf           8:80   1    20T  0 disk
19
└─md127       9:127  0 280.1T  0 raid0
20
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
21
sdg           8:96   1    20T  0 disk
22
└─md127       9:127  0 280.1T  0 raid0
23
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
24
sdh           8:112  1    20T  0 disk
25
└─md127       9:127  0 280.1T  0 raid0
26
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
27
sdi           8:128  1    20T  0 disk
28
└─md127       9:127  0 280.1T  0 raid0
29
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
30
sdj           8:144  1    20T  0 disk
31
└─md127       9:127  0 280.1T  0 raid0
32
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
33
sdk           8:160  1    20T  0 disk
34
└─md127       9:127  0 280.1T  0 raid0
35
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
36
sdl           8:176  1    20T  0 disk
37
└─md127       9:127  0 280.1T  0 raid0
38
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
39
sdm           8:192  1    20T  0 disk
40
└─md127       9:127  0 280.1T  0 raid0
41
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
42
sdn           8:208  1    20T  0 disk
43
└─md127       9:127  0 280.1T  0 raid0
44
  └─bcache0 252:0    0 280.1T  0 disk  /hdd
45
nvme1n1     259:0    0     7T  0 disk
46
├─nvme1n1p1 259:2    0     1G  0 part
47
├─nvme1n1p2 259:3    0     7T  0 part
48
│ └─bcache0 252:0    0 280.1T  0 disk  /hdd
49
└─nvme1n1p3 259:4    0     1M  0 part
50
nvme0n1     259:1    0     7T  0 disk
51
├─nvme0n1p1 259:5    0     1G  0 part  /boot
52
├─nvme0n1p2 259:6    0     7T  0 part  /
53
└─nvme0n1p3 259:7    0     1M  0 part

Enable Bcache in the Kernel#

1
modprobe bcachelsmod |grep bcache

Install bcache-tools#

1
apt install bcache-tools

Wipe Existing Metadata#

1
wipefs -a /dev/md127
2
wipefs -a /dev/nvme1n1p2

Create the Backing Device#

1
make-bcache -B /dev/md127

Create the Cache Device#

1
make-bcache -C /dev/nvme1n1p2

Check Current Block Devices#

1
root@Debian ~ # lsblk
2
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
3
sda           8:0    1    20T  0 disk
4
└─md127       9:127  0 280.1T  0 raid0
5
  └─bcache0 252:0    0 280.1T  0 disk
6
sdb           8:16   1    20T  0 disk
7
└─md127       9:127  0 280.1T  0 raid0
8
  └─bcache0 252:0    0 280.1T  0 disk
9
sdc           8:32   1    20T  0 disk
10
└─md127       9:127  0 280.1T  0 raid0
11
  └─bcache0 252:0    0 280.1T  0 disk
12
sdd           8:48   1    20T  0 disk
13
└─md127       9:127  0 280.1T  0 raid0
14
  └─bcache0 252:0    0 280.1T  0 disk
15
sde           8:64   1    20T  0 disk
16
└─md127       9:127  0 280.1T  0 raid0
17
  └─bcache0 252:0    0 280.1T  0 disk
18
sdf           8:80   1    20T  0 disk
19
└─md127       9:127  0 280.1T  0 raid0
20
  └─bcache0 252:0    0 280.1T  0 disk
21
sdg           8:96   1    20T  0 disk
22
└─md127       9:127  0 280.1T  0 raid0
23
  └─bcache0 252:0    0 280.1T  0 disk
24
sdh           8:112  1    20T  0 disk
25
└─md127       9:127  0 280.1T  0 raid0
26
  └─bcache0 252:0    0 280.1T  0 disk
27
sdi           8:128  1    20T  0 disk
28
└─md127       9:127  0 280.1T  0 raid0
29
  └─bcache0 252:0    0 280.1T  0 disk
30
sdj           8:144  1    20T  0 disk
31
└─md127       9:127  0 280.1T  0 raid0
32
  └─bcache0 252:0    0 280.1T  0 disk
33
sdk           8:160  1    20T  0 disk
34
└─md127       9:127  0 280.1T  0 raid0
35
  └─bcache0 252:0    0 280.1T  0 disk
36
sdl           8:176  1    20T  0 disk
37
└─md127       9:127  0 280.1T  0 raid0
38
  └─bcache0 252:0    0 280.1T  0 disk
39
sdm           8:192  1    20T  0 disk
40
└─md127       9:127  0 280.1T  0 raid0
41
  └─bcache0 252:0    0 280.1T  0 disk
42
sdn           8:208  1    20T  0 disk
43
└─md127       9:127  0 280.1T  0 raid0
44
  └─bcache0 252:0    0 280.1T  0 disk
45
nvme1n1     259:0    0     7T  0 disk
46
├─nvme1n1p1 259:2    0     1G  0 part
47
├─nvme1n1p2 259:3    0     7T  0 part
48
└─nvme1n1p3 259:4    0     1M  0 part
49
nvme0n1     259:1    0     7T  0 disk
50
├─nvme0n1p1 259:5    0     1G  0 part  /boot
51
├─nvme0n1p2 259:6    0     7T  0 part  /
52
└─nvme0n1p3 259:7    0     1M  0 part

Get the Cache Device UUID#

1
bcache-super-show /dev/nvme1n1p2
2
如下图所示，就是cset.uuid

The cset.uuid shown in the output is the value you need.

Attach the Cache Device#

1
echo "0c07a77e-3735-410b-adae-60ea5d708009" >/sys/block/bcache0/bcache/attach

Check Current Block Devices Again#

1
root@Debian ~ # lsblkNAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTSsda           8:0    1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdb           8:16   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdc           8:32   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdd           8:48   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sde           8:64   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdf           8:80   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdg           8:96   1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdh           8:112  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdi           8:128  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdj           8:144  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdk           8:160  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdl           8:176  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdm           8:192  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  sdn           8:208  1    20T  0 disk  └─md127       9:127  0 280.1T  0 raid0   └─bcache0 252:0    0 280.1T  0 disk  nvme1n1     259:0    0     7T  0 disk  ├─nvme1n1p1 259:2    0     1G  0 part  ├─nvme1n1p2 259:3    0     7T  0 part  │ └─bcache0 252:0    0 280.1T  0 disk  └─nvme1n1p3 259:4    0     1M  0 part  nvme0n1     259:1    0     7T  0 disk  ├─nvme0n1p1 259:5    0     1G  0 part  /boot├─nvme0n1p2 259:6    0     7T  0 part  /└─nvme0n1p3 259:7    0     1M  0 part

Check Cache State#

no cache: this backing device has no caching device attached
Normal, cache is clean
Normal, write-back enabled and cache is dirty
Error: backing device and cache device are out of sync

1
cat /sys/block/bcache0/bcache/state

Change Cache Policy#

1
Bcache有三种缓存策略

Bcache supports three cache modes:

writeback: data is first written to the cache device and later flushed to the backing disk
writethrough: data is written to both cache and backing disk at the same time (this is the default mode)
writearound: data is written directly to the backing disk

For better performance, switch to writeback mode here.

1
查看缓存模式
2
cat /sys/block/bcache0/bcache/cache_mode
3
修改缓存策略
4
 echo writeback > /sys/block/bcache0/bcache/cache_mode
5
允许缓存顺序I/O（非常重要）
6
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff

Format the Data Device#

1
mkfs.xfs /dev/bcache0

Configure Auto-Mount on Boot#

1
查看设备UUIDblkid /dev/bcache0添加到/etc/fstabvim /etc/fstab添加上面UUIDUUID=f9c51924-f6d5-4466-84ee-14fcfbb1bb14 /hdd                   xfs     defaults        0 0

Test Cache Performance#

YABS Fio Benchmark#

With Cache#

Block Size	4k (IOPS)	64k (IOPS)
Read	240.07 MB/s (60.0k)	1.53 GB/s (24.0k)
Write	240.70 MB/s (60.1k)	1.54 GB/s (24.1k)
Total	480.77 MB/s (120.1k)	3.08 GB/s (48.2k)

Block Size	512k (IOPS)	1m (IOPS)
------	--- ----	---- ----
Read	2.94 GB/s (5.7k)	3.09 GB/s (3.0k)
Write	3.10 GB/s (6.0k)	3.29 GB/s (3.2k)
Total	6.04 GB/s (11.8k)	6.39 GB/s (6.2k)

Without Cache#

Block Size	4k (IOPS)	64k (IOPS)
Read	24.43 MB/s (6.1k)	306.10 MB/s (4.7k)
Write	24.44 MB/s (6.1k)	307.72 MB/s (4.8k)
Total	48.88 MB/s (12.2k)	613.82 MB/s (9.5k)

Block Size	512k (IOPS)	1m (IOPS)
------	--- ----	---- ----
Read	772.93 MB/s (1.5k)	1.88 GB/s (1.8k)
Write	814.00 MB/s (1.5k)	2.01 GB/s (1.9k)
Total	1.58 GB/s (3.0k)	3.90 GB/s (3.8k)