Virtio-Balloon Deep Dive and How to Disable It on a VPS

Preface#

I recently bought a VPS whose provider had Virtio-Balloon enabled by default, which made the system very unstable. So I decided to dig into how Virtio-Balloon actually works.

The Virtio-Balloon driver acts like a memory “balloon” that can be used to dynamically adjust guest memory.

Feature Overview#

The Virtio-Balloon driver works by allocating memory inside the guest VM, then notifying qemu of that allocation. qemu then frees the corresponding memory on the host side so that other VMs can use it, achieving memory overcommit and reuse.

To enable this feature, you must add the Virtio-Balloon backend device at VM startup, and install the Virtio-Balloon driver inside the guest.

Installation#

Add the device. If you use libvirt, add the following XML snippet to the VM definition:

1
 <devices>
2
 <memballoon model='virtio'>
3
      <alias name='balloon0'/>
4
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
5
      <stats period='10'/>
6
    </memballoon>
7
  </devices>

For qemu command-line, add the following device:

1
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0×4

Install the driver. For Windows, there are plenty of tutorials online, so we’ll skip that here. On Linux, the Virtio-Balloon driver has been in the kernel for a long time, so mainstream distributions usually ship it by default.

1
[root@localhost _posts]# modinfo virtio-balloon
2
filename:       /lib/modules/3.10.0-327.el7.x86_64/kernel/drivers/virtio/virtio_balloon.ko
3
license:        GPL
4
description:    Virtio balloon driver
5
rhelversion:    7.2
6
srcversion:     F2D65C53D0AFD06A3668942
7
alias:          virtio:d00000005v*
8
depends:        virtio,virtio_ring
9
intree:         Y
10
vermagic:       3.10.0-327.el7.x86_64 SMP mod_unload modversions
11
signer:         CentOS Linux kernel signing key
12
sig_key:        79:AD:88:6A:11:3C:A0:22:35:26:33:6C:0F:82:5B:8A:94:29:6A:B3
13
sig_hashalgo:   sha256

Usage#

With the above in place, once the VM is started you can begin using the ballooning feature.

Check memory usage via libvirt:

1
virsh # dommemstat test

Check via qemu’s HMP monitor:

1
info balloon

Set the target memory size for the VM. Note this directly sets the current usable memory of the VM. For example, if the VM initially has 8G and you want to reclaim 2G, you should set the current memory to 6G.

On the libvirt side:

1
virsh # setmem test 4096

In qemu’s HMP:

1
balloon 4096

Code Analysis#

Let’s look at Virtio-Balloon from the code perspective. Starting from the Linux driver side: compared to other virtio drivers, the balloon driver is relatively small and easy to read. All its code lives in drivers/virtio/virtio_balloon.c.

First, look at the definition of virtio_balloon_driver. It’s easy to see there aren’t many callbacks: virtballoon_probe (called on driver load), virtballoon_remove (on unload), and virtballoon_changed. That strongly suggests most of the core logic is in virtballoon_changed.

1
static struct virtio_driver virtio_balloon_driver = {
2
    .feature_table = features,
3
    .feature_table_size = ARRAY_SIZE(features),
4
    .driver.name =  KBUILD_MODNAME,
5
    .driver.owner = THIS_MODULE,
6
    .id_table = id_table,
7
    .probe =    virtballoon_probe,
8
    .remove =   virtballoon_remove,
9
    .config_changed = virtballoon_changed,
10
#ifdef CONFIG_PM_SLEEP
11
    .freeze =   virtballoon_freeze,
12
    .restore =  virtballoon_restore,
13
#endif
14
};

The virtballoon_changed function itself is quite simple: it just queues a work item whose main job is to run the update_balloon_size_work callback.

1
static void virtballoon_changed(struct virtio_device *vdev)
2
{
3
    struct virtio_balloon *vb = vdev->priv;
4
    unsigned long flags;
5

6
    spin_lock_irqsave(&vb->stop_update_lock, flags);
7
    if (!vb->stop_update)
8
        queue_work(system_freezable_wq, &vb->update_balloon_size_work);
9
    spin_unlock_irqrestore(&vb->stop_update_lock, flags);
10
}

To really understand this work item, we need to look at the Virtio-Balloon initialization function virtballoon_probe. From the snippet below you can see that on initialization, it registers the stats_request callback to handle stats requests coming from the backend. It also defines two work items: update_balloon_size_func for adjusting memory and update_balloon_stats_func for updating memory statistics—this latter one becomes obvious when you skim the code again.

1
static int virtballoon_probe(struct virtio_device *vdev)
2
{
3
  struct virtqueue *vqs[3];
4
    vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request };
5
    static const char * const names[] = { "inflate", "deflate", "stats" };
6
    int err, nvqs;
7

8
    nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
9
    err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names,
10
            NULL);
11

12
 INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func);
13
 INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func);
14
}

Let’s first introduce the basic update_balloon_size_func functionality. The code is very straightforward: it gets the diff between the current and target size, then either inflates or deflates the balloon, and finally calls update_balloon_size to refresh the current memory info.

1
static void update_balloon_size_func(struct work_struct *work)
2
{
3
    struct virtio_balloon *vb;
4
    s64 diff;
5

6
    vb = container_of(work, struct virtio_balloon,
7
              update_balloon_size_work);
8
    diff = towards_target(vb);
9

10
    if (diff > 0)
11
        diff -= fill_balloon(vb, diff);
12
    else if (diff < 0)
13
        diff += leak_balloon(vb, -diff);
14
    update_balloon_size(vb);
15

16
    if (diff)
17
        queue_work(system_freezable_wq, work);
18
}

fill_balloon and leak_balloon are quite similar, so we’ll just look at fill_balloon. It calls balloon_page_enqueue to add pages to the balloon, then calls tell_host to update the host-side structures. balloon_page_enqueue is a helper implemented by the kernel for balloon-style page accounting. Regardless of whether it’s KVM or Xen ballooning, they all end up calling this function, which lives in mm/balloon_compaction.c. We won’t go deeper here, since this article focuses on Virtio-Balloon itself; we’ll revisit this function when we dive into memory management internals.

1
static unsigned fill_balloon(struct virtio_balloon *vb, size_t num)
2
{
3
    struct balloon_dev_info *vb_dev_info = &vb->vb_dev_info;
4
    unsigned num_allocated_pages;
5

6
    /* We can only do one array worth at a time. */
7
    num = min(num, ARRAY_SIZE(vb->pfns));
8

9
    mutex_lock(&vb->balloon_lock);
10
    for (vb->num_pfns = 0; vb->num_pfns < num;
11
         vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
12
        struct page *page = balloon_page_enqueue(vb_dev_info);
13

14
        if (!page) {
15
            dev_info_ratelimited(&vb->vdev->dev,
16
                         "Out of puff! Can't get %u pages\n",
17
                         VIRTIO_BALLOON_PAGES_PER_PAGE);
18
            /* Sleep for at least 1/5 of a second before retry. */
19
            msleep(200);
20
            break;
21
        }
22
        set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
23
        vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
24
        if (!virtio_has_feature(vb->vdev,
25
                    VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
26
            adjust_managed_page_count(page, -1);
27
    }
28

29
    num_allocated_pages = vb->num_pfns;
30
    /* Did we get any? */
31
    if (vb->num_pfns != 0)
32
        tell_host(vb, vb->inflate_vq);
33
    mutex_unlock(&vb->balloon_lock);
34

35
    return num_allocated_pages;
36
}

With the basic functionality covered, let’s go back to the original question of this article: the Virtio-Balloon driver also implements a periodic memory monitoring feature. As we saw during initialization, when the backend issues a stat command, stats_request triggers another work item, update_balloon_stats_func.

1
static void stats_request(struct virtqueue *vq)
2
{
3
    struct virtio_balloon *vb = vq->vdev->priv;
4

5
    spin_lock(&vb->stop_update_lock);
6
    if (!vb->stop_update)
7
        queue_work(system_freezable_wq, &vb->update_balloon_stats_work);
8
    spin_unlock(&vb->stop_update_lock);
9
}

update_balloon_stats_func mainly calls stats_handle_request:

1
static void update_balloon_stats_func(struct work_struct *work)
2
{
3
    struct virtio_balloon *vb;
4

5
    vb = container_of(work, struct virtio_balloon,
6
              update_balloon_stats_work);
7
    stats_handle_request(vb);
8
}

stats_handle_request gathers all the statistics into the vb struct and then sends vb back to the backend.

1
static void stats_handle_request(struct virtio_balloon *vb)
2
{
3
    struct virtqueue *vq;
4
    struct scatterlist sg;
5
    unsigned int len;
6

7
    update_balloon_stats(vb);
8

9
    vq = vb->stats_vq;
10
    if (!virtqueue_get_buf(vq, &len))
11
        return;
12
    sg_init_one(&sg, vb->stats, sizeof(vb->stats));
13
    virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
14
    virtqueue_kick(vq);
15
}

update_balloon_stats is what actually samples the guest memory usage.

1
static void update_balloon_stats(struct virtio_balloon *vb)
2
{
3
    unsigned long events[NR_VM_EVENT_ITEMS];
4
    struct sysinfo i;
5
    int idx = 0;
6
    long available;
7

8
    all_vm_events(events);
9
    si_meminfo(&i);
10

11
    available = si_mem_available();
12

13
    update_stat(vb, idx++, VIRTIO_BALLOON_S_SWAP_IN,
14
                pages_to_bytes(events[PSWPIN]));
15
    update_stat(vb, idx++, VIRTIO_BALLOON_S_SWAP_OUT,
16
                pages_to_bytes(events[PSWPOUT]));
17
    update_stat(vb, idx++, VIRTIO_BALLOON_S_MAJFLT, events[PGMAJFAULT]);
18
    update_stat(vb, idx++, VIRTIO_BALLOON_S_MINFLT, events[PGFAULT]);
19
    update_stat(vb, idx++, VIRTIO_BALLOON_S_MEMFREE,
20
                pages_to_bytes(i.freeram));
21
    update_stat(vb, idx++, VIRTIO_BALLOON_S_MEMTOT,
22
                pages_to_bytes(i.totalram));
23
    update_stat(vb, idx++, VIRTIO_BALLOON_S_AVAIL,
24
                pages_to_bytes(available));
25
}

At this point we’ve covered the key parts of the Virtio-Balloon driver implementation.

The qemu backend code is relatively simple. During balloon device initialization, it calls qemu_add_balloon_handler to register the callbacks that will be triggered by QMP commands. It also initializes the virtqueues ivq, dvq, and svq to receive requests and stats, and then invokes the relevant callbacks.

1
static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
2
{
3
    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
4
    VirtIOBalloon *s = VIRTIO_BALLOON(dev);
5
    int ret;
6

7
    virtio_init(vdev, "virtio-balloon", VIRTIO_ID_BALLOON,
8
                sizeof(struct virtio_balloon_config));
9

10
    ret = qemu_add_balloon_handler(virtio_balloon_to_target,
11
                                   virtio_balloon_stat, s);
12

13
    if (ret < 0) {
14
        error_setg(errp, "Adding balloon handler failed");
15
        virtio_cleanup(vdev);
16
        return;
17
    }
18

19
    s->ivq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
20
    s->dvq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
21
    s->svq = virtio_add_queue(vdev, 128, virtio_balloon_receive_stats);
22
    reset_stats(s);
23

24
    register_savevm(dev, "virtio-balloon", -1, 1,
25
                    virtio_balloon_save, virtio_balloon_load, s);
26

27
    object_property_add(OBJECT(dev), "guest-stats", "guest statistics",
28
                        balloon_stats_get_all, NULL, NULL, s, NULL);
29

30
    object_property_add(OBJECT(dev), "guest-stats-polling-interval", "int",
31
                        balloon_stats_get_poll_interval,
32
                        balloon_stats_set_poll_interval,
33
                        NULL, s, NULL);
34
}

Using the memory stats refresh path as an example (memory grow/shrink is similar): after registering the guest-stats-polling-interval property, you can configure a polling period. Once set, qemu starts a periodic timer balloon_stats_poll_cb to query memory stats in real time.

1
static void balloon_stats_set_poll_interval(Object *obj, struct Visitor *v,
2
                                            void *opaque, const char *name,
3
                                            Error **errp)
4
{
5
    VirtIOBalloon *s = opaque;
6
    Error *local_err = NULL;
7
    int64_t value;
8

9
    visit_type_int(v, &value, name, &local_err);
10
    if (local_err) {
11
        error_propagate(errp, local_err);
12
        return;
13
    }
14

15
    if (value < 0) {
16
        error_setg(errp, "timer value must be greater than zero");
17
        return;
18
    }
19

20
    if (value > UINT32_MAX) {
21
        error_setg(errp, "timer value is too big");
22
        return;
23
    }
24

25
    if (value == s->stats_poll_interval) {
26
        return;
27
    }
28

29
    if (value == 0) {
30
        /* timer=0 disables the timer */
31
        balloon_stats_destroy_timer(s);
32
        return;
33
    }
34

35
    if (balloon_stats_enabled(s)) {
36
        /* timer interval change */
37
        s->stats_poll_interval = value;
38
        balloon_stats_change_timer(s, value);
39
        return;
40
    }
41

42
    /* create a new timer */
43
    g_assert(s->stats_timer == NULL);
44
    s->stats_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, balloon_stats_poll_cb, s);
45
    s->stats_poll_interval = value;
46
    balloon_stats_change_timer(s, 0);
47
}

virtio_balloon_receive_stats reads the stats structure we discussed earlier from the virtqueue and updates the in-memory stats via balloon_stats_enabled.

1
static void virtio_balloon_receive_stats(VirtIODevice *vdev, VirtQueue *vq)
2
{
3
    VirtIOBalloon *s = VIRTIO_BALLOON(vdev);
4
    VirtQueueElement *elem = &s->stats_vq_elem;
5
    VirtIOBalloonStat stat;
6
    size_t offset = 0;
7
    qemu_timeval tv;
8

9
    if (!virtqueue_pop(vq, elem)) {
10
        goto out;
11
    }
12

13
    /* Initialize the stats to get rid of any stale values.  This is only
14
     * needed to handle the case where a guest supports fewer stats than it
15
     * used to (ie. it has booted into an old kernel).
16
     */
17
    reset_stats(s);
18

19
    while (iov_to_buf(elem->out_sg, elem->out_num, offset, &stat, sizeof(stat))
20
           == sizeof(stat)) {
21
        uint16_t tag = virtio_tswap16(vdev, stat.tag);
22
        uint64_t val = virtio_tswap64(vdev, stat.val);
23

24
        offset += sizeof(stat);
25
        if (tag < VIRTIO_BALLOON_S_NR)
26
            s->stats[tag] = val;
27
    }
28
    s->stats_vq_offset = offset;
29

30
    if (qemu_gettimeofday(&tv) < 0) {
31
        fprintf(stderr, "warning: %s: failed to get time of day\n", __func__);
32
        goto out;
33
    }
34

35
    s->stats_last_update = tv.tv_sec;
36

37
out:
38
    if (balloon_stats_enabled(s)) {
39
        balloon_stats_change_timer(s, s->stats_poll_interval);
40
    }
41
}

Summary#

We’ve now walked through the implementation of Virtio-Balloon and its periodic memory statistics feature. Memory overcommit via Virtio-Balloon has some inherent issues:

The guest can perceive memory size changes. Ballooning is implemented in the kernel and works by changing the visible amount of memory, which can be unfriendly to applications.
Effective memory overcommit requires real-time monitoring: when a guest’s memory usage rises, the system must promptly return memory, which is hard to do robustly at the hypervisor level alone.
Balloon-based overcommit doesn’t truly provide redundant memory. It’s essentially just “robbing Peter to pay Paul”—when all guests are under heavy memory load, you can’t exceed the physical memory limit.

How to Disable Virtio-Balloon#

Check whether the module is loaded: use lsmod to list loaded modules and see whether virtio_balloon is present. Run:

_**lsmod | grep virtio_balloon**_

If there is no output, the module is not loaded.
Unload the module with rmmod: if virtio_balloon is loaded, you can unload it with:

**_sudo rmmod virtio_balloon_**

This attempts to remove the virtio_balloon module. If it succeeds, you shouldn’t see any error messages.

How to Prevent It from Loading at Boot#

Go to the /etc/modprobe.d/ directory by running:

**_sudo cd /etc/modprobe.d/_**
Create a new configuration file, e.g. blacklist-virtio-balloon.conf:

**_sudo nano /etc/modprobe.d/blacklist-virtio-balloon.conf_**
Add a rule to blacklist the module. In the file, add the following line:

**_blacklist virtio_balloon_**

This tells the system to blacklist virtio_balloon during boot.
Save and close the file: press Ctrl + X, then Y to confirm saving.
Update initramfs by running:

**_sudo update-initramfs -u_**