Disk Space Usage Leak In Docker For Mac

(I'm a Docker employee working on this very issue. I guess my plan to take a break by reading HN failed!) You're completely right - it's a problem caused by lack of TRIM in the storage path. In the next beta of Docker for Windows (beta 31 due today hopefully) TRIM should be enabled.

The Mac will take a little longer as we need to switch protocols and do more work on the host side - unfortunately the default Apple filesystem doesn't support sparse files so we can't 'cheat' by simply passing the TRIM down to the filesystem layer. We'll probably need some kind of explicit block-level compaction to shuffle blocks from the end of the file into holes that have been created by TRIM.

We'll probably use AHCI since hyperkit (the Mac hypervisor based on xhyve based on bhyve) lacks a virtio-scsi implementation at the moment. When the VM sends a TRIM ('these blocks aren't used any more, you can free them') we need to make use of this on the host. On Linux/BSD you could tell the host OS that the blocks aren't needed in the file using something like FALLOCFLPUNCHHOLE and the kernel would take care of it e.g. By send TRIMs to the real storage device or shuffling filesystem metadata around. Unfortunately the Mac filesystems in common use don't support this so we'll need to manage this ourselves, e.g.

By moving blocks from the end of the file into the middle and shrinking the file. Everything old is new again. I posted a Docker issue about the default storage driver (DeviceMapper) not correctly freeing unused space nearly three years ago: From what I can tell, that issue was never truly resolved.

The advice is like the adage about finding romance, 'Have enough storage, don't run out of storage.' As long as you follow that advice, you will never have a problem. Oh, you installed Docker on a cloud provider with limited local disk, or on your laptop? Silly you for thinking that a finite amount of storage and the default configuration of Docker was adequate. Forget usage leaks, Docker for Mac doesn't even work on my system, and five days later after reporting it, nobody cares.

It used to work, many versions ago, but they've somehow broken it entirely between redesigning the UI 2-3 times and switching from VirtualBox (which worked) to OSX Virtualization (which so far doesn't). See In addition, they don't seem to care that the whole thing is useless in China -also after emailing, no response. I have contributed to Docker since very early days but am frankly not using it now because the project has for all intents and purposes entered a toxic-to-the-community stage where hype and marketing exceed capacity to resolve issues, total fails are occurring for me on all platforms, and AFAIK nobody in their over-funded San Franciscan office bubble seems to care.

It's a typically arrogant startup: not listening to users, heading for a fall. It's like they've decided to re-invent downloads (badly), re-invent cross-platform hypervisors (badly), re-invent orchestration (badly), re-invent storage (badly) and roll it all up in branded glue. I can't help but wonder if an approach with broader applicability and longevity would segregate the OS (anything container or VM-like at any layer, from Erlang to BSD jails to diskless clusters), and environment paradigm (container-based, paravirtualization-based, bare metal) from the workload and truly enable infrastructure agnoticism by removing the dependency on a single shifting-sands component, and allowing people to A/B test identical workloads on disparate paradigm infrastructures. I tried a few years ago, it worked. Realistically, how they choose to divide their issue resolution is irrelevant except in terms of how much friction it causes with users and developers.

From a consumer perspective, if you can't docker pull then docker is useless, so being a key software feature within the docker binary it's not unrealistic to expect the average person to report directly to docker on github. The fact we have to report fundamental issues that should never have passed basic testing in an extremely well funded company is the really amazing thing here. Even if they wish to maintain a separate issue resolution process via email, a better resolution would be 'I have forwarded this to the email support team, please expect an email' (since github IDs all have email associated anyway, this should be easy). Right now they have maximum friction, and no response at all. You will be pleased to note that email works fine in China (obviously - otherwise how do you think global trade operates?), unless you use NSA/Google as your provider.

Expected behavior Amount of disk space used under /Library/Containers/com.docker.docker/ should shrink when deleting images and/or containers Actual behavior Amount of disk space used under /Library/Containers/com.docker.docker/ doesn't shrink after deleting 50 containers and 141 images (out of a total of 194). My Docker.qcow2 was 41GB both before and after doing these deletions. I've found the quickest way (as mentioned in other issues / forum threads) to fix this is to reset the client which forces the recreation of Docker.qcow2. If you don't want to do that however, there is another fix, once you've cleaned up any dangling images or exited containers. NB: You'll need to install qemu via Homebrew as this process requires qemu-img to recompress the qcow2 disk image. Connect to the VM with screen /Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty and then login as root Fill up the disk with zeroes: dd if=/dev/zero of=/var/tempfile This'll take at least couple of minutes on SSD. I wouldn't bother if you're on spinning rust.

Then you need to rm /var/tempfile, logout of the VM, and quit the Docker client entirely. Now we can recompress the disk: $ pwd /Users/nick/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux $ mv Docker.qcow2 Docker.qcow2.original $ du -hs Docker.qcow2.original 12G Docker.qcow2.original $ qemu-img convert -O qcow2 Docker.qcow2.original Docker.qcow2 $ rm Docker.qcow2.original $ du -hs Docker.qcow2 772M Docker.qcow2. To add to this, I just deleted my qcow2 file and restart docker and disconnect completely from the internet, and docker ps shows no running containers, and docker images -a lists two containers, both of which are.

The current state is:. on edge 18.05.0-ce-mac67 we default to Docker.raw on macOS 10.13.4 (and later) running on APFS, otherwise we use Docker.qcow2. on stable 18.03.1-ce-mac65 we default to Docker.qcow2. The next version of stable will default to Docker.raw on macOS 10.13.4 (and later) running on APFS. Note that APFS (needed by Docker.raw) is not supported by all Apple hardware. I believe it's currently only supported on SSD drives, although it might work on fusion drives / HDDs in the future.

Both Docker.raw and Docker.qcow2 support TRIM for space reclamation. A TRIM command is issued after every image delete event, but can also be manually triggered with docker run -privileged -pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker TRIM on Docker.raw frees space immediately.

TRIM on Docker.qcow2 only marks the blocks as free; they will be physically released later by a garbage collection process- this can take several minutes. I recommend:. use Docker.raw if possible. This requires an SSD, a recent version of macOS (10.13.4+) and running the edge version. Note if you are currently using a Docker.qcow2 you will need to reset to factory defaults and rebuild your images.

monitor images ( docker image ls), stopped containers ( docker container ls -a) and prune old ones regularly ( docker system prune). It's easy to build up stale objects by accident particularly if you use docker run a lot without the -rm argument.

use ls -s to query the size of the Docker.raw to measure the allocated blocks, not the theoretical size of the disk. The disk space usage of Docker on Mac has been difficult to manage and has confused lots of people, see This patch adds a top-level section to the Docker for Mac docs alongside networking which explains - where the space has gone - how to reclaim some of it In future I'd like to add more about why some systems use `Docker.raw` and some `Docker.qcow2` but we're in a transitional state at the moment. It'll be simpler to explain after the next stable version has been released. Signed-off-by: David Scott.

The disk space usage of Docker on Mac has been difficult to manage and has confused lots of people, see This patch adds a top-level section to the Docker for Mac docs alongside networking which explains - where the space has gone - how to reclaim some of it In future I'd like to add more about why some systems use `Docker.raw` and some `Docker.qcow2` but we're in a transitional state at the moment. It'll be simpler to explain after the next stable version has been released. Signed-off-by: David Scott. I just wiped all of my data and upgraded to Docker for Mac 18.06.0-ce-mac70, enabling the raw format by default. Now while decompressing a large.tar.gz archive, I'm running into which appears to still be outstanding.

It would seem the raw format has trouble with large images and files (?). Is there a way to reset Docker for Mac to qcow other than downgrading, resetting data, and re-upgrading?. FWIW with raw I'm still seeing storage leaking even after docker system prune -volumes.

I still need to run docker run -privileged -pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker to reclaim space, however it does return immediately with raw as mentioned. One way that I try to reduce 'churn' within my Docker.qcow2 file is bind-mount external directories. While this breaks the container model, it allows me to regularly rebuild the containers (albeit reusing the datadir) and the idea of having my dev environment have 10GB of working data to simulate an issue within the qcow2 file just fills me with dread.

Reading through this thread, I had some hope, and gave it a whirl: $ docker run -rm -it -privileged -pid=host walkerlee/nsenter -t 1 -m -u -i -n fstrim /var fstrim: /var: the discard operation is not supported Is there something special required in order to allow fstrim to operate? (Given that my Docker.qcow2 file is now approaching 53G (measured with ls -klsh and with du -ksh), I guess I have another 11G before armaggedon.). Have you qcow2 people tried switching to the raw image format? You can do this with the reset instructions you'll find in this thread.

Disk Space Usage Leak In Docker For Mac Download

My understanding is that the raw image format is the way forward so long as you can fulfill the requirements? So while I'm not on the Docker team and this is purely speculative, I would imagine that if what I've said is correct there is probably little chance you'll ever see this issue fixed if you stick with qcow2. I've certainly found a huge improvement with the raw image format.