VHD tools: vhdcompact and regedit

It is terrible to be unsure of what to blog about.  The weight of choice tends to create a vacuum.  This is most obvious when months pass by before anything new is written.  There are certainly things that have happened in that time that are worth writing about.  However, it is sometimes difficult to pick the right one.

A topic dear to my heart this last year is changing VHDs while offline.  Indirectly I have blogged about VHDs and what is inside them.  What has not been discussed is what is really going on with work.

Recently things have changed a bit and the release of the code is becoming much more real.  It is time to spill the beans.

There are two specific challenges I was given for XenClient.  The first was the ability to compact the VHDs as much as possible.  The second was to be able to get/set registry settings.  Both tasks sound fairly simple but turned out to be very hard due to constraints.  First, the tools must run in Linux.  Second, these tools must be able to run with minimum requirements.

The compaction problem was solved quite some time ago.  The core issue is that VHDs have no knowledge of the file system.  Therefore, the VHD cannot safely delete any blocks that have been written to.  In order to delete a block, it requires knowledge of the NTFS usage which, until recently, did not exist.  The tool I wrote, called vhdcompact, corresponds knowledge of VHD and NTFS to determine which blocks can be safely removed.

Without too much effort, the tool can cut out upwards of 30% of a VHD.  Given that VHDs can grow into the tens of gigabytes, this is a large amount of storage.  The real interesting bit is that the disk loses no information since the blocks are not currently being used by anyone anyways.  This only applies to dynamic and difference disks.

The registry editing can only happen once the tools understand NTFS.  Thankfully LIBNTFS handled this in Linux.  Unfortunately no one, that I know of, has created a registry editor for VHDs.  So, again luckily, the registry format has been documented by sources outside Microsoft.  I created code to support registry editing without any Windows code.  It has taken quite some time to get it to the state it is today but it is now possible to make changes in the registry from Linux.

I can hear a few of you say “Why would you do this?”.

The answer is that we need to make some minor adjustments to the VHD before it boots.  It is too late after the VHD has booted.  It is also important for us to be able to save/restore sections of the registry from the hypervisor code.  Without Windows being there, we have to depend solely on ourselves.

It does feel like I am letting the cat out of the bag but the truth is that this stuff is probably old news to most.

One of the side effects of a project like this is that it is now possible to correspond blocks to files.  This means that given any sector in the VHD, we can determine which file owns it.  This, historically, was a difficult task.  Now, it seems like a minor detail in a much larger problem space.

In theory it would be possible to re-parent a difference disk with this kind of information.  From a IT point of view, this enables the admin to update the base with the user being able to keep their changes, even it includes application installations.  At one point this seemed to be a very important part of the project.  Now it seems like a very futuristic and potentially improbable dream.  The potential for conflict is always there and it is probably the methods of handling the conflicts that are more important than the actual re-base.

I am working towards making the tools available on the Internet.  There is some hope that CDN can host items.  I would like to see Citrix Labs have tools listed so that the VHD tools could be posted there.

There are many more tools that have been written along the way.  Some of the more useful ones are strictly diagnostic.  For example, it is possible to dump the entire file directory tree from a VHD.

The point I want to clarify before going is that the tools are completely self enclosed.  In other words, vhdcompact does not use any NTFS mount code to determine the VHD content.  It directly reads the VHD metadata and determines where the NTFS information is.  It essentially knows everything about the VHD and only uses the operating system to open/reads/writes/close on the VHD file itself.

As a side benefit of how the code was created, it also supports doing the same things in Windows.  This was largely due to me being more comfortable building and debugging with Visual Studio.  The code is in C++ and is very generalized so it works with either platform.  The abstraction is good enough that the code will run exactly the same way for the same VHDs.

It has been a focus for me for more than a year now.  I am happy with things are turning out and soon the tools will find their way into the real world.  I just decided that it is worth writing a few more posts about what is going on with them in the next few weeks.

About Jeff Muir

Live near Brisbane, Australia. Software developer currently focused on iOS and Android. Avid Google Local Guide