This article explains how to reliably enlarge a QEMU qcow or raw image that contains an NTFS or FAT32 bootable partition.
Like a lot of other people I needed to increase the size of a QCOW disk image to 10GB from the original maximum size of 5GB, I searched all over and could only find three web pages that described methods of doing it, each of which was imperfect and not totally reliable. There were tens of pages discussing how it is impossible.
My scenario is QEMU 0.9 running on Ubuntu 7.04 Feisty (kernel 2.6.20-16) with Windows XP in a 5GB qcow disk image.
After trying the various suggestions for how to resize the image I always got the dreaded:
A disk read error occurred
Press Ctrl+Alt+Del to restart
I read of a suggestion by Fabrice that it was to do with "CHS translation" but no one seemed to have taken it further.
If you just want the solution rather than reading the technical details skip to "How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition" further down.
Discovering why it fails
First I converted the qcow image to raw:
$ qemu-img convert -f qcow hda.qcow -O raw hda.raw
We are going to require super-user (root) privileges for most of the following operations so lets switch to super-user:
$ sudo su
Password:
root $
Using the loopback device in Linux I attached the raw image:
$ losetup /dev/loop1 hda.raw
Then I inspected the Master Boot Record (MBR) using fdisk:
$ fdisk -u /dev/loop1
Command (m for help): p
Disk /dev/loop1: 4194 MB, 4194816000 bytes
128 heads, 63 sectors/track, 1015 cylinders, total 8193000 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/loop1p1 * 63 8176895 4088416+ 7 HPFS/NTFS
Command (m for help): q
Notice that fdisk reports the disk has 128 (0x80) heads.
Now lets examine the NTFS partition's boot sector. To do this we need to detach the 'physical' disk:
$ losetup -d /dev/loop1
and attach the 'logical' disk contained in partition 1. This starts at sector offset 63 (0x3F) according to fdisk, which is 32256 bytes (63 x 512) into the image file:
$ losetup -o32256 /dev/loop1 hda.raw
Just to prove this is an NTFS partition we have attached lets check it:
$ ntfsresize -i /dev/loop1
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name : /dev/loop1
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 4194783744 bytes (4195 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!
Now let's examine the BIOS Parameter Block (BPB) of the partition's boot sector:
$ dd if=/dev/loop1 of=bootsector.bin bs=512 count=1
$ losetup -d /dev/loop1
I use hexdump to examine the data statically:
$ hexdump -C bootsector.bin
The BPB bytes we are interested in are the ones describing the geometry of the drive:
Offset Purpose
0x18 Sectors per track
0x1A Heads (tracks per cylinder)
So here is the fragment that shows these:
00000000 eb 52 90 4e 54 46 53 20 20 20 20 00 02 08 00 00 |.R.NTFS .....|
00000010 00 00 00 00 00 f8 00 00 3f 00 80 00 3f 00 00 00 |........?...?...|
You can see at offset 0x18 "3f 00" (63) and at 0x1A "80 00" (128).
When a disk image is resized by a considerable margin the number of heads reported for the entire disk image is likely to increase, but QEMU doesn't know the embedded Operating System (in this case Windows XP) has stored the disk geometry in its BPB.
As a result when the QEMU BIOS tries to load the OS it gets the correct disk geometry from the MBR but when the MBR passed execution to the boot sector code in the partition, the BPB causes the boot-code to get confused and the system fails to boot.
So the solution is to edit the single byte in the BPB that describes the number of heads (offset 0x1A) so it matches whatever fdisk reports.
How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition
Make sure you have plenty of disk space for the following operations. Adjust paths to point to alternative disks that have space if necessary.
Note: If you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image, read the next post in this thread for how you can enlarge the original file directly. It is more dangerous since it is changing your only copy of the original.
Convert the compressed qcow image to raw:
$ qemu-img convert -f qcow hda.qcow -O raw hda.raw
Most of the commands will require super-user privileges so switch now to save using sudo for every command:
$ sudo su
Create the new raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value after seek= is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB:
$ dd if=/dev/zero of=hdb.raw bs=512 count=0 seek=20971520
$ ls -l hdb.raw
-rw-r--r-- 1 root root 10737418240 2007-06-14 15:14 hdb.raw
Attach the new drive:
$ losetup /dev/loop0 hdb.raw
Use fdisk to create the new, larger, NTFS partition table entry:
$ fdisk -u /dev/loop0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help):
Create a new partition (press n p 1 <enter> <enter> ):
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-20971519, default 63):
Using default value 63
Last sector or +size or +sizeM or +sizeK (63-20971519, default 20971519):
Using default value 20971519
Set the partition type to NTFS (press t 7):
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)
Set the partition to be bootable aka active (press a 1):
Command (m for help): a
Partition number (1-4): 1
Display aka print the new configuration (press p):
Command (m for help): p
Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 63 20971519 10485728+ 7 HPFS/NTFS
Notice the number of heads is 255? This is the value we will have to patch into the NTFS BPB in the partition boot sector later.
Now write the changes to the disk (press w):
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
We now have a bootable disk image with a partition table, but no Operating System to boot.
Detach the disk image:
$ losetup -d /dev/loop0
Re-attach the image so the Linux kernel will read the new partition table, and then check it with fdisk:
$ losetup /dev/loop0 hdb.raw
$ fdisk -ul /dev/loop0
Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 63 20971519 10485728+ 7 HPFS/NTFS
Copy just the MBR boot-code into the new disk's MBR:
$ dd if=hda.raw of=/dev/loop0 bs=1 count=446
This is the code the BIOS boot-loader executes when the system boots from this disk device. It is also what a Windows/DOS fdisk /mbr installs.
Now copy the NTFS partition from the original image into the new image. Remember, the file-system partitions start at sector offset 63 according to fdisk for both images:
$ dd if=hda.raw of=/dev/loop0 bs=512 skip=63 seek=63
Detach the disk image:
$ losetup -d /dev/loop0
Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256:
$ losetup -o32256 /dev/loop0 hdb.raw
Confirm it is a valid NTFS partition:
$ ntfsresize -i /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!
Now it is time to edit the NTFS BPB. I use the tool hexedit which you may need to install:
$ apt-get install hexedit
Load the new disk image into hexedit:
$ hexedit hdb.raw
Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>):
00007E00 EB 52 90 4E 54 46 53 20 20 20 20 00 02 08 00 00 .R.NTFS .....
00007E10 00 00 00 00 00 F8 00 00 3F 00 80 00 3F 00 00 00 ........?...?...
Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here.
You can now over-type the hex-value 80 (128) with the correct number of heads, which in this case is FF (255):
00007E10 00 00 00 00 00 F8 00 00 3F 00 FF 00 3F 00 00 00 ........?...?...
Save the change by pressing Ctrl-X and then 'y' to confirm the write.
The image is now ready for QEMU.
$ qemu -boot c -snapshot -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
If that boots successfully we can now resize the NTFS file-system:
$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
New volume size : 10737381888 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 3263 MB (77.9%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.
If you get the error:
$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported
The NTFS journal file is unclean. Please shutdown Windows properly before
using this software! Note, if you have run chkdsk previously then boot
Windows again which will automatically initialize the journal correctly.
you should do a complete startup/shutdown sequence of Windows so the disk is marked clean:
$ losetup -d /dev/loop0
$ qemu -boot c -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
All that is left to do is detach it:
$ losetup -d /dev/loop0
and convert the raw image back to a compressed qcow image. I usually explicitly make it qcow2, the later version:
$ qemu-img convert -f raw hdb.raw -O qcow2 hdb.qcow2
Make sure the qcow image boots:
$ qemu -boot c -snapshot -m 512 -hda 'hdb.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
Delete the large raw images and the original qcow image:
$ rm hdb.raw hda.raw hda.qcow
The final step is to set the owner to your regular user, because the file was created as root, and exit super-user:
$ chown <username>:<username> hdb.qcow2
$ exit
I hope this helps - it took a while to perfect the process :D
How to enlarge without using twice the disk space
This method is an alternative to the one described in the previous post. It is intended for more confident users, where you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image. It is more dangerous since it is changing your only copy of the original.
Convert the compressed qcow image to raw:
$ qemu-img convert -f qcow hda.qcow -O raw hda.raw
Most of the commands will require super-user privileges so switch now to save using sudo for every command:
$ sudo su
Enlarge the existing raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value afterseek= is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB:
$ dd if=/dev/zero of=hda.raw bs=512 count=0 seek=20971520
$ ls -l hda.raw
-rw-r--r-- 1 root root 10737418240 2007-06-18 12:10 hda.raw
Attach the enlarged drive image:
$ losetup /dev/loop0 hda.raw
Use fdisk to check how many heads the disk is using:
$ fdisk -ul /dev/loop0
Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/loop1p1 * 63 8176895 4088416+ 7 HPFS/NTFS
Notice the number of heads is 255? This is the value we will have to patch into the NTFS BPB in the partition boot sector later.
Detach the drive image:
$ losetup -d /dev/loop0
Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256:
$ losetup -o32256 /dev/loop0 hda.raw
Confirm it is a valid NTFS partition:
$ ntfsresize -i /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!
Now it is time to edit the NTFS BPB. I use the tool hexedit which you may need to install:
$ apt-get install hexedit
Load the new disk image into hexedit:
$ hexedit hda.raw
Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>):
00007E00 EB 52 90 4E 54 46 53 20 20 20 20 00 02 08 00 00 .R.NTFS .....
00007E10 00 00 00 00 00 F8 00 00 3F 00 80 00 3F 00 00 00 ........?...?...
Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here.
You can now over-type the hex-value 80 (128) with the correct number of heads, which in this case is FF (255):
00007E10 00 00 00 00 00 F8 00 00 3F 00 FF 00 3F 00 00 00 ........?...?...
Save the change by pressing Ctrl-X and then 'y' to confirm the write.
Detach the drive image:
$ losetup -d /dev/loop0
The image is now ready for QEMU.
$ qemu -boot c -snapshot -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
If that boots successfully we can now resize the NTFS file-system:
Attach the file-system image:
$ losetup -o32256 /dev/loop0 hda.raw
Resize to fill the drive-image:
$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name : /dev/loop0
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
New volume size : 10737381888 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 3263 MB (77.9%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.
If you get the error:
$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported
The NTFS journal file is unclean. Please shutdown Windows properly before
using this software! Note, if you have run chkdsk previously then boot
Windows again which will automatically initialize the journal correctly.
you should do a complete startup/shutdown sequence of Windows so the disk is marked clean:
$ losetup -d /dev/loop0
$ qemu -boot c -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
All that is left to do is detach it:
$ losetup -d /dev/loop0
and convert the raw image back to a compressed qcow image. I usually explicitly make it qcow2, the later version:
$ qemu-img convert -f raw hda.raw -O qcow2 hda.qcow2
Make sure the qcow image boots:
$ qemu -boot c -snapshot -m 512 -hda 'hda.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu
Delete the large raw image and the original qcow image:
$ rm hda.raw hda.qcow
The final step is to set the owner to your regular user, because the file was created as root, and exit super-user:
$ chown <username>:<username> hda.qcow2
$ exit
