How-To

Run Older or Incompatible Software on Windows 10 in a VM

By Andre Da Costa

Updated June 8, 2021

Old software doesn’t always install and work well in Windows 10, and sometimes the best option is to create a virtual machine to run it.

While many of us have embraced Windows 10 and its new era of universal apps and cloud computing, many users still depend on the legacy desktop platform, Microsoft built over the past 30 years. As a result, we have looked at various methods for running older applications not designed to work on Windows 10. Some of these solutions include compatibility mode and enabling the 16-bit subsystem, only available on 32-bit versions.

Microsoft Windows Features

Over the past 10 years, Microsoft has refined Windows to resist attacks that were notorious during the Windows XP era. Some of these improvements include User Account Control, Patch Guard, Device Driver Signing, Secure Boot, and Address Space Randomization Layout. Unfortunately, older applications tend not to be receptive to these changes. As a result, Windows 10 itself breaks a lot from the past, but there is more to this problem. Some applications have fallen by the wayside, or newer versions don’t meet the expectations of older releases. In some cases, developers either have gone out of business or refuse to support an older release.

Reason for a VM

For users who need to run that exotic application but need to invest in the future, there is a reasonable solution in virtualization. We have covered virtualization in the past to test out Windows 10 itself or even use it for compatibility on platforms such as the Mac. Virtualization too can help users committed to Windows 10 bring their legacy investments forward. There are many options available, but I have narrowed this down to Oracle VirtualBox, which includes cross-platform support and works on Windows 10 Home and Pro. Hyper-V is an available option for users running Windows 10 Pro or Enterprise editions. Hyper-V, while powerful, is quite involved. Oracle VirtualBox, on the other hand, is small, quick, and easy to set up.

Determining the version of Windows for Application Compatibility

Settling on VirtualBox is one thing, but what about the version of Windows you will use for your legacy application? Most applications users need to run were likely designed within the last 15 years for Windows XP. Since that is the version users will need, they will also need to find a way to procure it. If you like to keep around old copies of software like me for nostalgia purposes, you must select the ISO image manually to go. There might be a challenge finding an older release to run your application for those who don’t. Websites such as Amazon.com, eBay, or checking your local flea market or garage sale; might help you score that older version of Windows you need to run your application.

Download and Setup Oracle VirtualBox

Before you begin installing Windows, you need to obtain Oracle VirtualBox, which you can download for free from the official page. Several versions are available; make sure you download the right one for Windows, listed as VirtualBox 5.1.4 for Windows hosts x86/amd64.

After downloading, launch the setup file, then follow the on-screen instructions for installing VirtualBox.

Creating the Virtual Machine

Once you have an Oracle VirtualBox setup, you can create a new virtual machine suitable for your legacy application. For this article, I will be using Windows XP. Microsoft ended support in early 2014 for what some would consider being one of the most popular versions of Windows to date. Launch VirtualBox, then click the New button on the command bar.

Enter a name for the VM or choose from the Type: list box. If VirtualBox detects the operating system name, it will populate the list boxes with the appropriate type and version. Click Next.

Create Size of the Machine

Windows XP is a 15-year-old operating system, so the memory requirements don’t have to be extremely high, especially for a legacy application. To sufficiently run Windows XP or 2000 Professional, 256 MBs to 512 MBs some. Also, take into account the applications you might be running in the virtual machine. In this case, I would go with 1 GB to be safe. The older the version of Windows you are using; the requirements might need to be modified. If you are using Windows 95 or Windows 98 in a virtual machine, 480 to 512 MBs of RAM is the limit for those versions. If it’s an MS-DOS virtual machine, 64 MBs is the maximum since it won’t address anymore.

Next, we will create a virtual hard disk, which will store the operating system. Select Create a virtual hard disk now, then click Create.

There are several choices offered for the format you can use to store the hard disk file. If, in the future, you would like to migrate your virtual machine to another platform such as Hyper-V or VMWare, then you should choose one of the available options such as VMDK or VHD supported by those hypervisors, respectively. For now, I will go with the default, which is Oracle VirtualBoxes VDI.

If you are short on space, using Dynamically allocated is the best option since it uses space only when needed. However, if performance is a factor, then a Fixed-size is recommended. I like using Fixed size just for the performance benefits.

Fixed Size

The Fixed size selection will require choosing the amount of space you will be allocated to the virtual disk. This will be dependent on your needs and availability of space. Factors include the operating system size, type, and applications. Legacy systems don’t require a lot. In the case of Windows XP, if you are using more than 32 GBs, you will need to format it as NTFS. Also, operating systems such as Windows 95 and 98 will need to use FAT32 for disk larger than 512 MBs. Windows NT 4 can only recognize 8 GBs. So, take this into account. Since I will only be using a single app and Windows XP uses about 1 GB of space, I will be going with 3 GBs. Click Create to confirm the amount.

Wait while your Virtual Hard Disk is created.

Installing the Operating System

How you install the operating system will vary. Older versions of Windows such as Windows 95, Windows 3.1, and MS-DOS will require creating Virtual Floppy disks with the appropriate boot files. Since we are working with Windows XP, we will focus on that. If you are using a system without an optical drive, you will need to find a computer and create an ISO image of your install disc, which you can then mount in Oracle VirtualBox. To create ISO images, I use ISO Recorder by Alex Feinman, which is quite old but still works, even on Windows 10. Creating an image is as simple as inserting the disc, right-click the disc in Computer, then click Create image from CD/DVD. Once you have an ISO image ready, you can proceed with the installation.

In VirtualBox, double-click the Windows XP virtual machine to start it.

Because you are installing for the first time, you will need to select the ISO image manually. First, click the Browse button, select the Windows ISO image, then click Open. Finally, click Start to load the operating system.

Follow the on-screen instructions to install your operating system.

After installing Windows XP or an earlier version of Windows, you should install the Guest Additions to ensure you can move between the virtual machine and host easily. First, click Devices > Insert Guest Additions CD image, then click Next.

Installing Legacy Applications

Once you have your Windows installation fully configured, you can proceed to install your legacy applications. There are several ways to do this. Depending on the format the application is available in, you might need to convert it into a format acceptable by the VM. If your application is on the disc, you can use ISO Recorder, described earlier, to create a copy you can mount as an ISO image in the VM.

Mount ISO Image

To mount an image, click Devices > Choose disk image, select the ISO file, then click open.

Proceed to install the application. If the application setup does not start automatically, click Start > Computer, then double click the disc icon or open it, then double click the called setup.exe.

There you have it; your application is now up and running without issue.

What about sharing your files or opening them?

Oracle VirtualBox includes drag-and-drop support, so you can easily drag files between the host and guest or vice versa.

What about security?

Running an unsupported virtual machine is no different from running it on a physical host. Oracle VirtualBox automatically configures your virtual machine to access the Internet. You might want to perform several security modifications, such as disabling network access within the VM settings. You can also try installing a supported Antivirus utility if you need to have network access for whatever reason. My personal recommendation is to keep it off the network.

Summing Up

While setting up a supported version of Windows in a virtual machine might seem overkill, it is, in fact, the best way to ensure your application works. With all its merits, Compatibility Mode suffers from the Windows 10 service model, where an update or revision of the OS can easily break an app. There is some overhead with running a VM, but the requirements are quite low for that specific legacy application. If you need to get that app up and running, and you have tried all options so far, then this is the way to go. From now on, users of next-generation Intel silicon, such as the recently launched Intel Kabylake and AMD Bristol Ridge CPUs, will need to look to solutions such as virtualization to run applications supported on older releases like Windows 7 and Windows 8/8.1.