Virtual appliances (VAs) are ready-to-use virtual machine images that are configured for specific purposes. For example, a virtual machine image that contains all the software necessary to develop and host a JSP-based website is typically available as a "Java Web Starter" VA. Currently there are many VA repositories from which users can download VAs and instantiate them on Infrastructure-as-a-Service (IaaS) clouds, allowing them to quickly launch their services. This marketplace, however, lacks adequate mechanisms that allow users to a priori assess whether a specific VA is really configured with the software that it is expected to be configured with. This paper evaluates the integrity of software packages installed on real-world VAs, through the use of a software whitelist-based framework, and finds that indeed there is a lot of variance in the software integrity of packages across VAs. Analysis of 151 Amazon VAs using this framework shows that about 9% of real-world VAs have significant numbers of software packages that contain unknown files, making them potentially untrusted. Virus scanners flagged just half of the VAs in that 9% as malicious, demonstrating that virus scanning alone is not sufficient to help users select a trustable VA and that a priori software integrity assessment has a role to play.