Computer-based decision support systems are increasingly used to aid human decision makers in dynamic, uncertain, time-stressed and high-stakes contexts. The decision of whether, and if so, when to evacuate New Orleans as Hurricane Katrina approached landfall is a prime example. An evaluation of the "HURREVAC" decision support system (DSS) used during Katrina is presented. The evaluation is based on real-time screen-shots of the graphical and numerical information displayed to emergency response managers and other users. While the system is clearly an improvement over methods used prior to advances in information technology and realtime networking, design deficiencies were identified as well. The most crucial of these concern insufficient resources provided by the design to support users in reasoning effectively about uncertainty, and about the interactions among uncertainty and other aspects of the decision situation. The paper concludes by providing lessons learned and by identifying needs for cognitive engineering research to improve future DSS design in operational contexts.