Pervasive computing environments with their interconnected devices and services promise seamless integration of digital infrastructure into our everyday lives. While the focus of current research is on how to connect new devices and build useful applications to improve functionality, the security and privacy issues in such environments have not been explored in any depth. While traditional distributed computing research attempts to abstract away physical location of users and resources, pervasive computing applications often exploit physical location and other context information about users and resources to enhance the user experience. The need to share resources and collaborate introduces new types of interaction among users as well as between the virtual and physical worlds. In this context, it becomes difficult to separate physical security from digital security. Existing policies and mechanisms may not provide adequate guarantees to deal with new exposures and vulnerabilities introduced by the pervasive computing paradigm. In this paper we explore the challenges for building security and privacy into pervasive computing environments, describe our prototype implementation that addresses some of these issues, and propose some directions for future work.