This paper explores the feasibility of automatically extracting passwords from a user's daily activity logs, such as her Facebook activity, phone activity etc. As an example, a smartphone might ask the user: "Today morning from whom did you receive an SMS ?" In this paper, we observe that infrequent activities (i.e., outliers) can be memorable and unpredictable. Building on this observation, we have developed an end to end system ActivPass and experimented with 70 users. With activity logs from Facebook, browsing history, call logs, and SMSs, the system achieves 95% success (authenticates legitimate users) and is compromised in 5:5% cases (authenticates impostors). While this level of security is obviously inadequate for serious authentication systems, certain practices such as password sharing can immediately be thwarted from the dynamic nature of passwords. With security improvements in the future, activity-based authentication could fill in for the inadequacies in today's password-based systems.