Delay tolerant network (DTN) is an emerging research area that exploits user mobility for transporting information. While user mobility connects disconnected network components, it causes a large end to end communication latency. Towards reducing this latency, the habitual nature of human movements have been widely exploited for prediction-based routing protocols. We observe that while habitual mobility is useful in reducing the average communication latency, irregular deviation from habits can seriously affect worst-case performance. This paper motivates the need to address such deviations, characterizes their impact on latency, and addresses them through a protocol called Diverse Routing (DR). Evaluation of our protocol on a variety of real-life traces offers promising results. Experimental results reveal that DR provides efficient handling on worst-case latency, i.e., DR's delivery delay is hardly affected by those irregular deviations, while it only incurs a moderate communication overhead. Moreover, DR can be easily tuned to meet different requirements of delivery delay and communication overhead.