Computer environments are often complex and require operators to work on multiple tasks at once. Diagnostic automation is purported to help by telling the operator when and where to look, but imperfect reliability and the question of experience with the system provide an open question to research. In this study, we asked 60 participants to interact with a multiple-task computer system representative of the demands of work to test whether experience with the automation would change the way participants allocated their attention between the tasks and efficiency in interacting with the system. We found that highly reliable automation provided not only an overall benefit but an improvement with experience. Less reliable automation provided an overall benefit but no additional gains with experience. The reliability of diagnostic automation in computer applications, then, should be designed with consideration for reliability; only at greater levels will experience with the system provide additional benefits.