When an observer is searching through the environment for a target, what are the consequences of not finding a target in a given environment? We examine this issue in detail and propose that the visual system systematically tags environmental information during a search, in an effort to improve performance in future search events. Information that led to search successes is positively tagged, so as to favor future deployments of attention toward that type of information, whereas information that led to search failures is negatively tagged, so as to discourage future deployments of attention toward such failed information. To study this, we use an oddball-search task, where participants search for one item that differs from all others along one feature or belongs to a different visual category, from the other stimuli in the display. We find that when participants perform oddball-search tasks, the absence of a target delays identification of future targets containing the feature or category that was shared by all distractors in the target-absent trial. We interpret this effect as reflecting an implicit assessment of performance: target-absent trials can be viewed as processing "failures" insofar as they do not provide the visual system with the information needed to complete the task. Here, we study the goal-oriented nature of this bias in three ways. First, we show that the direction of the bias is determined by the experimental task. Second, we show that the effect is independent of the mode of presentation of stimuli: it happens with both serial and simultaneous stimuli presentation. Third, we show that, when using categorically defined oddballs as the search stimuli (find the face among houses or vice versa), the bias generalizes to unseen members of the "failed" category. Together, these findings support the idea that this inter-trial attentional biases arise from high-level, task-constrained, implicit assessments of performance, involving categorical associations between classes of stimuli and behavioral outcomes (success/failure), which are independent of attentional modality (temporal vs. spatial attention).