Language models for image captioning: The quirks and what works

Two recent approaches have achieved state-of-the-art results in image caption-ing. The first uses a pipelined process where a set of candidate words is gen-erated by a convolutional neural network (CNN) trained on images, and then a max-imum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this pa-per, we compare the merits of these dif-ferent language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine is-sues in the different approaches, includ-ing linguistic irregularities, caption repe-tition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the bench-mark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.