Recently, sparse representation based classification (SRC) has been successfully used for visual recognition and showed impressive performance. Given a testing sample, SRC computes its sparse linear representation with respect to all the training samples and calculates the residual error for each class of training samples. However, SRC considers the training samples in each class contributing equally to the dictionary in that class, i.e., the dictionary consists of the training samples in that class. This may lead to high residual error and instability. In this paper, a class specific dictionary learning algorithm is proposed. First, by introducing the dual form of dictionary learning, an explicit relationship between the bases vectors and the original image features is represented, which enhances the interpretability. SRC can be thus considered to be a special case of the proposed algorithm. Second, blockwise coordinate descent algorithm and Lagrange multipliers are then applied to optimize the corresponding objective function. Extensive experimental results on three benchmark face recognition datasets demonstrate that the proposed algorithm has achieved superior performance compared with conventional classification algorithms.