This paper introduces retargeted matrix factorization (R-MF); a novel approach for learning the user-wise ranking of items in the context of collaborative filtering. R-MF learns to rank by "retargeting" the item ratings of each user, searching for a monotonic transformation of the ratings that results in a better fit while preserving the ranked order of each user's ratings. The retargeting is combined with an underlying matrix factorization regression model that couples the user-wise rankings to exploit shared low dimensional structure. We show that R-MF recovers a unique solution under mild conditions, and propose a simple and efficient optimization scheme that alternates between retargeting the ratings subject to ordering constraints, and matrix factorization regression. The retargeting step is independent for each user, and is trivially parallelized. The ranking performance of retargeted matrix factorization is evaluated on benchmark movie recommendation datasets and results in superior ranking performance compared to collaborative filtering algorithms specifically designed to optimize ranking metrics.