Budget-constrained bulk data transfer via internet and shipping networks

Cloud collaborators wish to combine large amounts of data, in the order of TBs, from multiple distributed locations to a single datacenter. Such groups are faced with the challenge of reducing the latency of the transfer, without incurring excessive dollar costs. Our Pandora system is an autonomic system that creates data transfer plans that can satisfy latency and cost needs, by considering transferring the data through both Internet and disk shipments. Solving the planning problem is a critical step towards a truly autonomic bulk data transfer service. In this paper, we develop techniques to create an optimal transfer plan that minimizes transfer latency subject to a budget constraint. To systematically explore the solution space, we develop efficient binary search methods that find the optimal shipment transfer plan. Our experimental evaluation, driven by Internet bandwidth traces and actual shipment costs queried from FedEx web services, shows that these techniques work well on diverse, realistic networks.