Budget-constrained bulk data transfer via internet and shipping networks

Brian Cho, Indranil Gupta

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Cloud collaborators wish to combine large amounts of data, in the order of TBs, from multiple distributed locations to a single datacenter. Such groups are faced with the challenge of reducing the latency of the transfer, without incurring excessive dollar costs. Our Pandora system is an autonomic system that creates data transfer plans that can satisfy latency and cost needs, by considering transferring the data through both Internet and disk shipments. Solving the planning problem is a critical step towards a truly autonomic bulk data transfer service. In this paper, we develop techniques to create an optimal transfer plan that minimizes transfer latency subject to a budget constraint. To systematically explore the solution space, we develop efficient binary search methods that find the optimal shipment transfer plan. Our experimental evaluation, driven by Internet bandwidth traces and actual shipment costs queried from FedEx web services, shows that these techniques work well on diverse, realistic networks.

Original languageEnglish (US)
Title of host publicationProceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops
Pages71-80
Number of pages10
DOIs
StatePublished - Jul 15 2011
Event8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops - Karlsruhe, Germany
Duration: Jun 14 2011Jun 18 2011

Publication series

NameProceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops

Other

Other8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops
CountryGermany
CityKarlsruhe
Period6/14/116/18/11

Fingerprint

Data Transfer
Data transfer
Freight transportation
Internet
Latency
Costs
Binary search
Budget Constraint
Web services
Search Methods
Experimental Evaluation
Web Services
Bandwidth
Planning
Trace
Minimise

Keywords

  • cloud computing
  • data-intensive computing
  • wide-area data transfer

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Applied Mathematics

Cite this

Cho, B., & Gupta, I. (2011). Budget-constrained bulk data transfer via internet and shipping networks. In Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops (pp. 71-80). (Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops). https://doi.org/10.1145/1998582.1998595

Budget-constrained bulk data transfer via internet and shipping networks. / Cho, Brian; Gupta, Indranil.

Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops. 2011. p. 71-80 (Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cho, B & Gupta, I 2011, Budget-constrained bulk data transfer via internet and shipping networks. in Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops. Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops, pp. 71-80, 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops, Karlsruhe, Germany, 6/14/11. https://doi.org/10.1145/1998582.1998595
Cho B, Gupta I. Budget-constrained bulk data transfer via internet and shipping networks. In Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops. 2011. p. 71-80. (Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops). https://doi.org/10.1145/1998582.1998595
Cho, Brian ; Gupta, Indranil. / Budget-constrained bulk data transfer via internet and shipping networks. Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops. 2011. pp. 71-80 (Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops).
@inproceedings{163a0aa3125e49789df2c1ce2b45e425,
title = "Budget-constrained bulk data transfer via internet and shipping networks",
abstract = "Cloud collaborators wish to combine large amounts of data, in the order of TBs, from multiple distributed locations to a single datacenter. Such groups are faced with the challenge of reducing the latency of the transfer, without incurring excessive dollar costs. Our Pandora system is an autonomic system that creates data transfer plans that can satisfy latency and cost needs, by considering transferring the data through both Internet and disk shipments. Solving the planning problem is a critical step towards a truly autonomic bulk data transfer service. In this paper, we develop techniques to create an optimal transfer plan that minimizes transfer latency subject to a budget constraint. To systematically explore the solution space, we develop efficient binary search methods that find the optimal shipment transfer plan. Our experimental evaluation, driven by Internet bandwidth traces and actual shipment costs queried from FedEx web services, shows that these techniques work well on diverse, realistic networks.",
keywords = "cloud computing, data-intensive computing, wide-area data transfer",
author = "Brian Cho and Indranil Gupta",
year = "2011",
month = "7",
day = "15",
doi = "10.1145/1998582.1998595",
language = "English (US)",
isbn = "9781450306072",
series = "Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops",
pages = "71--80",
booktitle = "Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops",

}

TY - GEN

T1 - Budget-constrained bulk data transfer via internet and shipping networks

AU - Cho, Brian

AU - Gupta, Indranil

PY - 2011/7/15

Y1 - 2011/7/15

N2 - Cloud collaborators wish to combine large amounts of data, in the order of TBs, from multiple distributed locations to a single datacenter. Such groups are faced with the challenge of reducing the latency of the transfer, without incurring excessive dollar costs. Our Pandora system is an autonomic system that creates data transfer plans that can satisfy latency and cost needs, by considering transferring the data through both Internet and disk shipments. Solving the planning problem is a critical step towards a truly autonomic bulk data transfer service. In this paper, we develop techniques to create an optimal transfer plan that minimizes transfer latency subject to a budget constraint. To systematically explore the solution space, we develop efficient binary search methods that find the optimal shipment transfer plan. Our experimental evaluation, driven by Internet bandwidth traces and actual shipment costs queried from FedEx web services, shows that these techniques work well on diverse, realistic networks.

AB - Cloud collaborators wish to combine large amounts of data, in the order of TBs, from multiple distributed locations to a single datacenter. Such groups are faced with the challenge of reducing the latency of the transfer, without incurring excessive dollar costs. Our Pandora system is an autonomic system that creates data transfer plans that can satisfy latency and cost needs, by considering transferring the data through both Internet and disk shipments. Solving the planning problem is a critical step towards a truly autonomic bulk data transfer service. In this paper, we develop techniques to create an optimal transfer plan that minimizes transfer latency subject to a budget constraint. To systematically explore the solution space, we develop efficient binary search methods that find the optimal shipment transfer plan. Our experimental evaluation, driven by Internet bandwidth traces and actual shipment costs queried from FedEx web services, shows that these techniques work well on diverse, realistic networks.

KW - cloud computing

KW - data-intensive computing

KW - wide-area data transfer

UR - http://www.scopus.com/inward/record.url?scp=79960152104&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960152104&partnerID=8YFLogxK

U2 - 10.1145/1998582.1998595

DO - 10.1145/1998582.1998595

M3 - Conference contribution

AN - SCOPUS:79960152104

SN - 9781450306072

T3 - Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops

SP - 71

EP - 80

BT - Proceedings of the 8th ACM International Conference on Autonomic Computing, ICAC 2011 and Co-located Workshops

ER -