StorkCloud: Data transfer scheduling and optimization as a service

Tevfik Kosar, Engin Arslan, Brandon Ross, Bing Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Wide-area transfer of large data sets is still a big challenge despite the deployment of high-bandwidth networks with speeds reaching 100 Gbps. Most users fail to obtain even a fraction of theoretical speeds promised by these networks. Effective usage of the available network capacity has become increasingly important for wide-area data movement. We have developed a "data transfer scheduling and optimization system as a Cloud-hosted service", StorkCloud, which will mitigate the large-scale end-to-end data movement bottleneck by efficiently utilizing underlying networks and effectively scheduling and optimizing data transfers. In this paper, we present the initial design and prototype implementation of StorkCloud, and show its effectiveness in large dataset transfers across geographically distant storage sites, data centers, and collaborating institutions.

Original languageEnglish (US)
Title of host publicationScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing
PublisherAssociation for Computing Machinery
Pages29-36
Number of pages8
ISBN (Print)9781450319799
DOIs
StatePublished - Jan 1 2013
Event4th ACM Workshop on Scientific Cloud Computing, ScienceCloud 2013 - New York, NY, United States
Duration: Jun 17 2013Jun 17 2013

Publication series

NameScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing

Conference

Conference4th ACM Workshop on Scientific Cloud Computing, ScienceCloud 2013
CountryUnited States
CityNew York, NY
Period6/17/136/17/13

Fingerprint

Data transfer
Scheduling
Bandwidth

Keywords

  • big data
  • cloud computing
  • data scheduling.
  • end-to-end throughput optimization
  • software as a service (SAAS)

ASJC Scopus subject areas

  • Software

Cite this

Kosar, T., Arslan, E., Ross, B., & Zhang, B. (2013). StorkCloud: Data transfer scheduling and optimization as a service. In ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing (pp. 29-36). (ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing). Association for Computing Machinery. https://doi.org/10.1145/2465848.2465855

StorkCloud : Data transfer scheduling and optimization as a service. / Kosar, Tevfik; Arslan, Engin; Ross, Brandon; Zhang, Bing.

ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing. Association for Computing Machinery, 2013. p. 29-36 (ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kosar, T, Arslan, E, Ross, B & Zhang, B 2013, StorkCloud: Data transfer scheduling and optimization as a service. in ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing. ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing, Association for Computing Machinery, pp. 29-36, 4th ACM Workshop on Scientific Cloud Computing, ScienceCloud 2013, New York, NY, United States, 6/17/13. https://doi.org/10.1145/2465848.2465855
Kosar T, Arslan E, Ross B, Zhang B. StorkCloud: Data transfer scheduling and optimization as a service. In ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing. Association for Computing Machinery. 2013. p. 29-36. (ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing). https://doi.org/10.1145/2465848.2465855
Kosar, Tevfik ; Arslan, Engin ; Ross, Brandon ; Zhang, Bing. / StorkCloud : Data transfer scheduling and optimization as a service. ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing. Association for Computing Machinery, 2013. pp. 29-36 (ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing).
@inproceedings{d22f4a27d34e4de5b3aa41bc473a399c,
title = "StorkCloud: Data transfer scheduling and optimization as a service",
abstract = "Wide-area transfer of large data sets is still a big challenge despite the deployment of high-bandwidth networks with speeds reaching 100 Gbps. Most users fail to obtain even a fraction of theoretical speeds promised by these networks. Effective usage of the available network capacity has become increasingly important for wide-area data movement. We have developed a {"}data transfer scheduling and optimization system as a Cloud-hosted service{"}, StorkCloud, which will mitigate the large-scale end-to-end data movement bottleneck by efficiently utilizing underlying networks and effectively scheduling and optimizing data transfers. In this paper, we present the initial design and prototype implementation of StorkCloud, and show its effectiveness in large dataset transfers across geographically distant storage sites, data centers, and collaborating institutions.",
keywords = "big data, cloud computing, data scheduling., end-to-end throughput optimization, software as a service (SAAS)",
author = "Tevfik Kosar and Engin Arslan and Brandon Ross and Bing Zhang",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2465848.2465855",
language = "English (US)",
isbn = "9781450319799",
series = "ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing",
publisher = "Association for Computing Machinery",
pages = "29--36",
booktitle = "ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing",

}

TY - GEN

T1 - StorkCloud

T2 - Data transfer scheduling and optimization as a service

AU - Kosar, Tevfik

AU - Arslan, Engin

AU - Ross, Brandon

AU - Zhang, Bing

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Wide-area transfer of large data sets is still a big challenge despite the deployment of high-bandwidth networks with speeds reaching 100 Gbps. Most users fail to obtain even a fraction of theoretical speeds promised by these networks. Effective usage of the available network capacity has become increasingly important for wide-area data movement. We have developed a "data transfer scheduling and optimization system as a Cloud-hosted service", StorkCloud, which will mitigate the large-scale end-to-end data movement bottleneck by efficiently utilizing underlying networks and effectively scheduling and optimizing data transfers. In this paper, we present the initial design and prototype implementation of StorkCloud, and show its effectiveness in large dataset transfers across geographically distant storage sites, data centers, and collaborating institutions.

AB - Wide-area transfer of large data sets is still a big challenge despite the deployment of high-bandwidth networks with speeds reaching 100 Gbps. Most users fail to obtain even a fraction of theoretical speeds promised by these networks. Effective usage of the available network capacity has become increasingly important for wide-area data movement. We have developed a "data transfer scheduling and optimization system as a Cloud-hosted service", StorkCloud, which will mitigate the large-scale end-to-end data movement bottleneck by efficiently utilizing underlying networks and effectively scheduling and optimizing data transfers. In this paper, we present the initial design and prototype implementation of StorkCloud, and show its effectiveness in large dataset transfers across geographically distant storage sites, data centers, and collaborating institutions.

KW - big data

KW - cloud computing

KW - data scheduling.

KW - end-to-end throughput optimization

KW - software as a service (SAAS)

UR - http://www.scopus.com/inward/record.url?scp=84880082389&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84880082389&partnerID=8YFLogxK

U2 - 10.1145/2465848.2465855

DO - 10.1145/2465848.2465855

M3 - Conference contribution

AN - SCOPUS:84880082389

SN - 9781450319799

T3 - ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing

SP - 29

EP - 36

BT - ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing

PB - Association for Computing Machinery

ER -