TY - GEN
T1 - VerdictDB
T2 - 44th ACM SIGMOD International Conference on Management of Data, SIGMOD 2018
AU - Park, Yongjoo
AU - Mozafari, Barzan
AU - Sorenson, Joseph
AU - Wang, Junhao
N1 - Publisher Copyright:
© 2018 Association for Computing Machinery.
Copyright:
Copyright 2018 Elsevier B.V., All rights reserved.
PY - 2018/5/27
Y1 - 2018/5/27
N2 - Despite 25 years of research in academia, approximate query processing (AQP) has had little industrial adoption. One of the major causes of this slow adoption is the reluctance of traditional vendors to make radical changes to their legacy codebases, and the preoccupation of newer vendors (e.g., SQL-on-Hadoop products) with implementing standard features. Additionally, the few AQP engines that are available are each tied to a specific platform and require users to completely abandon their existing databases-an unrealistic expectation given the infancy of the AQP technology. Therefore, we argue that a universal solution is needed: a databaseagnostic approximation engine that will widen the reach of this emerging technology across various platforms. Our proposal, called VerdictDB, uses a middleware architecture that requires no changes to the backend database, and thus, can work with all off-the-shelf engines. Operating at the driver-level, VerdictDB intercepts analytical queries issued to the database and rewrites them into another query that, if executed by any standard relational engine, will yield sufficient information for computing an approximate answer. VerdictDB uses the returned result set to compute an approximate answer and error estimates, which are then passed on to the user or application. However, lack of access to the query execution layer introduces significant challenges in terms of generality, correctness, and efficiency. This paper shows how VerdictDB overcomes these challenges and delivers up to 171× speedup (18.45× on average) for a variety of existing engines, such as Impala, Spark SQL, and Amazon Redshift, while incurring less than 2.6% relative error. VerdictDB is open-sourced under Apache License.
AB - Despite 25 years of research in academia, approximate query processing (AQP) has had little industrial adoption. One of the major causes of this slow adoption is the reluctance of traditional vendors to make radical changes to their legacy codebases, and the preoccupation of newer vendors (e.g., SQL-on-Hadoop products) with implementing standard features. Additionally, the few AQP engines that are available are each tied to a specific platform and require users to completely abandon their existing databases-an unrealistic expectation given the infancy of the AQP technology. Therefore, we argue that a universal solution is needed: a databaseagnostic approximation engine that will widen the reach of this emerging technology across various platforms. Our proposal, called VerdictDB, uses a middleware architecture that requires no changes to the backend database, and thus, can work with all off-the-shelf engines. Operating at the driver-level, VerdictDB intercepts analytical queries issued to the database and rewrites them into another query that, if executed by any standard relational engine, will yield sufficient information for computing an approximate answer. VerdictDB uses the returned result set to compute an approximate answer and error estimates, which are then passed on to the user or application. However, lack of access to the query execution layer introduces significant challenges in terms of generality, correctness, and efficiency. This paper shows how VerdictDB overcomes these challenges and delivers up to 171× speedup (18.45× on average) for a variety of existing engines, such as Impala, Spark SQL, and Amazon Redshift, while incurring less than 2.6% relative error. VerdictDB is open-sourced under Apache License.
UR - http://www.scopus.com/inward/record.url?scp=85048759343&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048759343&partnerID=8YFLogxK
U2 - 10.1145/3183713.3196905
DO - 10.1145/3183713.3196905
M3 - Conference contribution
AN - SCOPUS:85048759343
T3 - Proceedings of the ACM SIGMOD International Conference on Management of Data
SP - 1461
EP - 1476
BT - SIGMOD 2018 - Proceedings of the 2018 International Conference on Management of Data
A2 - Das, Gautam
A2 - Jermaine, Christopher
A2 - Eldawy, Ahmed
A2 - Bernstein, Philip
PB - Association for Computing Machinery
Y2 - 10 June 2018 through 15 June 2018
ER -