@article{3aa7091cb1594722aa5e6e1c330382ad,
title = "Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams",
abstract = "We present Lightcurve Anomaly Identification and Similarity Search (LAISS), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1-5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes (spectroscopic anomalies), of uncommon host galaxy environments (contextual anomalies), and of peculiar or interaction-powered phenomena (behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.",
author = "Aleo, {P. D.} and Engel, {A. W.} and G. Narayan and Angus, {C. R.} and K. Malanchev and K. Auchettl and Baldassare, {V. F.} and A. Berres and {de Boer}, {T. J.L.} and Boyd, {B. M.} and Chambers, {K. C.} and Davis, {K. W.} and N. Esquivel and D. Farias and Foley, {R. J.} and A. Gagliano and C. Gall and H. Gao and S. Gomez and M. Grayling and Jones, {D. O.} and Lin, {C. C.} and Magnier, {E. A.} and Mandel, {K. S.} and T. Matheson and Raimundo, {S. I.} and Shah, {V. G.} and Soraisam, {M. D.} and {de Soto}, {K. M.} and S. Vicencio and Villar, {V. A.} and Wainscoat, {R. J.}",
note = "Parts of this work are based on observations obtained with the Samuel Oschin Telescope 48 inch and the 60 inch Telescope at the Palomar Observatory as part of the Zwicky Transient Facility project. ZTF is supported by the National Science Foundation under grant Nos. AST\u20131440341 and AST\u20132034437 and a collaboration including current partners Caltech, IPAC, the Weizmann Institute of Science, the Oskar Klein Center at Stockholm University, the University of Maryland, Deutsches Elektronen-Synchrotron and Humboldt University, the TANGO Consortium of Taiwan, the University of Wisconsin at Milwaukee, Trinity College Dublin, Lawrence Livermore National Laboratories, IN2P3, University of Warwick, Ruhr University Bochum, Northwestern University and former partners the University of Washington, Los Alamos National Laboratories, and Lawrence Berkeley National Laboratories. Operations are conducted by COO, IPAC, and UW. The ZTF forced-photometry service was funded under the Heising-Simons Foundation grant No. 12540303 (PI: Graham). Pan-STARRS is a project of the Institute for Astronomy of the University of Hawaii, and is supported by the NASA SSO Near Earth Observation Program under grants 80NSSC18K0971, NNX14AM74G, NNX12AR65G, NNX13AQ47G, NNX08AR22G, 80NSSC21K1572 and by the State of Hawaii. The PS1 Surveys and the PS1 public science archive have been made possible through contributions by the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, the Queen\u2019s University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, STScI, NASA under grant NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, NSF grant AST-1238877, the University of Maryland, Eotvos Lorand University (ELTE), the Los Alamos National Laboratory, and the Gordon and Betty Moore Foundation. The Young Supernova Experiment (YSE) and its research infrastructure is supported by the European Research Council under the European Union\u2019s Horizon 2020 research and innovation program (ERC grant agreement 101002652, PI K. Mandel), the Heising-Simons Foundation (2018-0913, PI R. Foley; 2018-0911, PI R. Margutti), NASA (NNG17PX03C, PI R. Foley), NSF (AST\u20131720756, AST\u20131815935, AST\u20132307710, PI R. Foley; AST-1909796, AST-1944985, PI R. Margutti), the David & Lucille Packard Foundation (PI R. Foley), VILLUM FONDEN (project 16599, PI J. Hjorth), and the Center for AstroPhysical Surveys (CAPS) at the National Center for Supercomputing Applications (NCSA) and the University of Illinois Urbana-Champaign. P.D.A.'s and G.N.'s contributions to this work were directly supported by NSF AST\u20132206195, and we gratefully acknowledge this funding. P.D.A. has also been supported by the Illinois Survey Science Graduate Fellowship from the Center for AstroPhysical Surveys (CAPS; https://caps.ncsa.illinois.edu/ ) at the National Center for Supercomputing Applications (NCSA). A.W.E. was partially supported by the Open Call Initiative, under the Laboratory Directed Research and Development (LDRD) Program at Pacific Northwest National Laboratory (PNNL). PNNL is a multiprogram national laboratory operated for the U.S. Department of Energy (DOE) by Battelle Memorial Institute under contract No. DE-AC05-76RLO 1830. G.N. is also supported by NSF CAREER grant AST\u20132239364, funded in-part by a grant by Charles Simonyi, and NSF OAC\u20132311355, DOE support through the Department of Physics at the University of Illinois, Urbana-Champaign (No. 13771275), and support from the HST Guest Observer Program through HST-GO-16764 and HST-GO-17128 (PI: R. Foley). Support was provided by Schmidt Sciences, LLC. for K.M. V.F.B.'s contributions to this work were supported by NSF AST-2206165. B.M.B. is supported by the Cambridge Centre for Doctoral Training in Data-Intensive Science funded by the UK Science and Technology Facilities Council (STFC). The UCSC team is supported in part by NASA grant NNG17PX03C, NSF grants AST\u20131815935 and AST\u20132307710, the Gordon & Betty Moore Foundation, the Heising-Simons Foundation, and a fellowship from the David and Lucile Packard Foundation to R.J.F. A.T.G. is supported by the National Science Foundation under cooperative agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/ ). C.G. is supported by a VILLUM FONDEN Young Investigator Grant (project No. 25501). M.G. and K.S.M. are supported by the European Union's Horizon 2020 research and innovation program under ERC grant agreement No. 101002652 and Marie Sk\u0142odowska-Curie grant agreement No. 873089. K.M.d.S. acknowledges support by the NSF through grant AST\u20132108676. K.d.S. thanks the LSST-DA Data Science Fellowship Program, which is funded by LSST-DA, the Brinson Foundation, and the Moore Foundation; her participation in the program has benefited this work. V.S. acknowledges the support of the LSST Corporation\u2019s 2021 Enabling Science award for undergraduates. V.A.V. acknowledges support by the NSF through grant AST\u20132108676. YSE-PZ was developed by the UC Santa Cruz Transients Team. The UCSC team is supported in part by NASA grants NNG17PX03C, 80NSSC18K0303, 80NSSC19K0113, 80NSSC19K1386, 80NSSC20K0953, 80NSSC21K2076, 80NSSC22K1513, 80NSSC22K1518, and 80NSSC23K0301; NSF grants AST\u20131720756, AST\u20131815935, and AST\u20131911206; grants associated with Hubble Space Telescope programs DD\u201314925, DD\u201315600, GO\u201315876, GO\u201316238, SNAP\u201316239, GO\u201316690, SNAP\u201316691, and GO\u201317128; the Gordon & Betty Moore Foundation; the Heising-Simons Foundation; fellowships from the Alfred P. Sloan Foundation and the David and Lucile Packard Foundation to R.J.F.; Gordon and Betty Moore Foundation postdoctoral fellowships and a NASA Einstein fellowship, as administered through the NASA Hubble Fellowship program and grant HST-HF2-51462.001, to D.O.J.; and a National Science Foundation Graduate Research Fellowship, administered through grant No. DGE-1339067, to D.A.C. The ANTARES project has been supported by the National Science Foundation through a cooperative agreement with the Association of Universities for Research in Astronomy (AURA) for the operation of NOIRLab, through an NSF INSPIRE grant to the University of Arizona (CISE AST-1344024, PI: R. Snodgrass), and through a grant from the Heising-Simons Foundation. Parts of this research were supported by the Australian Research Council Discovery Early Career Researcher Award (DECRA) through project No. DE230101069.",
year = "2024",
month = oct,
day = "1",
doi = "10.3847/1538-4357/ad6869",
language = "English (US)",
volume = "974",
journal = "Astrophysical Journal",
issn = "0004-637X",
publisher = "American Astronomical Society",
number = "2",
}