Study Programs

Digging Deeper: Algorithms for Computationally-Limited Searches in Astronomy

Team Leads
George Djorgovski
Curt Cutler
Bruce Elmegreen

Astronomy, like most other fields, is being deluged by exponentially growing streams of ever more complex data. While these massive data streams bring a great discovery potential, their full scientific exploitation poses many challenges, due to both data volumes and data complexity. Moreover, the need to discover and characterize interesting, faint signals in such data streams quickly and robustly, in order to deploy costly follow-up resources that are often necessary for the full scientific returns, makes the challenges even sharper.

Examples in astronomy include transient events and variable sources found in digital synoptic sky surveys, gravitational wave signals, faint radio transients, pulsars, and other types of variable sources in the next generation of panoramic radio surveys, etc. Similar situations arise in the context of space science and planetary exploration, environmental monitoring, security, etc. In most cases, rapid discovery and characterization of interesting signals is highly computationally limited.

The goal of this study was to define a number of interesting, often mission-critical challenges of this nature in the broader context of time-domain astronomy, but with an eye on their applicability elsewhere. Three types of challenges were identified and followed through the duration of this study:

  • Searching for Long, Weak Gravitational Wave Chirps and for Microlensing Events
    The first part of this problem is of a critical importance for the nascent field of gravitational wave astronomy, but it is also highly relevant for the searches for heavily dispersed pulsar signals in radio data cubes, or in γ-rays. The second aspect of the problem is to find gravitational microlensing events with characteristic signatures of planets around the lensing star. We invented of a couple new techniques to increase search efficiency, and the effort continues, with another technique added since the study’s completion. The current set of methods for this analysis yet has to be optimally combined into a full data analysis pipeline, requiring manpower, and this remains a very worthy and a attainable goal for future work in the near-to-mid-term.
  • Intermittent, Sub-Significant Detections in Data Cubes
    In a series of images where the third axis represents time or different wavelengths, there may be sources that appear only intermittently, but that are not statistically significant in any one epoch or channel. If the right subset of these were to be averaged, the detection would be significant, but averaging all of them would dilute the signal. An easier version of the problem is if the position of a possible source is already defined; a more challenging application is to blind searches. A solution to this problem could increase the effective depth of multi-epoch sky surveys from both ground  or space. A novel, statistically based method was developed for this purposes, and implemented as a software package. It is now being scientifically validated on the data from actual sky surveys.
  • Rapid, Automated Classification of Variable and Transient Sources
    Scientific returns from synoptic sky surveys are now increasingly limited by the ability to follow up the most interesting sources and events. Given the time-critical nature of such events, their rapid characterization or classification is essential for an optimal deployment of limited follow-up resources. The problem is complicated by the sparsity and heterogeneity of the data, and the presence of rtifacts that may masquerade as transient signals. The process has to be complete (no good signals are missed) and with a low contamination by false alarms. Automated classification of light curves is also essential for the archival exploration of synoptic sky survey archives. We explored and developed a number of new statistical and Machine learning approaches, that are now being scientifically validated on the actual sky survey data streams. Work continues along all of these avenues that were started or substantially expanded during the KISS study.

For questions contact: George Djorgovski, Curt Cutler, Bruce Elmegreen or Michele Judd

George Djorgovski

Study Co-Lead George Djorgovski from Caltech.

Curt Cutler

Study Co-Lead Curt Cutler from JPL.

Bruce Elmegreen

Study Co-Lead Bruce Elmegreen from IBM.