Signal Recovery with Multistage Tests and Without Sparsity Constraints

A signal recovery problem is considered, where the same binary testing problem is posed over multiple, independent data streams. The goal is to identify all signals (resp. noises), i.e., streams where the alternative (resp. null) hypothesis is correct, subject to prescribed bounds on classical or generalized familywise error probabilities of both types. It is not required that the exact number of signals be a priori known, only upper bounds on the numbers of signals and noises are assumed instead. A decentralized formulation is adopted, according to which the sample size and the decision for each testing problem must be based only on observations from the corresponding data stream. A novel multistage testing procedure is proposed for this problem and is shown to enjoy a high-dimensional asymptotic optimality property. Specifically, it achieves the optimal, average over all streams, expected sample size, uniformly in the true number of signals, as the maximum possible numbers of signals and noises go to infinity at arbitrary rates, in the class of all sequential tests with the same global error control. In contrast, existing multistage tests in the literature are shown to achieve this high-dimensional asymptotic optimality property only under additional sparsity or symmetry conditions. These results are based on an asymptotic analysis for the fundamental binary testing problem as the two error probabilities go to zero. Moreover, they are supported by simulation studies and extended to problems with non-iid data and composite hypotheses.