pipeline

The term pipeline is used in technology, including research astronomy for a sequence of processing steps carried out on data, especially if data is passed through from step to step such that when data is passed from step 1 to step 2, additional data begins step 1, so data is being processed simultaneously (with analogous simultaneous activity as data reaches each further step). Thus a stream of data is being processed, at any instant, some by each step. The term may be used to indicate and explain this simultaneous-yet-sequential processing, but is also used sometimes without the literal intent, but merely to explain the order of the processing of the data.

Literal pipelines (i.e., when the term is more than an explaining notion) are a candidate-solution to certain kinds of challenges in processing the data: the need to finish the processing for some data quickly and/or to tackle a huge amount of data and/or to organize some complicated processing. A prime circumstance is an observation providing too much data to save, with considerable processing needed to reduce it to the point that it can be saved. This occurs in surveys of transient phenomena, often done with automated telescopes. They create something like a high-resolution digitized time-lapse movie of the sky, possibly even a data cube per frame, but the information of interest is undiscovered astronomical transients such as supernovae or near-Earth objects, rather than things like Earth weather or passing airplanes. The challenge in such a case is a means to classify the observed change promptly and open the possibility for prompt follow-up observation, and last part of the pipeline might consist of sending alerts to appropriate astronomers. Pipelines are ubiquitous for such surveys and the design, deployment, and operation of an effective pipeline can require a significant part of the resources dedicated to the survey. The concept of a pipeline is also a good description of typical radio astronomy observation/processing that incorporates radio interferometry.

Pipelines can also be an efficient way to handle huge sets of data to be processed even if there is not such an urgent deadline. Some computing can be looked upon as a pipeline or not, according to the manner in which the researcher decides to describe the process.

Steps of such a pipeline may each be a computer program, that could all running on a single computer, or could be spread out across interconnected computers. The pipeline may begin with analog electronics. Data communication may be a part of it, e.g., for transferring the data from computers near the telescope to those at some distant location more suitable for computer processing. Steps may also involve the actions of humans, they may be involved along the way, such as to verify or filter, and carry out other steps by hand though often the goal is to automate, and the "pipeline-like description" may describe to some degree actions of astronomers notified by an alert.

The term pipeline is also used in science and technology for other circumstances where something (or someone) is passed from step to step, such as the education of astronomers/astrophysicists (the education pipeline) or the development (designs, approvals, and funding decisions) of projects (a project pipeline).