For a long time, much of the scientific community has played the role of a recipient of standard packages of scientific data that have undergone specific and well defined calibration and data quality control. Access, reprocessing and recalibration of raw data, and sharing the results with collaborators have always been challenging and cumbersome. The WISE technology, developed by the Target partner OmegaCEN opens new possibilities for integrated, efficient and scalable data processing, storage and analysis.
A part of the larger astronomical IT community, OmegaCEN was faced with the overwhelming rates of data generated by modern astronomical facilities. The problem of efficient search through these data and subsequent processing and analysis were a real challenge. The established procedure involved the pushing of raw data through software pipelines with well-defined methods for calibration and quality control in order to deliver standard complete sets of data to the larger scientific community. This classical approach was not scalable enough to handle current data streams, and re-processing of raw data when new computational techniques became available was difficult and largely inefficient. The need to design novel software systems for data management of massive volumes of data gave rise to the WISE technology.
The WISE technology glues together databasing, storage, processing and visualization into a single integrated intelligent information system. Stored within this system is the entire "lifetime history" of every data item. This allows tracing of data products irrespective of when and how they were processed, all the way back to the original data set. This powerful tool, which we call 'backward chaining' is the backbone of the WISE technology. Consequently, instead of passively receiving standard processed bundles of data, a user can actively trigger the system by sending a specific query, or in WISE terminology - a request for a target. Driven by the users’ targets, the system can effectively pull out the relevant data in a quick, dynamic and flexible manner. Furthermore, the WISE technology creates a “living archive” and provides an environment, where both the results of the processing as well as the methods are fed back into the information system. As a result, it becomes a powerful tool for knowledge creation and sharing.
WISE technology is infrastructure-independent and flexible in interfacing with external systems. It also enables scalability of large-scale complex information systems. The novel approach to data handling, based on user-specified targets, by means of queries, provides the opportunity for efficient usage of both data storage and processing power.
The WISE technology was successfully implemented for the first time into the Astro-WISE information system (Astronomical Wide-Field Imaging System for Europe) - a unique advanced information system for astronomical research. Using the WISE technology, Astro-WISE connects databases, compute and storage facilities from national and international geographically-spread data centersin Netherlands, Germany, Italy and France.
|Last modified:||02 October 2015 10.56 p.m.|