Large Data Handling

For some instruments an essential part is Large-data handling. This concerns huge data transport and archiving, databases, virtual observatories, data analytics, and interaction with big data. Also, data security, quality, integrity, reproducibility and archiving of research results, as well as data access and ownership are important aspects. In many data-intensive applications, such as astronomy and next-generation gene sequencing, data volumes have become so high that during acquisition the data is analyzed on-the-fly and only 5-10% of the data is retained for further processing and storage, often in compressed form (A. Wright, Big data meets Big Science, and its technical challenges, Commun. ACM 57(7), pp. 13–15, 2014). A similar situation pertains to health care where huge amounts of data are collected and linked at an individual level, and efficient methodologies need to be applied to combine data in both causal and predictive research to derive valid results. This asks for fast and efficient methods for data selection and compression. However, big data does not always presuppose big instruments. Sometimes simple solutions using (large numbers of) cell phones are very effective.