Learning from 25 years of the extensible N-Dimensional Data Format
arXiv:1410.7513 · doi:10.1016/j.ascom.2014.11.001
Abstract
The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink applications were used extensively, primarily in the UK astronomical community, and form the basis of a number of advanced data reduction pipelines today. This paper provides an overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems.
19 pages, 7 figures, submitted to the Astronomy & Computing special issue on astronomy data formats
References in corpus (7)
- SCUBA-2: iterative map-making with the Sub-Millimetre User Reduction Facility
- ORAC-DR: A generic data reduction pipeline infrastructure
- Spectroscopic Analysis in the Virtual Observatory Environment with SPLAT-VO
- Learning from FITS: Limitations in use in modern astronomical research
- An overview of the planned CCAT software system
- Observatory/data centre partnerships and the VO-centric archive: The JCMT Science Archive experience
- Significant problems in FITS limit its use in modern astronomical research
Cited by in corpus (13)
- SONS: The JCMT legacy survey of debris discs in the submillimetre
- ORAC-DR: A generic data reduction pipeline infrastructure
- Spectroscopic Analysis in the Virtual Observatory Environment with SPLAT-VO
- Learning from FITS: Limitations in use in modern astronomical research
- SAMP, the Simple Application Messaging Protocol: Letting applications talk to each other
- Reimplementing the Hierarchical Data System using HDF5
- Astronomical Data Formats: What we have and how we got here
- An overview of the planned CCAT software system
- Your data is your dogfood: DevOps in the astronomical observatory
- The Past, Present and Future of Astronomical Data Formats
- Is HDF5 a good format to replace UVFITS?
- A new HDF5 based raw data model for CCAT
- The General Single-Dish Data Format: A Retrospective