NDLI: Massive-scale data management using standards-based solutions

Content Provider	IEEE Xplore Digital Library
Author	Shiers, J.
Copyright Year	1999
Description	Author affiliation: CERN, Geneva, Switzerland (Shiers, J.)
Abstract	In common with many large institutes, CERN has traditionally developed and maintained its own data management solutions. Recently, a significant change of direction has taken place and we have now adopted commercial tools, together with a small amount of site-specific code, for this task. The solutions chosen were originally studied as part of research and development projects oriented towards the Large Hadron Collider (LHC), which is currently under construction at CERN. They have since been adopted not only by the LHC collaborations, which are due to take production data starting in 2005, but also by numerous current experiments, both at CERN and at other High Energy Physics laboratories. Previous experiments, that used data management tools developed in-house, are also studying a possible move to the new environment. To meet the needs of today's experiments, data rates of up to 35 MB/second and data volumes of many hundred TB per experiment must be supported. Data distribution to multiple sites must be provided together with concurrent access for up to some 100 users. Meeting these requirements provides a convenient stepping stone towards those of the LHC, where data rates of 100-1500 MB/second must be supported, together with data volumes of up to 20 PB per experiment. We describe the current status of the production systems, database administration tools that have been developed, typically using the Java binding to the database, data distribution techniques and the performance characteristics of the system. Mechanisms to permit users to find and navigate through their data are discussed, including the issues of naming and meta-data and associated browsers. Coupled to the data management solution is a complete data analysis environment. This too is largely based on commercial, standards-conforming components with application-specific extensions as required. We discuss our experience in using such solutions including issues related to the integration of several commercial packages. Costs and risk factors are described, as well as issues related to responsiveness of the vendors to enhancement requests and support for additional platforms. Finally, we discuss the extent to which we have succeeded in our goal of using commodity solutions and the advantages of a database, as opposed to a file-based, approach for the management of vast volumes of data.
Starting Page	1
Ending Page	10
File Size	853291
Page Count	10
File Format	PDF
ISBN	0769502040
ISSN	10519173
DOI	10.1109/MASS.1999.829965
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	1999-03-15
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Large Hadron Collider Databases Environmental management Research and development Collaboration Laboratories Production systems Java Navigation Data analysis
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Massive-Scale Data Management using Standards-Based Solutions (1999)

Wide area data replication for scientific collaborations

CERN accelerator data logging and analysis

Developing location based services in Parlay/OSA

Grid Infrastructure and Large Scale Production for ATLAS

Incremental specification of views across databases

Navy Navigation Satellite System status

Forgetting bad behavior: memory for case-based navigation

Large-Scale ATLAS Simulated Production on EGEE

Massive-scale data management using standards-based solutions

Similar Documents

Massive-Scale Data Management using Standards-Based Solutions (1999)

Wide area data replication for scientific collaborations

CERN accelerator data logging and analysis

Developing location based services in Parlay/OSA

Grid Infrastructure and Large Scale Production for ATLAS

Incremental specification of views across databases

Navy Navigation Satellite System status

Forgetting bad behavior: memory for case-based navigation

Large-Scale ATLAS Simulated Production on EGEE

Massive-scale data management using standards-based solutions