NDLI: Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

Content Provider	Springer Nature Link
Author	Rogers, David Harvey, Ian Huu, Tram Truong Evans, Kieran Glatard, Tristan Kallel, Ibrahim Taylor, Ian Montagnat, Johan Jones, Andrew Harrison, Andrew
Copyright Year	2013
Abstract	In this paper, we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multi-workflow environment. We introduce a scalable and robust execution pool environment that supports workflows consisting of sub-workflows built upon a multitude of different workflow engines and environments, and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles. We also present a meta-workflow scenario based upon this system. Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange (ORE) Web-based standard, to provide a common format for representing and sharing workflows and the associated metadata required for their execution. This generalized bundling approach is already available within five workflow engines and has proven a useful environment for inter-workflow experimentation. The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool. Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol (XMPP), which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in meta-workflow pipelines. We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents, and that agent failures have very limited impact. We then demonstrate the applicability of our architecture by describing how a Java-based music analysis workflow can be distributed within such a multi-workflow environment consisting of the Triana and MOTEUR workflow engines.
Starting Page	457
Ending Page	480
Page Count	24
File Format	PDF
ISSN	15707873
Journal	Journal of Grid Computing
Volume Number	11
Issue Number	3
e-ISSN	15729184
Language	English
Publisher	Springer Netherlands
Publisher Date	2013-06-08
Publisher Place	Dordrecht
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Scientific workflows Distributed computing infrastructure Grid computing Cloud computing Interoperability Data modelling Processor Architectures Management of Computing and Information Systems User Interfaces and Human Computer Interaction
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Information Systems Software Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Fine-Grain Interoperability of Scientific Workflows in Distributed Computing Infrastructures

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

Infrastructure Aware Scientific Workflows and Infrastructure Aware Workflow Managers in Science Gateways

Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal

A Formal Approach to Support Interoperability in Scientific Meta-workflows

Parameter Sweep Workflows for Modelling Carbohydrate Recognition

Exploring Workflow Interoperability for Neuroimage Analysis on the SHIWA Platform

Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Towards the Scheduling of Multiple Workflows on Computational Grids

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

Similar Documents

Fine-Grain Interoperability of Scientific Workflows in Distributed Computing Infrastructures

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

Infrastructure Aware Scientific Workflows and Infrastructure Aware Workflow Managers in Science Gateways

Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal

A Formal Approach to Support Interoperability in Scientific Meta-workflows

Parameter Sweep Workflows for Modelling Carbohydrate Recognition

Exploring Workflow Interoperability for Neuroimage Analysis on the SHIWA Platform

Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Towards the Scheduling of Multiple Workflows on Computational Grids

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions