Enhancing dependability in Big Data Analytics enterprise pipelines
Faculty / School
Faculty of Computer Sciences (FCS)
Department
Department of Computer Science
Was this content written or created while at IBA?
Yes
Document Type
Conference Paper
Publication Date
1-1-2018
Conference Name
International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage
Conference Location
Melbourne, NSW, Australia
Conference Dates
11-13 December 2018
ISBN/ISSN
85058566105 (Scopus)
Volume
11342
First Page
272
Last Page
281
Publisher
Springer, Cham
Keywords
Big data analytics, DataOps, Dependability, Enterprise, Pipeline
Abstract / Description
Big Data Analytics (BDA) brings extensive opportunities to enterprises to extract valuable information from high volume, velocity and variety data streams. However, the BDA dynamics can lead to significant project failures due to high-risk factors in terms of data availability, reliability, integrity, security and resilience which are the key components of a dependable system and are strongly linked to BDA process execution. Specifically, the heterogeneity of big data sources, diverse set of challenges related to big data integration and processing, along with a rapidly-expanding landscape warrant the need to make dependable big data systems capable of providing standard analytical solutions. In this paper, we propose the first dependable pipeline architecture for the BDA process which has a layered front-end and back-end implementation, employs the standard lambda architecture in a DataOps analytical cycle, incorporates state-of-the-art tools which are all open-source, and is coded entirely in the standard Python language to remove cross-platform implementation dependencies. We have implemented this architecture in five enterprise BDA projects but we are unable to present implementation details and results due to space limitations.
DOI
https://doi.org/10.1007/978-3-030-05345-1_23
Citation/Publisher Attribution
Zahid, H., Mahmood, T., & Ikram, N. (2018, December). Enhancing Dependability in Big Data Analytics Enterprise Pipelines. In International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage (pp. 272-281). Springer, Cham.
Recommended Citation
Zahid, H., Mahmood, T., & Ikram, N. (2018). Enhancing dependability in Big Data Analytics enterprise pipelines., 11342, 272-281. https://doi.org/10.1007/978-3-030-05345-1_23
COinS
