Book Chapter or Conference Paper Title
Enhancing dependability in Big Data Analytics enterprise pipelines
Faculty / School
Faculty of Computer Sciences (FCS)
Department of Computer Science
Was this content written or created while at IBA?
International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage
Melbourne, NSW, Australia
11-13 December 2018
Abstract / Description
Big Data Analytics (BDA) brings extensive opportunities to enterprises to extract valuable information from high volume, velocity and variety data streams. However, the BDA dynamics can lead to significant project failures due to high-risk factors in terms of data availability, reliability, integrity, security and resilience which are the key components of a dependable system and are strongly linked to BDA process execution. Specifically, the heterogeneity of big data sources, diverse set of challenges related to big data integration and processing, along with a rapidly-expanding landscape warrant the need to make dependable big data systems capable of providing standard analytical solutions. In this paper, we propose the first dependable pipeline architecture for the BDA process which has a layered front-end and back-end implementation, employs the standard lambda architecture in a DataOps analytical cycle, incorporates state-of-the-art tools which are all open-source, and is coded entirely in the standard Python language to remove cross-platform implementation dependencies. We have implemented this architecture in five enterprise BDA projects but we are unable to present implementation details and results due to space limitations.
Zahid, H., Mahmood, T., & Ikram, N. (2018, December). Enhancing Dependability in Big Data Analytics Enterprise Pipelines. In International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage (pp. 272-281). Springer, Cham.
Zahid, H., Mahmood, T., & Ikram, N. (2018). Enhancing dependability in Big Data Analytics enterprise pipelines., 11342, 272-281. https://doi.org/10.1007/978-3-030-05345-1_23