Enhancing dependability in Big Data Analytics enterprise pipelines

Faculty / School

Faculty of Computer Sciences (FCS)


Department of Computer Science

Was this content written or created while at IBA?


Document Type

Conference Paper

Publication Date


Conference Name

International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage

Conference Location

Melbourne, NSW, Australia

Conference Dates

11-13 December 2018


85058566105 (Scopus)



First Page


Last Page



Springer, Cham

Abstract / Description

Big Data Analytics (BDA) brings extensive opportunities to enterprises to extract valuable information from high volume, velocity and variety data streams. However, the BDA dynamics can lead to significant project failures due to high-risk factors in terms of data availability, reliability, integrity, security and resilience which are the key components of a dependable system and are strongly linked to BDA process execution. Specifically, the heterogeneity of big data sources, diverse set of challenges related to big data integration and processing, along with a rapidly-expanding landscape warrant the need to make dependable big data systems capable of providing standard analytical solutions. In this paper, we propose the first dependable pipeline architecture for the BDA process which has a layered front-end and back-end implementation, employs the standard lambda architecture in a DataOps analytical cycle, incorporates state-of-the-art tools which are all open-source, and is coded entirely in the standard Python language to remove cross-platform implementation dependencies. We have implemented this architecture in five enterprise BDA projects but we are unable to present implementation details and results due to space limitations.

Citation/Publisher Attribution

Zahid, H., Mahmood, T., & Ikram, N. (2018, December). Enhancing Dependability in Big Data Analytics Enterprise Pipelines. In International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage (pp. 272-281). Springer, Cham.