BibTeX:

@article{IJIRSTV6I2001,
     title={Meta Data Driven Big Data Testing Automation},
     author={Anita Kamdi},
     journal={International Journal for Innovative Research in Science & Technology},
     volume={6},
     number={2},
     pages={1--4},
     year={},
     url={http://www.ijirst.org/articles/IJIRSTV6I2001.pdf},
     publisher={IJIRST (International Journal for Innovative Research in Science & Technology)},
}

More and more enterprises are adopting Big Data systems for the benefits it offer, however simpler systems/tools for data quality checks are not readily available and are not easier to implement. Lack of data quality checks leads to inconsistent or incorrect data flowing to the systems consuming it further. As a result data quality may not present correct picture of the insights and data analysis may not be useful for business and operational needs. This paper proposes a simple meta data driven solution to create a test automation tool that will help in executing regular big data loads with ease and perform data quality checks. It can be easily customized to include data quality checks as per requirement.

Big Data Quality Assurance, Big data testing Automation, Spark, Hive, Impala