Job Description

Big Data Engineer – PySpark ,Spark Streaming, Spark Core, Kafka, Spark SQL, HIVE, HDFS, HBASE and AWS snowflake Cloud DB, CSPO and AWS certified with ISTQB Build PySpark based applications for both batch and streaming requirements on Cloudera Distributed systems, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well. Develop and execute data pipeline testing processes and validate business rules and policies Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame. Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc.) and compression codec respectively. Ability to design & build real-time applications using Apache Kafka & Spark Streaming Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec. Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources. Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories Responsible for creating projects in Zephyr and Jira and maintaining project folders and structures

Job qualifications:

# 8 years experience in Business Intelligence in Banking domain. Should be Bachelors in Computer Science Engineering .