Data Engineer (python, Sql, Pyspark) Job in Clairvoyant
Data Engineer (python, Sql, Pyspark)
- Bengaluru, Bangalore Urban, Karnataka
- Not Disclosed
- Full-time
- Permanent
Job Responsibilities Collaborate with project stakeholders (client) to identify product and technical requirements. Conduct analysis to determine integration needs. Use different data warehousing concepts to build a data warehouse for reporting purpose. Build data pipelines to ingest and transform the data into our Data platform Apply best approaches for large scale data movement, capture data changes and apply incremental data load strategies Develop, implement and tune large-scale distributed systems and pipelines that process large volume of data Assist Data Science / Modeling teams in setting up data pipelines & monitoring daily jobs. Develop and test ETL components to high standards of data quality and act as hands-on development lead. Oversee and contribute to the creation and maintenance of relevant data artifacts (data lineages, source to target mappings, high level designs, interface agreements, etc.). Ensuring that developer responsibilities are being met by mentoring, reviewing code and test plans, verifying that design best practices as well as coding and architectural guidelines, standards, and frameworks are adhered to by offering guidance, communicating risk, and addressing roadblocks as they arise. Must have: Hands on coding experience in Java/Scala/python. Have worked on Spark Framework, Hadoop ecosystem. Comfortable coding in Scala-spark/Pyspark & Advanced SQL. 4+ years of experience in ETL (or) data engineering role in an analytics environment. Bachelor s degree in a technical field (Computer Science degree preferred not mandatory); Master s degree is a plus. Working knowledge of Relational Database Management Systems (RDBMS) like Oracle, Teradata, SQL server etc. Expertise in building data pipelines in Big data platforms; Good understanding of Data warehousing concepts Knowledge of Pyspark, Shell scripting, SQL, Python & some of the standard data science packages (Pandas, Numpy, etc.). Working knowledge of cloud platforms (GCP exposure would be a plus) Strong verbal and business communication skills. Strong business acumen & demonstrated aptitude for analytics that incite action.
Fresher
2 - 4 Hires
