Data Engineer Job in Invest4edu Private Limited

About the role

Position Overview: As a Data Engineer, you will be responsible for designing, optimizing, and integrating complex data systems that support our organization's data strategy. You will lead a team of engineers and collaborate with cross-functional teams to ensure data integrity, availability, and performance. Your expertise in data optimization and integration will be critical in driving data-driven decision-making across the organization.

Key Responsibilities:

• Design and Development: Architect and develop scalable data pipelines and data models to support analytics and reporting using Azure Data Factory
• Optimization: Analyze and optimize existing data processes and workflows to improve performance and efficiency.
• Integration: Work on integrating diverse data sources, including cloud-based and on-premise systems, ensuring seamless data flow and accessibility with Azure Database Migration Services
• Deployment and Monitoring: Use Data Factory to build and schedule pipelines with monitoring capabilities
• Leadership: Mentor and lead a team of data engineers, providing guidance on best practices and technical solutions.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand data needs and provide innovative solutions.
• Quality Assurance: Ensure data quality and integrity through rigorous testing and validation processes.
• Documentation: Maintain comprehensive documentation for data architectures, processes, and integrations.

• Innovation: Stay updated with industry trends and emerging technologies to continually improve our data infrastructure.

What you will need to have

Qualifications:

• Bachelor s or Master s degree in Computer Science, Data Engineering, or a related field.
• 5+ years of experience in data engineering, with a strong focus on data optimization and integration.
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with ETL tools (e.g., Apache NiFi, Talend) and data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).
• Strong understanding of database technologies, both SQL and NoSQL (e.g., MySQL, PostgreSQL, MongoDB, Cassandra).
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and data services (e.g., AWS Glue, Azure Data Factory).
• Knowledge of data governance, data privacy, and compliance best practices.
• Excellent problem-solving skills and the ability to work in a fast-paced environment.
• Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical teams.

Preferred Qualifications:

• Experience with big data technologies (e.g., Hadoop, Spark).
• Knowledge of machine learning concepts and frameworks.
• Previous experience in a leadership role or managing data engineering teams.

Our Values

Make It Happen

We strive to deliver results with the highest impact and are committed to following through. We continuously optimise through collaboration to achieve the best collective outcomes.

Be Proactive

We seize the initiative and are always looking for continual improvement, for ourselves, our team and the cities we serve. We relish the chance for action even if it leads us outside of our normal scope.

Trust Facts Over Opinions

Our decisions are evidence-based. We assess situations fairly, wherever possible using reason and logic rather than unverified opinions.

Stay Curious

We ask questions, challenge assumptions, learn from mistakes and are ready to leave behind what we thought we knew.

Do More With Less

We understand that both time and resources are finite. We balance our priorities and think strategically about how we can maximise our impact and the return on investment.

If you are passionate about making a real-time impact and want the opportunity to play an instrumental part in our growth story, we want you riding for our team!