Senior Consultant - Kafka Job in Ness Digital Engineering

Job Description Senior Consultant - Kafka will be working on developing and maintaining the corporate messaging infrastructure ensuring message delivery, security, retention, and recoverability of the topics and messages. The role will have opportunities to work on various platforms managed by the team including Kafka, Apigee, Kong and Sonic MQ deployed on Cloud as well as on premise datacenters. The role will also work on on-call support troubleshooting issues reported by client teams to provide resolutions. Responsibilities Work on large scale multi-nodes Kafka cluster environments residing on AWS and on-prem OpenShift platform. Handle all Kafka environment builds, including design, capacity planning, cluster setup, performance tuning and ongoing monitoring. Perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka Cluster. Creation of key performance metrics, measuring the utilization, performance and overall health of the cluster. Deploy new/upgraded hardware and software releases as well as for storage infrastructure. Research and recommend innovative, and where possible, automate key administration tasks. Create topics, setup redundancy cluster, deploy monitoring tools, alerts Proactively monitor and setup alerting mechanism for Kafka Cluster and supporting hardware to ensure system health and maximum availability Perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka cluster Provide technical expertise and guidance to production support. Mandatory skills 5+ years of solid Kafka Admin experience in managing critical 24/7 applications Design, build, assemble, and configure application or technical architecture components using business requirements. Hands-on experience with Kafka clusters hosted on AWS and on-prem K8 Open Shift platform Experience in building Kafka pipelines using Terraform, Ansible, Cloud formation templates, shells etc. Experience in implementing security & authorization (permission based) on Kafka cluster. Experience in open source or Confluent Kafka, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center. Experience in High availability cluster setup, maintenance and ongoing 24/7 support Establish best practice standards for configuring Source and Sink connectors. Hands on experience in standing up and administrating Kafka platform from scratch which includes creating a backup & mirroring of Kafka Cluster brokers, broker sizing, topic sizing, h/w sizing, performance monitoring, broker security, topic security, consumer/producer access management (ACL) Knowledge of Kafka API (development experience is a plus) Knowledge of best practices related to security, performance, and disaster recovery. Ability to concentrate on a wide range of loosely defined complex situations, which require creativity and originality, where guidance and counsel may be unavailable. Demonstrate a product mindset with an ability to set forward thinking and direction. Ability to synthesize large amounts of complex data into meaningful conclusions and present recommendations. Ability to maintain a positive attitude while working with high demands and short deadlines that leads to working after hours. Must have excellent communications and interpersonal skills Optional Skills Experience in setting up Prometheus/Grafana or ELK monitoring tools is a plus. Experience as Linux (RHEL) /Unix administrator is huge plus. Experience in PostgreSQL, SQL Server, No-SQL (Couchbase), Oracle, and AWS Cloud is a plus. Understanding of or experience supporting .NET / Java application. Understanding of or experience with Programming languages like Python, PowerShell, etc is plus.