OpenOpportunity Sourced Jobs — Bengaluru, Hyderabad, Kolkata, Mumbai, New Delhi, Pune, Chennai
Full Time · On Site · Entry Level
Role: Data Engineer | Department: Engineering - Software & QA | Industry: IT Services & Consulting | Education: UG: Any Graduate PG: Any Postgraduate Job summary Our client is seeking a Senior Data Engineer to join our fast-growing team The successful candidate will join the testing team to work on ETL and development tasks This is an exciting and challenging opportunity to build out new pipelines combining and processing large amounts of structured data from a variety of sources with the power of PySpark at your fingertips Key Responsibilities Architect and build pipelines using AWS cloud computing solutions that make data available with robustness, maintainability, efficiency, scalability, availability and security Develop Python and/or Spark (Preferably PySpark, but Spark-Scala is also good to have) code that implements complex data transformations Design and maintain databases and APIs for storage and transmission of data between applications Monitor pipelines in production (and develop tools to facilitate this) Work collaboratively with other team members (brainstorming, troubleshooting, and code review) Liaise with other development teams to ensure the integrity of data pipelines Skills, Knowledge & Expertise Skills: Excellent Python and PySpark programming (including Pandas/PySpark dataframes, web and database connections) Excellent understanding of ETL processes within Amazon Web Services (AWS) Apache Spark, AWS Glue, Athena, S3, Step Functions, Lake Formation Software development lifecycle best practices Test-driven development Serverless computing (AWS Lambda, API Gateway, SQS, SNS, EventBridge, S3, etc,) SQL and NoSQL database design and management (DynamoDB, MySQL) Strong SQL coding skills (Spark-SQL, Presto SQL, MySQL, etc,) Infrastructure as code (CloudFormation) Experience in Shell Scripting (preferably Linux) Version control with Git/Github Agile principles, processes and tools Excellent written and verbal communication skills Experience: Designing, deploying and managing complex production data pipelines that interact with a range of data sources (file systems, web, database, users) Strong experience with Amazon Web Services (AWS) 5 years' work in data engineering field At least 2 years' experience with PySpark and AWS data tools (particularly, Glue) Knowledge: Data modeling, data pipeline architecture, Big Data implementation Software development lifecycle best practices Financial knowledge would be an asset Qualifications/Training: Bachelor's degree or equivalent in Computer Science or a related subject What you'll get in return Competitive salary package Private healthcare contribution Annual pay review Regular team socials Working within a culture of innovation and collaboration Opportunity to play a key role in a pioneering growth company Company Laptop will be provided Note: This posting advertises a "0-2 years" experience band, but its own body explicitly requires "5 years' work in data engineering field" and "At least 2 years' experience with PySpark and AWS data tools" -- a direct, quantified contradiction between the structured experience filter and the stated requirement. The title "Senior Data Engineer" reinforces that this is not an entry-level role. The posting also opens with "Our client is seeking..." -- Cybernetics Controls is acting as a staffing intermediary for an undisclosed end client, whose identity is never revealed anywhere in the listing, consistent with the masked-employer pattern seen elsewhere in this series.
api gateway, scala, sql coding, pyspark, sql, git, spark, writing, linux, shell scripting, mysql, etl, communication skills, python, github, dynamo db, aws lambda, nosql, pandas, amazon sqs, aws glue, eventbridge, sns, athena, agile, aws, etl process, presto