Middle Data Engineer

About the Team

The team at the moment is 50+ people, yearly plan is an active expansion to +25 specialists in different teams/locations.

Management (basic business decisions) – on the client-side, + each team has a Lead or PM.

We have 15 people: Solution Architect, React, Data Engineer, DevOps, BA/PM.

We have been involved in the project for exactly one year.

Responsibilities:

• Write, and review AWS Lambda functions, and microservices running on AWS ECS for Search Platform
• Build robust, high performance and massively scalable Rest API
• Design, code, and test massive scale data pipelines and solutions
• Deploy functions and microservices multiple times in a day
• Build distributed parallel processing tools and frameworks to process 100s of Terabytes of data and index them for optimal search performance

Requirements:

• 3+ years of Python (or GoLang)
• Development experience 3+ years of experience in S3, AWS ElasticSearch, and AWS DynamoDB for massive scale (100s of Terabytes of data)
• 3+ years of extensive experience in designing, developing, and running extremely efficient data pipelines across multiple AWS Data Management Platforms (DynamoDB, ElasticSearch, AWS S3, etc)
• 3+ years of experience in AWS Batch/EMR
• Aspiration to learn and become Data Scientist in the next 6-12 months
• Proficient in English
• Preferred Skills: AI/ML – NLP, NumPy, PyTorch, Pandas, etc.
• AWS EMR/Spark, AWS Glue, Athena, Redshift, etc.
• AWS Cloud Watch, ELK Stack, TICK Stack SEO, Google Search Console, and Google Analytics

Methodologies/processes

• Standard flow Agile – Scrum, all standard ceremonies.
• Release 2 times a week on Tuesdays and Thursdays.
• Sprints last for 2 weeks.
• Daily rallies (internal with the team).
• Planning, Retrospectives, Calls to clarify the requirements.
• Once a quarter for all employees – All hands session (results of the previous period, plan for the next period)

What else?

The motivation to develop and introduce something new is welcomed.

Management is very open for communication and feedback, they are worried about organizing comfortable conditions for everyone in order to show their best qualities.

The interaction between the teams is generally friendly, there are moments with Front / Back or Back / Data, minor misunderstandings on work issues, but everything is always resolved peacefully 🙂