Data Engineer/Architect (Databricks)

CHI Software is a global software development services provider delivering AI-driven solutions, cloud services, and data engineering across EdTech, FinTech and HealthTech. We help businesses modernize, scale and unlock new value through the latest technologies.

We are looking for a Data Engineer/Data Architect (Databricks) to help build and evolve our clients’ data platforms. You’ll design, build, and maintain the pipelines and architecture that power analytics and machine learning across a range of projects.

This is a fully remote position, work from anywhere in the U.S., in the time zone that suits you, with a flexible schedule and minimal bureaucracy.

What You’ll Do

Design and build ETL/ELT pipelines on Databricks and Apache Spark
Build and maintain Delta Lake architectures (bronze/silver/gold layers)
Contribute to data governance, access control, and metadata management (e.g., Unity Catalog)
Optimize Spark job performance and manage cloud infrastructure costs
Design data models and architectural patterns for analytics and ML use cases
Set up and maintain pipeline orchestration (Databricks Workflows, Airflow)
Ensure data quality, reliability and observability across the platform
Evaluate tools, frameworks and architectural decisions as the platform scales
Collaborate with Data Scientists, Analysts and Product teams to translate business needs into technical solutions
Review code and architecture, share knowledge and help raise the engineering bar across the team

Requirements

Must-have:

5+ years of experience in Data Engineering, Data Architecture, or a related field
Hands-on production experience with Databricks
Strong command of Apache Spark (PySpark or Scala)
Advanced SQL and experience working with large-scale datasets
Experience with Delta Lake
Familiarity with at least one major cloud platform (AWS, Azure or GCP)
Solid understanding of data modeling, data warehousing, and Lakehouse architecture principles
Ability to make and communicate sound technical decisions, whether at the pipeline or platform level

Nice to have:

Databricks Certified Data Engineer certification (Associate or Professional)
Experience with Unity Catalog or similar data governance tools
Experience with streaming data processing (Structured Streaming, Kafka)
Familiarity with dbt, Airflow, or other orchestration tools
Exposure to MLOps / supporting ML pipelines, especially when collaborating with a Data Science team
Experience designing or scaling data platforms in a product company with high-volume data operations
Experience mentoring engineers or leading technical decisions across a team