Data Engineer

Role, Key Accountabilities, and Requirements: Senior Data Engineer

The Data Engineer is a senior technologist with at least ten years of expertise in data design, development, and architecture. They will be highly skilled in the Azure Cloud Environment and possess excellent communication skills complemented by strong functional and consultative skills. The senior technologist will work collaboratively in a dynamic, multidisciplinary team and interact daily with the client to drive solutions.

Key Accountabilities

• Design, develop, and maintain data pipelines and workflows to ingest, process, and analyze large-scale data.
• Provide thought leadership and collaborate daily with the team and client.
• Proactively contribute to technical solutions, participate as a multidisciplinary team member to analyze sponsor requirements and designs, implement software, and perform analyses to address these requirements.
• Engage frequently with other data engineers to share relevant knowledge, opinions, and recommendations, and work to meet deliverables as a team.
• Utilize the Azure Databricks platform to deploy, configure, and manage the platform and services.
• Develop, publish, and schedule reports and dashboards to meet business requirements.
• Assist business users with functional and data requirements to enhance data models.
• Develop self-service models and data analytics using Power BI service
• Development and support of visualization tools
• Develop tabular, multidimensional models compatible with standards
• T-SQL queries, creating clustered and non-clustered Indexing, working on DTS, Data Cleansing, and Data Migration.
• Integrations with CSV, Excel, and intake of files and consumption of data from APIs and web services.
• Perform other duties as assigned.

Skill Requirements

• Senior-level experience in Python, Azure, Databricks, and Azure Data Lake Storage.
• Proficient with advanced and new Databricks tools including but not limited to Lakeflow, Lakebase, and Genie.
• Experience with financial data, POS, payments, and transactions in the design and use of models.
• A high-level data engineering competency in design and development. The ability to effectively handle concurrent technical tasks with conflicting priorities and approach complex problems enthusiastically and creatively.
• The ability to change focus when necessary and to work independently and implement research concepts in a multi-disciplinary team environment, where commitments and deadlines are essential to project success.
• Excellent interpersonal skills are necessary to interact with personnel at all levels.
• Excellent verbal and written communication skills are necessary to effectively collaborate in a team environment and present and explain technical information.
• Excellent analytical, problem-solving, and decision-making skills to develop creative solutions to complex problems.
• Design
• Design, develop, and maintain data pipelines and workflows to ingest, process, and analyze large-scale data.
• Design and implement ETL systems to handle a variety of data sources.
• Responsible for design methodology and project documentation.
• Elicit requirements and propose improvements; trace and track requirements to completion.
• Design of SQL Server Integration Services (SSIS) packages, including various control flow tasks and data flow transformations.
• Concepts
• Exceptional knowledge and understanding of capacities and workspaces.
• Dashboards, reports, workbooks, datasets, and data flows.
• Create, modify, and publish dashboards to the Power BI service.
• Prototyping, designing, and requirements analysis.
• Participate in planning activities, including defining and organizing tasks, roles, and responsibilities, and prioritization of deliverables.
• Development
• Strong Python skills.
• Work with the Apache Spark API PySpark.
• Experience with Databricks.
• Experience with Azure DataLake Storage (ADLS).
• Experience in TimeTravel.
• Experience with Datatables.
• Experience with AutoLoader.
• Experience with Lakebase.
• Experience with Lakeflow.
• Strong skills with IDEs JupyterLab, RStudio, or equivalent.
• Customize, extend, and embed Power BI analytics.
• Write, edit, and debug complex T-SQL queries.
• Develop SQL Server Integration Services (SSIS) packages, including various control flow tasks and data flow transformations.
• Apply comprehensive APIs and use SDK libraries.
• SSRS, SSIS, SSAS, and other BI stack technologies.
• Build and deliver dashboard and analytical solutions using Power BI and related Azure data services.
• Develop custom visuals.
• Data prep, data modeling, and data visualization.
• Transform and consolidate data from disparate sources.
• Develop tabular and multidimensional models.
• Data modeling and integration through extracting, transforming, and loading (ETL) data from various sources.
• Write calculations using Data Analysis Expressions (DAX).
• Advanced knowledge in all phases of the Software Development Life Cycle (SDLC).
• Advanced knowledge of Agile/Scrum development methodologies.
• Conduct testing as needed to ensure that the solution meets the requirements.
• Update and maintain related documentation (e.g., user guides, FAQ, site improvement documentation).
• Architecture
• The architecture of datasets and dashboards.
• The full range of modern data architecture, from ingestion to storage to consumption.
• Drive enterprise architecture and technology roadmaps.
• Power Platform architecture, development & troubleshooting.
• Service architecture design and administration.
• Guidance to Data Engineers on best practices for developing high-quality datasets and visually stunning reports.

Certification Requirements

• Hold multiple Azure certifications.
• Hold multiple process certifications, i.e., Scrum.
• Obtain certifications immediately or show dedication to acquiring them. The employment offer letter will explicitly outline this requirement, and you must agree to it, initiate it, and sign it as a condition of employment. Non-compliance may lead to termination.

Professional Development

• Training Roadmap.
• Learning Agreement.

Compensation

* Salary of $115,000 - $125,000.

Benefits Package

• Company-paid medical/dental/vision/life coverage.
• 401K with company match.

The Perks

• Take the time when you need it - flex days for vacation, sick, and self.
• Flexible work hours.
• Shorter days near the holidays.
• Attend company-sponsored events and team outings.
• Participate in employee care and recognition programs.
• Wellness Program

Background Screening

• A background check is required.
• A drug screen is required.

Citizenship

* U.S. citizenship is required.

Education

• BS degree in Computer Science or a related field.
• Advanced degree preferred.

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, creed, religion, disability, age, sex, ethnic or national origin, marital status, sexual orientation, gender identity or presentation, pregnancy, genetics, veteran status, or any other status protected by state or federal law.

Job Type: Full-time

Pay: $115,000.00 - $125,000.00 per year

Benefits

• 401(k)
• 401(k) matching
• Dental insurance
• Flexible schedule
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Professional development assistance
• Referral program
• Retirement plan
• Vision insurance

Compensation Package

• Bonus opportunities
• Performance bonus
• Yearly pay

Application Question(s)

* Do you have experience with Databricks?

Experience

* technical: 10 years (Required)

Work Location: Remote