Our partner is looking for a motivated Data Engineer with a strong foundation in SQL, JavaScript, and Python, with additional experience in Databricks and PySpark being a significant plus. This role requires a detail-oriented professional who can design, develop, and maintain robust data pipelines and systems to support our data-driven initiatives.

The company specializes in the fine wine investment market, providing a unique opportunity to combine a passion for wine with investment strategy. The product leverages sophisticated data analysis and machine learning to optimize investment decisions in the wine market, supported by a global network that spans over 80 countries and manages assets worth £250M. With a focus on transparency, security, and expert guidance, the company offers an enriching investment experience.

Responsibilities:

• Data Pipeline Development: Design, develop, and maintain scalable ETL (Extract, Transform, Load) processes to collect, process, and store large volumes of data from various sources
• Database Management: Optimize and manage SQL databases to ensure efficient data storage, retrieval, and overall performance
• Data Integration: Integrate data from various data sources, ensuring data quality, consistency, and accessibility
• Scripting and Automation: Use JavaScript and Python to automate data processing tasks and workflows
• Collaboration: Work closely with stakeholders to understand data needs and provide robust data solutions
• Performance Optimization: Analyse and optimize data workflows and queries for maximum performance and efficiency
• Documentation: Maintain comprehensive documentation of data engineering processes, architectures, and workflows
• Troubleshooting: Identify and resolve data-related issues, ensuring data integrity and reliability

Requirements:

• Proficiency in writing complex SQL queries, optimizing queries, and managing relational databases
• Experience in using JavaScript for data manipulation and automation
• Strong programming skills in Python, with experience in using libraries such as Pandas, NumPy, and others for data processing
• Experience with ETL tools and frameworks
• Strong problem-solving abilities and analytical thinking
• Excellent communication skills, with the ability to collaborate effectively with cross-functional teams
• English at B2 level

Preferred qualifications:

• Databricks Experience: Familiarity with Databricks for data processing and analytics
• PySpark Proficiency: Experience in using PySpark for big data processing
• Cloud Platforms: Experience with cloud platforms such as Azure