Jobs in Germany
Our client is an innovative digital platform that seamlessly connects people with opportunities, offering solutions across key industries such as real estate and mobility, seeking a skilled and experienced Data Engineer to lead the migration of their Hive Metastore to AWS Glue Catalog.
Position: Data Engineer
Start Date: Immediately
Country restrictions: Continental Europe
Language: English
Nearshoring requirement: 100% Remote(From EU region)
Duration of the project: 11 Months (start 15th Jan - End 15th Dec.)
Activities to be performed
Lead the migration of metadata from Hive Metastore to AWS Glue Catalog, ensuring data consistency and integrity throughout the process.
Utilize AWS Glue Hive Metastore Connector and Glue Crawlers to automate metadata migration and table creation.
Replicate Hive partitioning, data formats (e.g., Parquet, ORC), and schema to Glue, ensuring compatibility with AWS services like Athena and EMR.
Work with data analysts and data scientists to ensure seamless integration with data-driven analytics platforms.
Configure and manage AWS Glue security, including IAM roles and Lake Formation for data access control and governance.
Monitor data quality, performance, and cost efficiency in Glue and Athena, ensuring optimal data access and query performance.
Perform thorough testing and validation of the Glue Catalog migration, ensuring data integrity and functionality.
Troubleshoot and resolve issues related to the migration process and post-migration data pipelines.
