Engenheiro de Dados jr
Envie uma proposta de trabalho diretamente para este candidato
Hi there! I am a data engineer and continuous learner, driven by a passion for leveraging data to drive meaningful insights and solutions. My focus is on designing and implementing ETL and ELT data pipelines and managing the end-to-end data lifecycle.
Skilled in Python, Pandas, Spark, Pyspark, dbt, and Seaborn for data transformations and insightful analysis.
Utilized various AWS services such as Glue, S3, Lambda, Kinesis, documentDB, ECS, Fairgate and API Gateway, alongside with Databricks and Snowflake for analytics and data processing.
Working in SQL, Postgres and MongoDB databases to implement data storage solutions.
Familiarity with CI/CD practices, containerization using Kubernetes and Docker, version control with Git & GitHub, and infrastructure automation with Terraform.
While my experience with Kafka is limited, I have some exposure to it for real-time data streaming and event-driven architectures.
I have a foundational understanding of PyTorch, TensorFlow, exploratory data analysis and basic supervised, semi and unsupervised learning techniques, as well as recommender systems.
I have gained experience in developing back-end systems using Python, Django, Django REST API, and following software engineering principles.
Adhere to industry best practices such as Pep 8, SOLID, and Domain Driven Design principles.
With a background in computer science, law (+10 years of experience in international private law and BA's degree with honors) and entrepreneurship (product, marketing, sales), I bring a unique perspective to problem-solving with the ability to collaborate effectively in interdisciplinary environments
In my most recent personal project, I developed an automated ELT data pipeline using a variety of tools and technologies including multiple AWS Services, Spark, Python, Pandas, Terraform, Docker, Ansible, Airflow, Databricks, GitHub Actions and dbt cloud. This project provided insightful video analysis and trained a Longformer Language Model (LLM) using YouTube and Whisper transcriptions.
Competências: Terraform · Databricks · Infrastructure as code (IaC) · Data Lake · AWS Glue · AWS Athena · AWS Lambda · Amazon Kinesis · dbt · AWS API Gateway · Data Warehousing · Data Engineering · Data Modeling · Docker · Pandas (Software) · Apache Spark · Ansible · GitHub · PySpark · Data Pipelines · Apache Airflow · SQL · Extract, Transform, Load (ETL) · Python · Design Patterns · DevOps
Universidade Presbiteriana MackenzieUniversidade Presbiteriana Mackenzie
