Monday, June 22, 2026

Priorities

Migrate Git repositories, and it's CICD pipeline from GitLab to GitHub — DA-1346, DA-1344

Work Log

GitLab to GitHub Migration — DA-1346, DA-1343

Today, I migrated the following repositories:

Repository	Jira Ticket	GitLab	GitHub
warehouseflow	DA-1346	https://gitlab.com/allofresh/data/warehouseflow	https://github.com/allofresh/warehouseflow
etl-batch	DA-1343	https://gitlab.com/allofresh/data/etl-batch	https://github.com/allofresh/etl-batch

Find the details of this migration in this page.

Insider Data Pipeline — Data Duplication Issue

Situation

The Insider team informed us that their integration system produce duplicated data. They sent the CSV files to our cloud storage bucket twice. Only CSV files whose names contain the 2531 code are impacted, and they all have exactly the same size. If this happens I could deduplicate the data by session_id, event_name, user_id , timestamp fields.

Task

Find out the duplicated files in the cloud storage.

Action

I manually validate the duplicated files by eye-balling since I only need to check from partition 4/6/26 until 22/6/26.

Result

Fortunately, There is no duplicated file so that I don't need to do deduplication process.

Blockers

N/A

Carry-overs

Create a live demo for LLM Inference

Reflection

N/A