Monday, June 15, 2026

Priorities

Work Log

Insider Ingestion Pipeline - Resolve Delayed Data — DA-1329

Situation

We found a discrepancy in the event count between Insider dashboard and BigQuery. After taking a look at the CSV files produced by the Insider team, we found out that the CSV files for specific date can be delayed by up to 2 days. Therefore, I need to handle this delayed data so that the data in BigQuery can be complete.

Task

Update the script for the Insider ingestion pipeline to handle delayed data.

Action

I updated the script, build the Docker image, and bumped the Kubernetes Cronjob container to use image insider-exporter:0.0.11.

Result

The ingestion pipeline is now handled delayed data, and discrepancy issue is resolved.

AppsFlyer Ingestion Pipeline — Resolve Missing Data — DA-1335

Situation

The marketing team found that the number of registered users on the iOS platform was null. This is very contradictory, because they are now running a campaign for iOS, so the number was not supposed to be null. On the other hand, I am not receiving any alert related to the appsflyer ingestion pipeline.

Task

Find the root cause and apply the solution for this issue.

Action

I found out that the issue due to the related Kubernetes Jobs were failed due to out-of-memory. This is because the number of in-app events on iOS is growing due to the running campaign, which is causing the resource to reach its limit. Therefore, I adjusted the related Kubernetes CronJob to resolve the issue.

Additionally, to resolve today's reporting issue, I did several things, such as backfilling from certain days, re-running the related data mart jobs, and re-running the related dbt models.

Result

As a result, the marketing team has confirmed that they Superset dashboard is now showing correct data.

Blockers

N/A

Carry-overs

N/A

Reflection

N/A