Friday, June 12, 2026

Priorities

Work Log

CDC connector failure - inventory service - DA-1331

Situation

The CDC connector for the inventory service was failed due to large records in the process_execution_logs table.

Task

find out whether the process_execution_logs table is enabled in the data warehouse (bigquery).

Action

It turns out that it was not enabled in the bigquery, so I added it to the blacklisted table in the related ConfigMap.

Result

As a result, the CDC connector is running now and ignore records from the process_execution_logs table.

Takeaway

If we later enable this table in the data warehouse, resolve it by adding the max.request.size configuration instead.

Revamp data warehouse - Insider table - DA-1329

Situation

The Insider tables were hard to use because each tables contained old data from weeks event months ago -- not just that day's data. This made the data scientist queries slow because it needs scan all tables.

The table suffix name (YYYWMMDD) was follow the Insider UCD sync time, and the design was like:

dataset: Insider
├── events_20260601
├── events_20260602
├── events_20260603
└── ...

We could not get events from the last 30 days with the following query because the results will not be complete.

SELECT *
FROM `insider.events_*`
WHERE _TABLE_SUFFIX >= FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))

Therefore, we need to scan all tables, which makes the query very slow as each table can be 3–4 GB in size.

SELECT *
FROM `insider.events_*`
WHERE date(timestamp) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY);

Task

Find the right table design so data scientists can filter data by actual event date.

Action

I redesigned the Insider tables into a single table with date-based partitioning, and updated the data ingestion pipeline to support it.

Result

The new Insider table is now live. Data scientists can query it simply, like this:

SELECT *
FROM insider.events
WHERE _event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY);

Blockers

N/A

Carry-overs

Reflection

N/A