Thursday, June 11, 2026

Priorities

1. Deactivate unused data ingestion pipeline.
2. Confirm data lateness issue to the Insider team.

Notes

Office

Jira tickets: 1. DA-1290 2. DA-1283 3. DA-1316

Today, I deactivated the unused data ingestion pipeline because they are not relevant again. Details are below:

• Deactivate FarEye ingestion pipeline
• Deactivate CDC Connector for Finance service
• Deactivate Spark Cluster & Job for Search events

additionally, this initiative can reduce Infrastructure costs.

\ Jira ticket: DA-1304

Data lateness has become a major concern for us because events from the last week, month, or even year are being processed today. Below are the impacts for this issue:

1. Reports cannot guarantee data completeness.
2. Reports are inconsistent for a specific time period. For example, the report for May will show different data each day it is generated.

To address this, I already raised this to the Insider team via Zendesk ticket.

Personal

N/A

Blockers

N/A

Done

Jira tickets: 1. DA-1290 2. DA-1283 3. DA-1316

I have created several merge/pull requests to deactivate the data ingestion pipelines.

1. kubernetes!8822
2. kubernetes!8821
3. schemahub-registry#23
4. allofresh-manifest!120

\ Jira ticket: DA-1304

I have replied to the existing zendesk ticket to raise the data lateness issue.

1. Zendesk ticket: #358435

Carry-overs

1. Create Inference service prototype utilize t5-small machine learning model.

Reflection

Get a better sleep so that the next day can be productive.