Wednesday, June 10, 2026

Priorities

  1. Validate the data issue in the Insider BigQuery tables.
  2. Create Inference service prototype utilize t5-small machine learning model

Notes

Office

Jira ticket: DA-1304

The data scientist team raised data issues in the Insider BigQuery tables. These tables populated by my ingestion pipeline that reads from the csv files produced by the Insider external platform integration.

This data is very important so that the company can monitor our product based on customer behaviors. Therefore, I need to address the issue so that the report showing the correct data.

The issues raised by the data scientist are: 1. the CSV files for the specified date contain events from previous days, months, and even years. 2. the csv files contains duplicate data.

To address these issues, I need to validate the data directly against the CSV files in cloud storage instead of BigQuery. For a single day, the data is quite large—around 10GB Therefore, I need a proper tool to validate the data, such as Apache Spark, which can run on my local machine.

Create a development environment using Devbox.

devbox init
devbox shell
devbox add spark@3.5.5

Download a JAR for reading csv files in the cloud storage.

curl -LJO https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v3.1.17/gcs-connector-3.1.17-shaded.jar

Sample of code to reads csv files in cloud storage with Spark.

from pyspark.sql import SparkSession


spark = SparkSession.builder \
    .appName("insider-data-exploration") \
    .config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") \
    .config("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS") \
    .config("spark.hadoop.fs.gs.project.id", "gcp-project-id") \
    .config("spark.hadoop.fs.gs.auth.type", "APPLICATION_DEFAULT") \
    .getOrCreate()

path = "gs://office-bucket"
df = spark.read.csv(path, header=True, inferSchema=True)
df.show()

Execute the script with the following command.

spark-submit \
--executor-memory 16g \
--driver-memory 4g \
--master "local[12]" \
--jars gcs-connector-3.1.17-shaded.jar \
--driver-class-path gcs-connector-3.1.17-shaded.jar \
duplicate_events.py

Result:

The issues were raised by the data science team because the data produced by Insider is supposed to look that way. Therefore, I need to create Zendesk tickets to the Insider team.

Jira: DA-1326

The data team got an adhoc task to update maps-intelligence platform to include out-of-range users. To address this, the data scientist updated the logic of hexagon, isochrone, and density maps. After that, I need to update the container image and kubernetes manifest related to this platform.

Below are the tasks to finish the task: 1. gudang-images!143 2. kubernetes!8818

Personal

I tried to understand the Text-To-Text Transfer Transformer (T5) article before develop the prototype of inference service.

My understanding is that this model suitable for the following tasks:

  1. translation
  2. linguistic acceptability
  3. sentence similarity
  4. document summarization

Therefore, I plan to create the prototype that covers the above tasks. Additionally, I intend to gain a deeper understanding of this machine learning model.

Blockers

N/A

Done

Jira ticket: DA-1304

I have created tickets to the Insider team. Tickets are follows:

  1. #358435
  2. #358439

Jira: DA-1326

I have created merge requests to update the map-intelligence website.

  1. gudang-images!143
  2. kubernetes!8818

Carry-overs

  1. Create Inference service prototype utilize t5-small machine learning model

Reflection

N/A