# Repo toolkit map — ground truth (with `file:line`)

The one source of truth for the seven add-on topics. Verified against source (surveys +
direct reads). Lessons cite this; if code moves, fix it here first.

---

## 1. MIGRATIONS (golang-migrate)

- **Layout:** `migrations/{service}/` — 17 services (bob 759, eureka 641, … zeus 9). Files are
  **`NNNN_migrate.up.sql`** — 4-digit sequence **starting at 1001**, literal word `migrate` (not a
  slug). Descriptions live in side files (`migrations/eureka/README.md`, `migrations/bob/CHANGELOG.md`).
- **NO down migrations.** `find migrations -name '*.down.sql'` → **0** (vs **2887** `.up.sql`). The
  wrapper only ever calls **`m.Up()`** — `internal/golibs/database/migrate.go:100`; `ErrNoChange`/
  `ErrNilVersion` swallowed (:101-104). Rollback isn't a motion — you write a new forward migration.
- **Wrapper:** `internal/golibs/database/migrate.go:35` `MigrateDatabase` (golang-migrate v4, pgx
  driver, Cloud SQL auto-detect :38-56, `migrate.NewWithDatabaseInstance` :81). Drivers blank-imported
  in the single binary: `cmd/server/main.go:49-50`.
- **Run:** `gjob sql_migrate` subcommand (`cmd/server/migrate.go:13`) — **not a Makefile target**.
  Local: `./local/run.bash start -m migration -s <svc>` (run.bash:135-148) → `docker-compose.migration.yaml`
  runs `[gjob, sql_migrate, --configPath=…, --secretsPath=…<svc>_migrate.secrets.encrypted.yaml]`. Prod:
  same subcommand as a Helm hook job. `source: file:///migrations/{service}` per service config.
- **RLS boilerplate:** every `CREATE TABLE` gets `resource_path text … default autofillresourcepath()`
  + two RLS policies (`rls_<t>`, `rls_<t>_restrictive`) + `ENABLE`/`FORCE ROW LEVEL SECURITY`
  (`.claude/skills/migration/SKILL.md:17-38`). `bob/1001_migrate.up.sql` = a pg_dump baseline.
- **Consolidated schema ↔ sqlc:** `internal/eureka/eureka.sql` + `dwh.sql` are `pg_dump` outputs used
  as sqlc `schema:` inputs (`v2/sqlc.yaml:71`). Kept in sync by **`make gen-db-schema`** (`Makefile:267`
  → `scripts/gendbschema.bash`): fresh `postgres:11.9` → apply ALL migrations → `pg_dump --schema-only`.
  Only **14** hardcoded services (`developments/dbschema.Dockerfile:20`). Migration skill step 4.
- **Docs:** `.claude/skills/migration/SKILL.md` (authoritative), `.claude/rules/{sqlc-queries,security}.md`.
- **Myths:** no `.down.sql`; not descriptive filenames; don't edit `eureka.sql` (generated); `gen-db-schema`
  ≠ all services; there is no `make migrate`.

---

## 2. LINTING & GITHOOKS

- **`.golangci.yaml`** (version `"2"`, `default: none`) enables: `bodyclose, copyloopvar, errcheck,
  goconst, gocritic, gocyclo, gosec, govet, ineffassign, misspell, prealloc, revive, rowserrcheck,
  staticcheck, unconvert, unparam, unused, whitespace` (:5-23). **`run.tests: false`** (:81 — test files
  are NOT linted). Formatters `gci/gofmt/goimports`, import grouping `standard → prefix(github.com/manabie-com)
  → default` (:54-65). `mock/`, `pkg/`, `examples/` excluded.
- **Custom `sqlclosecheck` — BESPOKE, not upstream.** `cmd/custom_lint/sqlclosecheck/sqlclosecheck.go`
  — a hand-written `go/analysis` SSA analyzer. Targets **`github.com/jackc/pgx/v4` `Rows`**
  (`:30 pgxPackage`, `:13-14`), reports (a) `pgx.Rows was not closed after query. Please add "defer
  rows.Close()"` (:74) and (b) `Close()` called without `defer` (:293). Entrypoint
  `cmd/custom_lint/main.go:10` (`singlechecker.Main`). NB its Doc string says "sql.Rows" but it checks
  *pgx* Rows. The upstream `sqlclosecheck` linter is **NOT** in golangci's list (only `bodyclose` +
  `rowserrcheck` cover DB).
- **It isn't actually enforced.** `make test-sqlclosecheck-lint` (`Makefile:115-120`) only vets
  `cmd/custom_lint/testdata/` and diffs — a self-test of the analyzer, not a scan of product code. The
  repo-wide CI invocation is **commented out** (`.github/workflows/tiered.pre_merge.yml:194,311,313`).
- **gitleaks:** `.gitleaks.toml` (`extend.useDefault`, allowlists, custom rules for GCP SA/API keys,
  Slack, AWS, FCM…) + `.gitleaksignore`. Pre-commit hook `check_gitleaks` → `gitleaks protect --staged
  --config=.gitleaks.toml --redact` (`.githooks/common.bash:54`) — but **silently skips if gitleaks
  isn't installed** (:44-48). Also CI (`tiered.pre_merge.yml:625` `gitleaks-scan`).
- **`.githooks/`** (installed by `make init` → `git config core.hooksPath .githooks`, `Makefile:17-25`):
  `commit-msg` requires 1–2 Jira IDs matching `\b(LT|RMP)-[0-9]{1,6}\b` (`common.bash:9`; skips merge/
  hotfix branches); `pre-commit` = gitleaks; `prepare-commit-msg` auto-prepends `[LT-XXX]` from branch.
  **No `pre-push`.**
- **Targets:** `make lint` = bare `golangci-lint run` (PATH version); CI pins `v2.8.0` (can disagree).
  `make lint-proto` = `buf check lint` (+ `check_breaking` against `develop`).
- **Myths:** custom sqlclosecheck doesn't scan product code (self-test only; CI commented); golangci
  doesn't enable sqlclosecheck; tests aren't linted; commit msgs need `LT-`/`RMP-` (not `JIRA-`);
  pre-commit skips silently w/o gitleaks; local vs CI golangci can disagree.

---

## 3. SECRETS / SOPS / KMS

- **`.sops.yaml`** — generated by `deployments/services-directory/hcl2sops.py`, **DO NOT EDIT** (:1).
  **317 `creation_rules`**, each `path_regex → gcp_kms`. Keys are per service/env/org — project
  `student-coach-e1e95`, keyRing `backend-services`, `cryptoKeys/prod-<service>` (partners use other
  keyrings).
- **Encrypted files:** `deployments/helm/backend/{service}/secrets/{org}/{env}/{service}.secrets.encrypted.yaml`
  (+ `{service}_migrate.secrets.encrypted.yaml`). SOPS encrypts **per value** (`ENC[AES256_GCM,data:…,
  iv:…,tag:…]`) — **keys/structure stay plaintext** — plus a trailing `sops:` block (`gcp_kms.resource_id`
  + encrypted DEK + `mac`). Other formats: `.encrypted.properties`, `.encrypted.env`.
- **Decryption — two mechanisms:**
  1. **Go services decrypt IN-PROCESS at boot.** `internal/golibs/configs/load.go:12` imports
     `getsops/sops/v3/decrypt`; `LoadAll` → passes `decrypt.File` (:19); `DecryptFile` (:109-117)
     SOPS-decrypts the `--secretsPath` and YAML-merges it over common+config (**secret = highest
     precedence** :16-17,58). KMS reached via `GOOGLE_APPLICATION_CREDENTIALS`. The container is handed
     the **encrypted** file (`docker-compose.migration.yaml` mounts it `:ro`) and decrypts in memory —
     **no plaintext on disk.** Used everywhere via `configs.MustLoadAll` (`cmd/server/main.go`).
  2. **Platform components (unleash/kafka/nats) use a SOPS sidecar** — `local/docker-compose.decrypt.yaml`
     runs `mozilla/sops:v3.7.3` → `local/decrypt.sh` writes plaintext into a shared volume.
- **App-level AES (second layer, independent of SOPS):** private keys stored in Postgres are
  AES-encrypted by the app — `internal/golibs/crypt/aes_gcm.go`, `aes.go`. The OpenAPI keypair repo
  `internal/usermgmt/modules/user/adapter/postgres/repository/domain_api_keypair.go:88` encrypts before
  insert (`crypt.AESEncrypt(privateKey, EncryptedKey, InitialVector)`), decrypts on read (:127,:170).
  Key material from config. (Ties to the OpenAPI HMAC keypair.)
- **Myths:** Go services aren't handed a decrypted file by a sidecar (they decrypt in-process); don't
  edit `.sops.yaml` (generated); SOPS encrypts values only; not all secrets are SOPS (DB AES layer);
  not one KMS key (317 rules).

---

## 4. BUILD SYSTEM / SKAFFOLD

- **One binary, all services.** `cmd/server/main.go` blank-imports every service (`:9-39`); each
  self-registers via bootstrap. Root cobra command `server` (:363) → `gserver` (gRPC) + `gjob` (jobs)
  subtrees (`internal/golibs/bootstrap/command.go`). The "service" is chosen at runtime by the cobra
  subcommand + which config/secrets are mounted — Kubernetes runs the **same image** many times with
  different args.
- **Cross-compile:** `local/build-code-golang.bash` → `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build`
  → `build/server` (hot-swapped into containers), `build/bdd.test` (gandalf), `build/stub`.
- **Skaffold:** ~30 `skaffold.*.yaml` at **repo root** (not `deployments/`). Aggregator
  `skaffold.local.yaml` (`skaffold/v4beta9`) `requires:` the rest. `skaffold.backend.yaml`:
  top level is only **`deploy.helm.releases`** (one per service, `chartPath: deployments/helm/backend/{service}`);
  **`build.artifacts` live inside `profiles`** — the `local` profile (:257) uses a **`custom.buildCommand`**
  (:300-321) that builds the 3 binaries, `docker build --file developments/development.Dockerfile
  --target developer`, tags `backend:locally`, `kind load`. `push:false`; fails if `PUSH_IMAGE=true`
  (:317-320). Profiles: `local`, `local-docker-compose`, `local-e2e`, `debug` (delve), … Launcher
  `deployments/sk.bash`.
- **Dockerfiles:** `developments/development.Dockerfile` target `developer` (debian:12.11, **COPIES**
  prebuilt `./build/server`, `ENTRYPOINT /server`) vs `developments/release.Dockerfile` target `runner`
  (alpine:3.21.7, **COMPILES** `/server` inside the build stage). They diverge. Makefile:
  `build-docker-runner` / `build-docker-dev`.
- **Two local paths:** Skaffold+kind+Helm (`deployments/sk.bash`) **vs** Docker-Compose
  (`./local/run.bash`, hot-swap binary). `./local/run.bash` does **not** use Skaffold.
- **Myths:** skaffold at repo root (not `deployments/`); not one Dockerfile (dev copies prebuilt/debian,
  prod compiles/alpine); `build/server` is linux-only; no top-level `build:` (inside profiles, custom
  cmd); `./local/run.bash` ≠ Skaffold; one binary/image, not one per service; local images can't push.

---

## 5. RELIABILITY — idempotency, retries, outbox

### 5a. Idempotency
- **`IdempotentService`** — `internal/notification/services/idempotent.go:17` (`:30` GetOne, `:39`
  Insert) over `IdempotentRepo`. Enum `internal/notification/consts/const.go:88` `IdempotentType`;
  only value `IdempotentTypeUpsertAsyncNotificationKafka = "upsert_async_notification_kafka"` (:91).
- **Table `idempotent` lives in the bob DB** — `migrations/bob/1720_migrate.up.sql:1` (PK
  `idempotent_id`). Built with `services.NewIdempotentService(bobDB, …)` (`cmd/server/notificationmgmt/consumers.go:80`).
- **Consumer usage** — `internal/notification/transports/kafka/consumer_upsert_async_notification.go:127-148`:
  `GetOne(id, type)` (:129); if exists → return `false` (drop) (:134-137); a `defer` **inserts the
  marker only on success or non-retryable failure** (`if err == nil || !retryable`, :139-147) — so a
  retryable failure leaves no marker, letting redelivery re-process.
- **NATS `Nats-Msg-Id`** dedup is separate — `internal/golibs/nats/jetstream.go:673` (JetStream
  server-side window). Both use `idutil.ULIDNow()` IDs.
- **Myth:** no generic golibs idempotency; the table is in **bob**, not notificationmgmt; spike has none.

### 5b. Retries / redelivery
- **Contract** `(retryable bool, err error)` — Kafka `internal/golibs/kafka/kafka.go:93`, NATS
  `internal/golibs/nats/jetstream.go:271` ("bool: whether this message should be retried" :270).
- **Retry helper** `internal/golibs/try/try.go` — `Do` (:24), `DoWithCtx` (:41), `DoBackOff` (:62);
  `maxRetries = 10` (:11). Same `(retry bool, err error)` shape reused for DB pool / NATS reconnect.
- **Kafka** turns `retryable=true` into an **in-process** retry first — `internal/golibs/kafka/consumer.go:110`
  (`try.DoWithCtx`), `:151-157` sleep+loop up to `retryLogicAttempts`. Only a **strict** consumer that
  errors leaves the offset uncommitted → broker redelivers (:232-242); **loose/AlwaysCommit** consumers
  commit regardless.
- **NATS** — `jetstream.go:719` `retry, err := cb(...)`; `:725-729` on error only Acks if `!retry`
  (retryable → no Ack → redeliver after `AckWait`, up to `MaxDeliver`). Knobs `:158 MaxDeliver`,
  `:181 AckWait`. Example `push_notification.go:36-39` (MaxDeliver 3, AckWait 90s).
- **Handlers:** spike `send_email_handler.go:46` (retryable except `sendgrid.ErrPayloadTooLarge` :110-112;
  post-send DB failure → `false` to avoid double-send :131-135). notification
  `consumer_upsert_async_notification.go` (:131 true, :136 false, :190 true). **Push failures swallowed**
  (`notification_modifier_push_notification.go:31-38` — non-retryable, to avoid re-pushing succeeded tokens).
- **Myth:** `retryable=true` is in-process first, not an immediate broker requeue; `try.Do` caps at 10.

### 5c. Transactional outbox
- **Real outbox exists — eureka v2 ONLY.** `internal/golibs/outboxevents/outbox_events.go:55`
  (`OutboxEvents` iface, :82 `NewOutboxEvents`, :105 `Insert` — doc :89-104 "Insert MUST be called
  inside the same tx as the business write … committed atomically … Debezium can never miss it").
  Table `migrations/eureka/1614_migrate.up.sql` (`outbox_events`, `aggregate_type`→topic). Relay =
  **Debezium CDC** (`deployments/docs/outbox_pattern.md`; connector
  `deployments/helm/backend/hephaestus/connectors/source/eureka_outbox.json` — the only `*_outbox.json`).
  Feature-flagged (`UnleashArchitectOutboxEvents`); caller
  `internal/eureka/v2/modules/assessment/usecase/complete_assessment_session.go:311,392,411`.
- **spike/notification do NOT use the outbox** — they **publish-after-commit (dual-write) with
  status-column compensation.** spike `create_email_handler.go:42` commits `emails`/`email_recipients`
  (status `CREATED`) then `email_modifier_send_email.go:47-49` publishes **after** commit; failure →
  `QUEUED_FAILED` (:59), success → `QUEUED` (:77). A crash between commit and publish is not auto-relayed
  — the caller retries (`ExternalBulkSendEmails` returns a per-email status map). Durability across the
  Kafka hop is provided by the **idempotent table**, not an outbox.
- **Myth:** outbox is NOT platform-wide (eureka-only + flagged); the relay is Debezium CDC, not a Go
  poller; spike/notification accept the dual-write risk on purpose.

---

## 6. NOTIFICATIONS & EMAIL DELIVERY

### 6a. Email (spike + SendGrid)
- **Pipeline:** `SendEmail` (`internal/spike/modules/email/controller/grpc/email_modifier_send_email.go:20`)
  → `CreateEmail` persists (status `CREATED`, `create_email_handler.go:27`) → `resolvePriority`/
  `TopicByPriority` (`priority.go:11,20` — `client_id`→ high vs normal Kafka topic; empty/unknown =
  normal) → `TracedPublishContext` (:49) → status `QUEUED`/`QUEUED_FAILED` (:59,:77).
- **Consumers:** `send_email_consumer.go:21 Consume()` (normal) + `:42 ConsumeHighPriority()` (high) —
  **both in the same pod**, own consumer groups, same `Handle`. → `send_email_handler.go:46 Handle`:
  load recipients, **two SendGrid clients** (`EmailTypeNotification` → `NotiSgClient` else `TransSgClient`,
  :80-86 — orthogonal to priority), build payload with `custom_args` (email_id/recipient_id → webhook
  correlation), `SendWithContext` (:107) → status `PROCESSED`/`PROCESSED_FAILED` + `sg_message_id`.
- **Webhook:** `email_http_service.go:45 EmailStatusReceiver` (gin) → `UpsertEmailEvent` → table
  `email_recipient_events`. Read API `email_reader_get_email_delivery_statuses.go`.
- **State machine:** `CREATED → QUEUED(/_FAILED) → PROCESSED(/_FAILED)` → per-recipient webhook events.
- **Client lib:** `internal/golibs/sendgrid/sendgrid.go` (`SendGridClient`, `SendWithContext`,
  `ErrPayloadTooLarge`); local → `NewSendGridMock()`. Config `cmd/server/spike/init_email_provider.go`.
- **Myth:** `EmailType`(client) ⟂ priority(topic); high-priority is not a separate pod (same pod, diff
  consumer group); spike has **no DB of its own** (uses notificationmgmt DB); `QUEUED` ≠ sent.

### 6b. Push (notification + FCM)
- **Two ingress → one handler:** NATS `push_notification.go:30 StartSubscribe` (`Notification.Created`,
  MaxDeliver 3, AckWait 90s) → `:50 HandleMessage` → `ProcessPushNotification` (:68); Kafka
  `consumer_upsert_async_notification.go:185-196` (`SendingMethodPushNotification`) → `processNotification`.
- **Domain:** `subscribers/notification_subscriber_push_nottification.go:14 ProcessPushNotification`
  → `services/notification_modifier_push_notification.go:13 pushNotificationToUsers` → `UserDeviceTokenRepo.FindByUserIDs`
  (:22) → `PushNotificationService.PushNotificationForUser` (`infra/push_notification_service.go:82`).
- **FCM wrapper in golibs:** `internal/golibs/firebase/messaging.go:36 NotificationPusher` (`:66 PushMultiple`
  chunks of `fcmBatchLimit=500`, `:155 Push`); SDK adapter `FCMClientV4` wraps
  `firebase.google.com/go/v4/messaging` (`SendEachForMulticast`). Error classes
  `internal/golibs/firebase/errors.go` (retryable 429/500/503/timeout/Unavailable vs unregistered).
- **Push service extras:** retries only the retryable failed-token subset (`push_notification_service.go:147-164`);
  **unregistered tokens** → published to `InvalidDeviceTokenTopic` for async cleanup (:274-306), counted
  as success. Push failures returned **non-retryable** to Kafka (avoid double-push).
- **The link:** conversationmgmt publishes — `.../conversation/infrastructure/nats/notification.go:77
  TracedPublish(…, Notification.Created)`; subject `internal/notification/transports/nats/consts.go:4`.
- **Cross-service:** notification **requests email via spike gRPC** — `notification_modifier_request_send_email.go:130
  svc.EmailModifierServiceClient.SendEmail(...)`; dial `consumers.go:69,74 rsc.GRPCDial("spike")`.
- **Myth:** notificationmgmt doesn't call FCM from the transport (golibs wrapper); unregistered ≠ failure;
  push failures aren't Kafka-retryable; mid NATS→Kafka migration (flag `Communication_Notification_Use_Kafka_Instead_Of_NATs`).

---

## 7. CRON / SCHEDULED-JOBS FRAMEWORK

- **Registration:** `internal/golibs/bootstrap/command.go:191 JobFunc[T]`, `:240 RegisterJob[T](name, fn)`
  — a cobra subcommand per job name, common flags, SIGTERM-aware ctx; returns a `JobBuilder` for opts.
  Each service calls `RegisterJob` in `init()`; the binary dispatches by the `name` arg.
- **notificationmgmt jobs:** `cronjob_send_scheduled_notifications.go:18` (dials itself, calls
  `SendScheduledNotification` RPC for a 1-min window), `cronjob_archive_notifications.go:16`,
  `cronjob_exec_scheduled_notification_bulk_job.go:24`, `trigger_upsert_system_notification.go:39`,
  plus one-off `migrate_notification_*`.
- **Scheduling = k8s CronJob from Helm, not a Go scheduler.** `deployments/helm/backend/notificationmgmt/values.yaml:58`
  `cronjobs:` with `cmd:` + `schedule: '*/1 * * * *'`. The `jobs:` block (:84-94) = **run-once** Jobs
  (migrations, trigger-upsert). Both dispatched by the same `RegisterJob` name. The Go entrypoint runs
  once and exits — no in-process cron loop.
- **gjob consumer pod (KEDA):** `cmd/server/notificationmgmt/consumers.go:25 RegisterJob("notificationmgmt_consumers",
  RunConsumers)` — long-running (blocks on `<-ctx.Done()` :51), serves `/metrics`+`/healthz`, runs only
  the async-notification consumer (:60). Separate Deployment (`templates/deployment-consumers.yaml`),
  autoscaled by KEDA `ScaledObject` (`templates/scaledobject-consumers.yaml`) on **Prometheus
  `kafka_consumergroup_lag`** (min 2 / max 3), NOT CPU.
- **Three runtime shapes, one binary:** (1) in-gRPC-pod consumers/subscribers (gserver.go); (2) the
  KEDA gjob consumer pod; (3) k8s CronJobs (thin triggers that call an RPC back into the gRPC service).
- **Local:** `./local/run.bash cron -n <name>` (run.bash:194) → `docker compose configs/cronjob/<name>.yaml`.
- **Myth:** no in-process Go cron scheduler (k8s CronJob schedule); cron/consumer/gRPC pod = same binary,
  different `RegisterJob` arg; KEDA scales on Kafka lag (not CPU); the scheduled-send cron just triggers
  an RPC; `jobs:`(run-once) vs `cronjobs:`(recurring).