Identity Event Pipeline & Analytics Architecture

Event delivery, Databricks loading, and RAG-based KPI access

Project Overview

An event pipeline and analytics architecture that delivers identity events to multiple targets and loads them into Databricks for operational and analytical use.

Challenge

Identity events had to be delivered to multiple targets and loaded into Databricks for operational and analytical use, while also supporting natural-language KPI access through a Knowledge Base and RAG structure.

Solution

Event pipeline and multi-target delivery structure

Designed and built an event pipeline on SNS, SQS, and Kafka so identity events could be delivered to multiple targets.

Designed an SNS / SQS / Kafka-based pipeline

Established multi-target delivery including Adobe

Designed the Databricks loading flow

Structured the data flow for operational and analytical use

Knowledge Base and natural-language KPI access

Built a natural-language KPI access flow using document indexing, a Vector DB, and RAG routing.

Extracted and indexed documentation sources such as Confluence

Implemented automatic Vector DB updates

Handled natural-language questions through RAG routing

Implemented a chatbot for natural-language KPI queries

Tech Stack

SNS / SQS / Kafka

Event delivery pipeline

Databricks

Operational and analytical data loading

Adobe

Multi-target integration

Knowledge Base

Document extraction and indexing

Vector DB

Index and retrieval structure

RAG

Natural-language KPI access

Key Results

Designed and built an SNS / SQS / Kafka-based event pipeline and Databricks loading structure

Established a multi-target architecture including Adobe

Created a basis for self-service KPI access

Implemented a chatbot for natural-language questions

Learnings

Identity events need to be designed for both operational and analytical use

Self-service depends on designing the data-loading path and query interface together

Knowledge Base and Vector DB designs must include document update flows

RAG quality depends heavily on routing criteria and source quality

← Back to Projects Contact Me →