Author: Baibhav Gond
University: Indian Institute of Technology Bhubaneswar
Department: Department of Civil Engineering
Email: baibhav0019@gmail.com
Live Web Application: Process Mining & Automation
In the company's back-office departments (HR, Finance, Logistics), staff spend their days moving back and forth between internal web systems and desktop applications like Excel and Word, processing routine paperwork.
The company ran a desktop agent that recorded every keystroke, mouse click, and application switch in chronological order. However, these logs only captured raw actionsβnothing indicated when an expense claim or onboarding task actually started or finished. The logs were piling up untouched because nobody knew what tasks were being performed, how long they took, or where people were getting slowed down.
Management had one core request:
"Use these logs to tell us where automation would have the greatest impact on our operations. And show us something that actually works."
My goal for this task was to turn these raw event streams into distinct business processes, identify the biggest bottlenecks eating up staff time, and build a working automation prototype that delivers tangible ROI.
To tackle this, I built a modular Python pipeline:
- Data Ingestion & Cleaning: Handled multi-chunk session logs, enforced UTF-8 Japanese character decoding, and filtered out corrupted event records (such as malformed
text_input_completeevents perDATA_SCHEMA.md). - Process Segmentation Approaches:
- v1 Heuristic State Machine: Built a rule-based segmenter tracking entity clipboard transfers (
Ctrl+C/Ctrl+V) and portal navigation hubs. This produced the 279 clean segments submitted insegments.jsonl(Deliverable 1). - v2 Computer Vision PoC: Tested a lightweight
MobileNet_V3_Smallvision model across 34,563 screenshots using a 4-frame rolling window to catch silent UI changes that text logs miss. - v3 Supervised ML Pipeline: Trained a two-stage classifier (
HistGradientBoostingfor boundaries andTF-IDF + LogisticRegressionfor process labels) on Dataset A, reaching 81.4% Boundary F1 and 93.9% Label Consistency.
- v1 Heuristic State Machine: Built a rule-based segmenter tracking entity clipboard transfers (
- Process Mining (Dataset B): Mined the 15 production sessions across 4 workstations to rank processes by volume and manual friction (app switches and copy-paste frequency). Two tasks account for 57.7% of all back-office volume:
supplier_communication(Rank #1) andexpense_processing(Rank #2). - Step 3 Working Prototypes: Built deterministic, rule-based automation engines for both top bottlenecks with Japanese business correspondence dispatch, spending checks, and manager escalation for exceptions, accompanied by an interactive Streamlit dashboard.
Here is a quick summary of all deliverables submitted in this repository:
| Deliverable | File / Location | Description |
|---|---|---|
| Deliverable 1: Step 1 Output | segments.jsonl |
279 recovered units of work from Dataset B, formatted in JSONL with ISO-8601 UTC timestamps across all 15 sessions. |
| Deliverable 2: Code & History | Root Git Repository (master) |
Modular source code in src/, reproduction CLI scripts in scripts/, and 46 automated unit tests. |
| Deliverable 3: Final Report | final_report.md |
Complete proposal: Step 2 process inventory, ROI ranking, prototype design choices, human-in-the-loop residual work, and risk analysis. |
| Deliverable 4: Work Log | work_log.md |
Day-by-day engineering diary documenting what was tried each day, what worked, what failed, and GenAI disclosure. |
| Step 3 Primary Prototype | src/automation/supplier_automation.py |
Working deterministic automation engine for the Rank #1 bottleneck (supplier_communication) with Japanese email/EDI dispatch. |
| Step 3 Secondary Prototype | src/automation/expense_automation.py |
Accounting reimbursement engine for Rank #2 (expense_processing) with spending caps and receipt verification. |
| Interactive Dashboard | app.py |
Streamlit web application to visually explore the process mining results, telemetry data, and run the prototypes interactively. |
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β RAW WORKSTATION TELEMETRY β
β (Events: Keystrokes, Window Titles, Mouse Clicks, Active URLs, Clipboard, Screenshots) β
ββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 1: ROBUST INGESTION & QUIRK MITIGATION β
β (src/pipeline/loader.py) β
β β’ Multi-chunk session re-stitching β’ Strict UTF-8 Japanese character decoding β
β β’ Chronological sorting across chunks β’ Mitigation of buggy text_input_complete events β
ββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 2: THREE-TIER PROCESS SEGMENTATION ENGINE β
β (src/segmentation/) β
β β
β [Tier 1: v1 Heuristic State Machine] [Tier 2: v2 Vision Anomaly PoC] [Tier 3: v3 Supervised ML] β
β β’ Entity-Centric Golden Thread β’ MobileNet_V3_Small vectorizer β’ HistGradientBoosting β
β β’ Clipboard entity tracing β’ 4-frame relational rolling-win β’ TF-IDF + LogisticRegressionβ
β β’ /dashboard hub detection β’ Suppresses micro-jitter β’ 81.4% Boundary F1 β
β β’ Deliverable 1 Output (279 segs) β’ Captures silent DOM updates β’ 93.9% Label Consistency β
ββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 3: PROCESS MINING & AUTOMATION ROI PRIORITIZATION β
β (src/analytics/process_miner.py) β
β β’ Quantitative Metrics: Volume (N), Cumulative Time, Average Duration, Active Staff Count β
β β’ Operational Friction: F = Average App Switches + Average Clipboard Operations β
β β’ Prioritization Formula: ROI_Score = (Volume * Friction) / Average_Duration β
ββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 4: DETERMINISTIC STEP 3 AUTOMATION PROTOTYPES β
β (src/automation/) β
β β
β [Candidate #1: Supplier PO Engine] [Candidate #2: Expense Policy Engine] β
β β’ Automated variance checking (Qty/Price/Date) β’ Enforcement of corporate spending caps β
β β’ Japanese business email & EDI dispatch β’ Receipt compliance & transit verificationβ
β β’ Sub-second execution (<50ms) β’ Human-in-the-Loop manager escalation β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-
Core Design: Employs domain-invariant behavioral primitives:
-
Entity Anchor Tracing: Captures business entities (PO numbers, employee codes, Case IDs) upon
clipboard_changeand tracks them across application boundaries (Web Portal$\leftrightarrow$ Word$\leftrightarrow$ Excel). -
Topological Navigation: Detects returns to portal navigation hubs (
/dashboard,/index). - Inactivity Windows: Emits natural task boundaries during prolonged operator pauses (>60 seconds).
- Semantic Merging: Consolidates adjacent micro-fragments ($\le 30$s gap) sharing identical semantic labels without conflicting entity anchors.
-
Entity Anchor Tracing: Captures business entities (PO numbers, employee codes, Case IDs) upon
- Motivation: Text-based telemetry cannot observe "silent" UI state updates (e.g., asynchronous AJAX data table reloads, modal dialog popups, and tab switches without keystrokes).
-
Implementation (
src/experiments/vision_poc.py):- Compressed all 34,563 1080p desktop screenshots from Dataset A into 1,000-dimensional semantic vectors using
MobileNet_V3_Smallon Google Colab T4 GPUs. - Engineered a 4-frame relational rolling window (
$f_1, f_2, f_3, f_4$ ) to evaluate transition drop magnitude against surrounding visual stability:
- Compressed all 34,563 1080p desktop screenshots from Dataset A into 1,000-dimensional semantic vectors using
- Cut false-positive visual cuts by 38.1% (from 9,418 to 5,831 segments) and more than doubled Segment IoU F1 from 7.3% to 15.7%.
-
Stage 1: Temporal & Interaction Boundary Detector (
scripts/train_boundary_model.py):- Extracts 18 tabular temporal and interaction features per event (
dt_prev,dt_next,is_app_sw,is_clip,clip_delta,has_id,hub,url_depth,app_cat,idle_10s,idle_30s, etc.) across 162,650 event samples from Dataset A. - Trained a
HistGradientBoostingClassifierwith balanced class weights, achieving 0.9274 ROC-AUC and 0.7674 PR-AUC on validation sets.
- Extracts 18 tabular temporal and interaction features per event (
-
Stage 2: Calibrated Semantic Process Classifier (
scripts/train_label_classifier.py):- Tokenizes system port signatures (
SYS_HR_5122,SYS_FIN_5123,SYS_OPS_5124), route hashes, native window titles, form input labels, and OCR text across 1,734 ground truth executions. - Trained a calibrated
TF-IDF + LogisticRegressionpipeline, achieving 95.1% validation accuracy and 0.952 Macro F1 across all 15 business processes.
- Tokenizes system port signatures (
-
Inference Engine (
src/segmentation/ml_segmenter.py):- Adaptive Refractory Peak Suppression (12s): Suppresses micro-jitter cuts by retaining only the highest-probability boundary candidate within a 12-second rolling gap.
-
Idle Pause Pruning: Discards non-operational idle gaps (
$<10$ events over $>15$s). - Semantic Post-Processing & Merging: Merges adjacent split fragments sharing identical labels within a 35-second temporal window, unifying multi-app alt-tab loops into complete business processes.
- Graceful Heuristic Fallback: Automatically switches to the v1 state machine if ML model weights are absent.
-
Why Accuracy Jumped to 81.4% F1 & 93.9% Consistency:
-
Zero Cross-Portal Leakage: Tokenizing port IDs mathematically isolates HR (
:5122), Finance (:5123), and Ops (:5124), boosting label purity from 65.8% to 93.9%. - Jitter Suppression: 12s refractory peak suppression elevated Boundary Precision from 45.4% to 79.7% (+34.3%).
-
End-to-End Overlap: 35s semantic merging boosted Segment IoU F1 (
$\ge 0.5$ ) from 55.5% to 77.3% (+21.8%) with 99.0% volume fidelity (1,989 predicted vs. 2,009 true executions).
-
Zero Cross-Portal Leakage: Tokenizing port IDs mathematically isolates HR (
- Cross-Department Distributional Shift: Telemetry in Dataset A was generated by specific operators (
Marcos,yuvraj,R36BQBTE,JAYESH) working in Google Chrome on portal ports5122β5124. Dataset B introduces unseen operators (CHAITANYA0BCF,LAPTOP-76QMG9DE,NEELA9BAF), operating in Microsoft Edge (Profile 1) across new departments and portal ports5132β5134. - Overfitting Immunity: An aggressive supervised model trained exclusively on Dataset A is prone to memorizing operator keystroke cadences and portal ports. The v1 heuristic relies strictly on universal human work patterns, ensuring robust out-of-domain transfer.
- Empirical Ground-Truth Validation: Yielded 279 segments with an average duration of 35.8 secondsβan almost exact mirror of Dataset A's verified ground truth (37.1 seconds).
Evaluating predictions against the ground-truth execution manifests of all 63 sessions in Dataset A using scripts/evaluate_dataset_a.py reveals the full engineering progression:
| Metric | Raw Heuristic Baseline | v1 Golden Thread (LLM + Merging) | v2 Vision Anomaly PoC | v3 Two-Stage Supervised ML | Net Delta (v3 vs. Raw) |
|---|---|---|---|---|---|
| Total Predicted Segments | 2,456 | 2,010 (True: 2,009) | 5,831 | 1,989 (True: 2,009) | -467 (Resolved Over-segmentation) |
| Boundary Precision | 41.5% | 45.4% | 13.9% | 79.7% | +38.2% |
| Boundary Recall | 53.5% | 48.1% | 43.8% | 83.9% | +30.4% |
| Boundary F1-Score | 46.5% | 46.4% | 20.9% | 81.4% | +34.9% |
| Segment Precision | 49.1% | 52.4% | 10.4% | 73.1% | +24.0% |
| Segment Recall | 56.2% | 59.6% | 33.8% | 82.7% | +26.5% |
| Segment IoU F1 ( |
34.2% | 55.5% | 15.7% | 77.3% | +43.1% |
| Label Consistency (Purity) | 9.2% | 65.8% | 16.8% | 93.9% | +84.7% |
| Process Category | Code | Classification Purity | Matched Samples | Operational Domain |
|---|---|---|---|---|
inventory_adjustment |
L |
100.0% | 75 / 75 | Supply Chain & Warehousing |
resident_tax_verification |
A |
99.1% | 116 / 117 | Human Resources & Payroll |
invoice_approval |
F |
99.1% | 113 / 114 | Financial Accounting |
return_processing |
O |
98.7% | 78 / 79 | Supply Chain Logistics |
payment_processing |
J |
98.6% | 70 / 71 | Treasury & Disbursements |
supplier_communication |
M |
98.4% | 122 / 124 | Procurement & Sourcing |
budget_variance_analysis |
I |
97.8% | 91 / 93 | Financial FP&A |
onboarding_verification |
E |
97.4% | 74 / 76 | Human Resources |
leave_application_processing |
C |
97.3% | 107 / 110 | Human Resources |
expense_processing |
G |
96.6% | 86 / 89 | General Accounting |
shipment_tracking |
N |
92.5% | 86 / 93 | Outbound Logistics |
bank_reconciliation |
H |
86.3% | 101 / 117 | Financial Accounting |
order_processing |
K |
85.6% | 77 / 90 | Customer Sales Operations |
payroll_adjustment |
B |
81.9% | 86 / 105 | Human Resources & Payroll |
insurance_pension_processing |
D |
79.6% | 78 / 98 | Statutory Compliance |
| Macro Average Accuracy | β | 93.9% | 1,245 / 1,326 | All Enterprise Domains |
Running the process mining script (src/analytics/process_miner.py) on the 15 production sessions in Dataset B gives the following process breakdown:
| Rank | Business Process | Volume ( |
Total Time (min) | Avg Dur (s) | App Switches | Clip Ops | Friction | Sessions | Staff | ROI Score |
|---|---|---|---|---|---|---|---|---|---|---|
| #1 | supplier_communication |
100 | 61.9 | 37.1s | 6.8 | 2.6 | 9.39 | 14 / 15 | 4 / 4 | 25.30 |
| #2 | expense_processing |
61 | 35.0 | 34.4s | 6.3 | 3.2 | 9.44 | 14 / 15 | 4 / 4 | 16.75 |
| #3 | onboarding_verification |
21 | 11.8 | 33.7s | 6.4 | 2.7 | 9.10 | 9 / 15 | 4 / 4 | 5.67 |
| #4 | leave_application_processing |
26 | 18.0 | 41.6s | 4.9 | 3.8 | 8.73 | 11 / 15 | 4 / 4 | 5.45 |
| #5 | inventory_adjustment |
25 | 15.8 | 38.0s | 3.4 | 4.7 | 8.12 | 11 / 15 | 4 / 4 | 5.34 |
| #6 | payroll_adjustment |
16 | 7.9 | 29.7s | 4.6 | 2.9 | 7.50 | 8 / 15 | 3 / 4 | 4.04 |
| #7 | invoice_approval |
13 | 5.5 | 25.3s | 2.0 | 2.3 | 4.31 | 8 / 15 | 4 / 4 | 2.21 |
| #8 | return_processing |
2 | 0.4 | 12.5s | 9.0 | 0.0 | 9.00 | 2 / 15 | 1 / 4 | 1.44 |
| #9 | resident_tax_verification |
6 | 2.7 | 27.0s | 2.7 | 2.3 | 5.00 | 4 / 15 | 3 / 4 | 1.11 |
| #10 | budget_variance_analysis |
4 | 3.2 | 48.2s | 6.8 | 3.0 | 9.75 | 2 / 15 | 2 / 4 | 0.81 |
| #11 | shipment_tracking |
2 | 1.5 | 44.0s | 2.0 | 5.5 | 7.50 | 2 / 15 | 2 / 4 | 0.34 |
| #12 | payment_processing |
1 | 0.4 | 22.0s | 0.0 | 3.0 | 3.00 | 1 / 15 | 1 / 4 | 0.14 |
| β | Total across all sessions | 279 | 166.3 min | 35.8s avg | 5.9 avg | 3.1 avg | 9.00 avg | 15 sessions | 4 staff | β |
Key Takeaway:
supplier_communicationandexpense_processingaccount for 57.7% of all operational volume (161 of 279 executions) and involve heavy context switching (averaging over 9 app switches and clipboard actions per task). Automatingsupplier_communicationalone saves over 410 hours of manual labor per year across the team.
-
Source:
src/automation/supplier_automation.py -
Target: Rank #1 Bottleneck (
supplier_communication) -
Logic & Policy Rules:
-
Quantity Variance:
$\le 25%$ change$\to$ AUTO_APPROVED -
Price Variance:
$\le 5%$ change$\to$ AUTO_APPROVED -
Delivery Schedule Shift:
$\le 5$ days$\to$ AUTO_APPROVED -
Non-Standard Exceptions: Breaching any threshold escalates immediately to
ESCALATED_TO_MANAGERwith operational rationale.
-
Quantity Variance:
- Output Payload: Generates complete business correspondence (supplier requests, confirm comments, change histories).
-
Source:
src/automation/expense_automation.py -
Target: Rank #2 Bottleneck (
expense_processing) -
Logic & Accounting Rules:
-
Entertainment Expenses:
$\le Β₯10,000$ per attendee$\to$ AUTO_APPROVED -
Domestic Transit:
$\le Β₯30,000$ per claim$\to$ AUTO_APPROVED -
Office Supplies:
$\le Β₯50,000$ per claim$\to$ AUTO_APPROVED - Compliance Guardrail: Missing receipts trigger mandatory escalation to Finance Directors.
-
Entertainment Expenses:
- Live Web Application: I'm beside you | FDE Process Mining & Automation
- Source:
app.py - Framework: Streamlit web application
- Features:
- Visual summary cards and process ranking charts.
- Interactive event trace explorer.
- Live simulation of the automation prototypes with rule checks and message generation.
c:/IBY_Japan/
βββ .gitignore # Clean exclusions (pycache, temporary files)
βββ README.md # Master architectural & usage guide
βββ DATA_SCHEMA.md # Telemetry specifications & recording quirk notes
βββ final_report.md # Deliverable 3: Executive proposal & strategy
βββ work_log.md # Deliverable 4: 7-day chronological diary + GenAI disclosure
βββ segments.jsonl # Deliverable 1: 279 production segments (Dataset B)
βββ process_mining_results.json # Step 2: Mined metrics & ROI rankings
βββ app.py # Live Streamlit dashboard (Procurement & Finance)
β
βββ experiments/ # Computer Vision Experimentation & Colab Pipeline
β βββ README.md # Vision anomaly detection documentation & methodology
β βββ vision_boundary.ipynb # Full executed Colab notebook on T4 GPU (34,563 screenshots)
β βββ colab_notebook.ipynb # Standalone portable Colab template notebook
β βββ vision_boundaries_multiframe.jsonl # Raw 4-frame relational anomaly detection boundaries
β βββ evaluated_segments.jsonl # Converted ISO-8601 session segments (5,831 segments)
β βββ convert_vision_boundaries.py # Boundary-to-segment conversion utility script
β
βββ src/ # Core Modular Library
β βββ pipeline/
β β βββ __init__.py # Clean package exports
β β βββ loader.py # Ingestion, chunk stitching, UTF-8 normalization
β βββ segmentation/
β β βββ __init__.py # Segmenter exports
β β βββ segmenter.py # Golden Thread v1 heuristic state machine
β β βββ llm_labeler.py # LLM prompt generator & semantic regex dictionary
β β βββ ml_segmenter.py # Two-stage supervised ML pipeline (81.4% F1)
β β βββ boundary_model.pkl # Trained HistGradientBoosting boundary model
β β βββ label_model.pkl # Trained TF-IDF + LogisticRegression label model
β βββ analytics/
β β βββ __init__.py # Analytics package exports
β β βββ process_miner.py # Process mining & ROI prioritization engine
β βββ automation/
β β βββ __init__.py # Automation exports
β β βββ supplier_automation.py # Step 3 Candidate #1 prototype (Procurement)
β β βββ expense_automation.py # Step 3 Candidate #2 prototype (Finance)
β βββ experiments/
β βββ vision_poc.py # MobileNet visual cosine anomaly detector
β
βββ scripts/ # Reproduction and utility scripts
β βββ run_segmentation.py # Dual-mode runner (--mode [heuristic|ml])
β βββ evaluate_dataset_a.py # Ground-truth evaluation harness (F1, IoU, Purity)
β βββ mine_dataset_b.py # Dataset B process mining CLI
β βββ train_boundary_model.py # Boundary model training script (HistGradientBoosting)
β βββ train_label_classifier.py # Label classifier training script (TF-IDF + LogReg)
β βββ convert_vision_boundaries.py # Vision boundary to segment converter
β βββ eda_dataset_a.py # Exploratory telemetry analysis
β
βββ tests/ # Complete Automated Test Suite (46 tests)
β βββ test_loader.py # Ingestion & quirk handling tests (3 tests)
β βββ test_segmenter.py # State machine boundary detection tests (9 tests)
β βββ test_evaluator.py # Boundary F1 & IoU evaluation tests (19 tests)
β βββ test_ml_segmenter.py # Machine learning pipeline tests (5 tests)
β βββ test_automation.py # Supplier workflow & approval tests (5 tests)
β βββ test_expense_automation.py # Expense workflow & compliance tests (5 tests)
β
βββ dataset_a/ # Ground Truth Benchmark (63 sessions)
βββ evaluated_segments_baseline.jsonl # v1 Heuristic predictions (2,010 segments)
βββ evaluated_segments_ml.jsonl # v3 Two-Stage ML predictions (1,989 segments)
βββ evaluated_segments_multiframe.jsonl # Multi-frame vision evaluated segments
βββ evaluated_segments_single.jsonl # Single-frame vision evaluated segments
βββ vision_boundaries_multiframe.jsonl # Multi-frame vision raw boundary detections
π Live Cloud Demo: Process Mining & Automation
Access the fully interactive Streamlit analytics platform directly in the browser without local setup.
Requirements: Python 3.10+ with dependencies installed (pip install -r requirements.txt).
Verify environment setup, data ingestion, segmentation logic, and prototype engines:
python -m pytest -vRuns all 46 unit and integration tests covering data loading, segmentation, evaluation, and automation prototypes.
Process raw workstation telemetry across all 15 sessions, extract the 279 task segments (segments.jsonl), and compute process ROI rankings:
python scripts/mine_dataset_b.pyRuns segmentation on Dataset B to generate segments.jsonl (Deliverable 1) and recalculate process rankings in process_mining_results.json.
Run the deterministic PO change approval engine against sample test cases (auto-approval vs. manager escalation):
python src/automation/supplier_automation.py --demoProcesses sample purchase order change requests to demonstrate automatic rule approvals and manager escalation.
Explore mined processes, inspect telemetry event traces, and test both automation prototypes in the browser:
python -m streamlit run app.pyOpens the web dashboard to visually inspect mined processes, view event traces, and test prototypes.
Evaluate segmentation predictions against Dataset A ground truth (63 sessions, 2,009 verified executions):
# Baseline heuristic state machine
python scripts/evaluate_dataset_a.py --predictions dataset_a/evaluated_segments_baseline.jsonl --dataset-dir dataset_a
# Two-stage supervised ML pipeline
python scripts/evaluate_dataset_a.py --predictions dataset_a/evaluated_segments_ml.jsonl --dataset-dir dataset_aRe-train the boundary detector and semantic process classifier from Dataset A features:
python scripts/train_boundary_model.py --dataset-dir dataset_a
python scripts/train_label_classifier.py --dataset-dir dataset_a