diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 8585a41..4120f61 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -7,20 +7,29 @@ Document one place you used an LLM during this assignment. +While working on the Week 11 Streamlit dashboard, I got a PostgreSQL error when the app tried to query `dev_halyna.fct_trips`. -TODO +The error was: + +`relation "dev_halyna.fct_trips" does not exist` + +At first I thought the problem was in my `SELECT` statement, but I was not sure whether the issue came from the SQL query, the database schema, or the Streamlit `.env` configuration. ## The prompt -TODO +I asked the LLM: + +> My Streamlit app connects to Azure Postgres, but when it runs a query on `dev_halyna.fct_trips`, I get this error: `relation "dev_halyna.fct_trips" does not exist`. Is this a problem with the SELECT query, the schema, or the table? Please explain it simply. ## The response -TODO +The LLM explained that the SQL aggregation itself was not the main problem. The error meant that PostgreSQL could not find the table `fct_trips` inside the schema `dev_halyna`. + +It suggested checking whether the table exists in the database, whether the app is reading the correct `DB_SCHEMA` from the `.env` file, and whether the Week 10 mart table had been created in my own schema. ## Reflection @@ -28,7 +37,11 @@ TODO Be specific: "I kept the cache_data suggestion but changed ttl from 60 to 300 to match the mart's once-a-day refresh cadence." --> -TODO +After checking the response, I verified the database table and confirmed that the problem was related to the missing `fct_trips` table in my schema, not to the `COUNT`, `AVG`, or `GROUP BY` logic. + +I used the explanation to debug the issue step by step. I kept my Streamlit code close to the Week 11 material, using `run_query`, `@st.cache_data`, `st.metric`, `st.columns`, `st.line_chart`, and `st.sidebar.selectbox`. + +I did not paste any real database password, full connection string, or private credentials into the LLM. --- diff --git a/docs/week11_azure_recording_upload_halyna.png b/docs/week11_azure_recording_upload_halyna.png new file mode 100644 index 0000000..0e87884 Binary files /dev/null and b/docs/week11_azure_recording_upload_halyna.png differ diff --git a/docs/week11_metabase_dashboard_halyna.png b/docs/week11_metabase_dashboard_halyna.png new file mode 100644 index 0000000..d325516 Binary files /dev/null and b/docs/week11_metabase_dashboard_halyna.png differ diff --git a/docs/week11_streamlit_dashboard_halyna.png b/docs/week11_streamlit_dashboard_halyna.png new file mode 100644 index 0000000..cd1353c Binary files /dev/null and b/docs/week11_streamlit_dashboard_halyna.png differ diff --git a/metric_definitions.template.md b/metric_definitions.template.md deleted file mode 100644 index fc22c21..0000000 --- a/metric_definitions.template.md +++ /dev/null @@ -1,49 +0,0 @@ -# Metric definitions - -Five fields per metric: Name, Description, Calculation, Data source, Refresh frequency. One block per panel. Copy this file to `metric_definitions.md` inside your `week11-streamlit/` folder and fill it in. - -## Metabase panels - - - -### Panel 1: TODO name - -- **Name**: TODO -- **Description**: TODO -- **Calculation**: TODO -- **Data source**: TODO -- **Refresh frequency**: TODO - -### Panel 2: TODO name - -- **Name**: TODO -- **Description**: TODO -- **Calculation**: TODO -- **Data source**: TODO -- **Refresh frequency**: TODO - -### Panel 3: TODO name - -- **Name**: TODO -- **Description**: TODO -- **Calculation**: TODO -- **Data source**: TODO -- **Refresh frequency**: TODO - -## Streamlit panels - - - -### Panel 1: TODO name - -- **Name**: TODO -- **Description**: TODO -- **Calculation**: TODO -- **Data source**: TODO -- **Refresh frequency**: TODO - - diff --git a/week11-streamlit/.env.example b/week11-streamlit/.env.example index 9d5542e..7a8460c 100644 --- a/week11-streamlit/.env.example +++ b/week11-streamlit/.env.example @@ -1,4 +1,6 @@ # PostgreSQL connection string (your Week 9/10 login) -POSTGRES_URL=postgresql://pipeline_user:your-pg-password@your-pg-host:5432/team1 +# Keep the database name as team1 and the ?sslmode=require suffix; only swap +# in your own user, password, and host. +POSTGRES_URL=postgresql://your-pg-user:your-pg-password@your-pg-host:5432/team1?sslmode=require # Your dev schema name (e.g. dev_jana) -DB_SCHEMA=dev_yourname +DB_SCHEMA=dev_yourname \ No newline at end of file diff --git a/week11-streamlit/app.py b/week11-streamlit/app.py index cf06bcb..9826509 100644 --- a/week11-streamlit/app.py +++ b/week11-streamlit/app.py @@ -18,7 +18,7 @@ load_dotenv() # reads .env file if present POSTGRES_URL = os.environ["POSTGRES_URL"] -DB_SCHEMA = os.environ.get("DB_SCHEMA", "dev_yourname") +DB_SCHEMA = os.environ.get("DB_SCHEMA", "dev_halyna") st.set_page_config(page_title="NYC Taxi Metrics", layout="wide") st.title("NYC Taxi Metrics") @@ -26,20 +26,109 @@ @st.cache_data(ttl=300) def run_query(sql: str) -> pd.DataFrame: + """Run a SQL query against the Postgres database and return a DataFrame.""" engine = sqlalchemy.create_engine(POSTGRES_URL) with engine.connect() as conn: return pd.read_sql(sql, conn) +# Sidebar filter: payment type +# ----------------------------- + +st.sidebar.header("Filters") + +payment_types = run_query(f""" + SELECT DISTINCT payment_type_label + FROM {DB_SCHEMA}.fct_trips + WHERE payment_type_label IS NOT NULL + ORDER BY payment_type_label +""")["payment_type_label"].tolist() + +selected_payment_type = st.sidebar.selectbox( + "Payment type", + ["All"] + payment_types +) + +if selected_payment_type == "All": + WHERE_CLAUSE = "" +else: + WHERE_CLAUSE = f"WHERE payment_type_label = '{selected_payment_type}'" + + +# ----------------------------- +# Headline KPIs +# ----------------------------- + st.subheader("Headline KPIs") -# TODO: query total trip count, average trip_distance, and average -# fare_per_mile from {DB_SCHEMA}.fct_trips through run_query(), then -# render three tiles side by side with st.columns(3) and .metric(). -# This is deliberately not the total-trips/avg-fare/total-revenue trio -# from the chapter: trip_distance and fare_per_mile are different columns, -# so copying the chapter's SQL verbatim will not answer this. -raise NotImplementedError( - "TODO: implement the headline KPIs panel (total trips, avg trip " - "distance, avg fare per mile) from fct_trips." +kpis = run_query(f""" + SELECT + COUNT(*) AS total_trips, + AVG(trip_distance) AS avg_trip_distance, + AVG(fare_per_mile) AS avg_fare_per_mile + FROM {DB_SCHEMA}.fct_trips + {WHERE_CLAUSE} +""").iloc[0] + +col1, col2, col3 = st.columns(3) + +col1.metric( + "Total trips", + f"{int(kpis['total_trips']):,}" +) + +col2.metric( + "Average trip distance", + f"{kpis['avg_trip_distance']:.2f} miles" +) + +col3.metric( + "Average fare per mile", + f"${kpis['avg_fare_per_mile']:.2f}" +) + + +# ----------------------------- +# Hour-of-day trend +# ----------------------------- + +st.subheader("Trips by pickup hour") + +hourly = run_query(f""" + SELECT + EXTRACT(HOUR FROM pickup_datetime)::int AS pickup_hour, + COUNT(*) AS trip_count + FROM {DB_SCHEMA}.fct_trips + {WHERE_CLAUSE} + GROUP BY 1 + ORDER BY 1 +""") + +st.line_chart(hourly.set_index("pickup_hour")) + + +# ----------------------------- +# Data freshness +# ----------------------------- + +st.subheader("Data freshness") + +fresh = run_query(f""" + SELECT + COUNT(*) AS row_count, + MAX(pickup_datetime) AS last_pickup + FROM {DB_SCHEMA}.fct_trips + {WHERE_CLAUSE} +""").iloc[0] + +col1, col2 = st.columns(2) + +col1.metric( + "Row count", + f"{int(fresh['row_count']):,}" +) + +col2.metric( + "Last pickup", + str(fresh["last_pickup"])[:16] ) diff --git a/week11-streamlit/metric_definitions.md b/week11-streamlit/metric_definitions.md new file mode 100644 index 0000000..8d8fab2 --- /dev/null +++ b/week11-streamlit/metric_definitions.md @@ -0,0 +1,72 @@ +# Metric definitions + +Five fields per metric: Name, Description, Calculation, Data source, Refresh frequency. One block per panel. Copy this file to `metric_definitions.md` inside your `week11-streamlit/` folder and fill it in. + +## Metabase panels + + + +### Panel 1: Trip Count by Payment Type + +- **Name**: `trip_count_by_payment_type` +- **Description**: Number of taxi trips grouped by payment type. This shows which payment methods are used most often in the dataset. +- **Calculation**: `COUNT(*)` grouped by `payment_type_label` +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Updated when the `fct_trips` mart is rebuilt. Metabase reads the current database table when the dashboard is opened. + +### Panel 2: Average Fare per Mile by Dropoff Borough + +- **Name**: `avg_fare_per_mile_by_dropoff_borough` +- **Description**: Average fare charged per mile, grouped by the borough where the trip ended. Unit: US dollars per mile. +- **Calculation**: `AVG(fare_per_mile)` grouped by `dropoff_borough` +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Updated when the `fct_trips` mart is rebuilt. Metabase reads the current database table when the dashboard is opened. + +### Panel 3: Average Trip Duration by Hour + +- **Name**: `avg_trip_duration_by_pickup_hour` +- **Description**: Average trip duration in minutes, grouped by pickup hour of day. This shows how trip duration changes during the day. +- **Calculation**: `AVG(EXTRACT(EPOCH FROM (dropoff_datetime - pickup_datetime)) / 60)` grouped by `EXTRACT(HOUR FROM pickup_datetime)` +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Updated when the `fct_trips` mart is rebuilt. Metabase reads the current database table when the dashboard is opened. + +## Streamlit panels + + + +### Panel 1: Headline KPIs + +- **Name**: `headline_kpis` +- **Description**: Three headline metrics for the selected payment type: total trips, average trip distance, and average fare per mile. +- **Calculation**: + - `total_trips`: `COUNT(*)` + - `avg_trip_distance`: `AVG(trip_distance)` + - `avg_fare_per_mile`: `AVG(fare_per_mile)` + - If a payment type is selected, the query adds `WHERE payment_type_label = selected_payment_type`. +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Streamlit caches the query result for 300 seconds using `@st.cache_data(ttl=300)`. The underlying data updates when the `fct_trips` mart is rebuilt. + +### Panel 2: Trips by Pickup Hour + +- **Name**: `trip_count_by_pickup_hour` +- **Description**: Number of taxi trips grouped by pickup hour of day. This shows the daily demand pattern across the 24-hour cycle. +- **Calculation**: `COUNT(*)` grouped by `EXTRACT(HOUR FROM pickup_datetime)`. If a payment type is selected, the query adds `WHERE payment_type_label = selected_payment_type`. +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Streamlit caches the query result for 300 seconds using `@st.cache_data(ttl=300)`. The underlying data updates when the `fct_trips` mart is rebuilt. + +### Panel 3: Data Freshness + +- **Name**: `data_freshness` +- **Description**: Data-quality panel showing the number of rows available and the latest pickup timestamp in the selected dataset. +- **Calculation**: + - `row_count`: `COUNT(*)` + - `last_pickup`: `MAX(pickup_datetime)` + - If a payment type is selected, the query adds `WHERE payment_type_label = selected_payment_type`. +- **Data source**: `dev_halyna.fct_trips` +- **Refresh frequency**: Streamlit caches the query result for 300 seconds using `@st.cache_data(ttl=300)`. The underlying data updates when the `fct_trips` mart is rebuilt. + +