diff --git a/02_activities/assignments/Microcredential_Cohort/Assignment2.md b/02_activities/assignments/Microcredential_Cohort/Assignment2.md index d91d3c9d3..f4f52693c 100644 --- a/02_activities/assignments/Microcredential_Cohort/Assignment2.md +++ b/02_activities/assignments/Microcredential_Cohort/Assignment2.md @@ -56,7 +56,8 @@ The store wants to keep customer addresses. Propose two architectures for the CU **HINT:** search type 1 vs type 2 slowly changing dimensions. ``` -Your answer... +Slowly changing dimensions describe how changes in attributes such as a customer’s address are handled over time. Type 1 overwrites the existing value, meaning that only the most recent address is stored and all historical information is lost. This approach is simple and efficient but does not allow any analysis of past data (in this case addresses). In contrast, Type 2 preserves history by creating a new record (row) each time an address changes, along with fields that indicate when each version was valid. This allows the system to retain both current and past addresses for customers. +If ths goal is for a bookstore to promotional mail and link addresses to purchase patterns for analysis/insights, Type 2 is generally the better option because it allows the store to retrieve the current address for accurate delivery while also maintaining a full history of where materials were sent in the past. This supports auditing, campaign analysis, and future marketing insights, whereas Type 1 would only provide the latest address and lose all historical context. However, Type 2 would require more data storate and management needs since the addresses are constantly updating and older information is retained, so it could be more costly though it is more beneficial. ``` *** diff --git a/02_activities/assignments/Microcredential_Cohort/Assignment2_complete.md b/02_activities/assignments/Microcredential_Cohort/Assignment2_complete.md new file mode 100644 index 000000000..8ed3b488c --- /dev/null +++ b/02_activities/assignments/Microcredential_Cohort/Assignment2_complete.md @@ -0,0 +1,179 @@ +# Microcredential Assignment 2: Design a Logical Model and Advanced SQL + +🚨 **Please review our [Assignment Submission Guide](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md)** 🚨 for detailed instructions on how to format, branch, and submit your work. Following these guidelines is crucial for your submissions to be evaluated correctly. + +#### Submission Parameters: +* Submission Due Date: `May 11, 2026` +* Weight: 70% of total grade +* The branch name for your repo should be: `assignment-two` +* What to submit for this assignment: + * This markdown (Assignment2.md) with written responses in Section 1 + * Two Entity-Relationship Diagrams (preferably in a pdf, jpeg, png format). + * One .sql file +* What the pull request link should look like for this assignment: `https://github.com//sql/pulls/` + * Open a private window in your browser. Copy and paste the link to your pull request into the address bar. Make sure you can see your pull request properly. This helps the technical facilitator and learning support staff review your submission easily. + +Checklist: +- [ ] Create a branch called `assignment-two`. +- [ ] Ensure that the repository is public. +- [ ] Review [the PR description guidelines](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md#guidelines-for-pull-request-descriptions) and adhere to them. +- [ ] Verify that the link is accessible in a private browser window. + +If you encounter any difficulties or have questions, please don't hesitate to reach out to our team via our Slack. Our Technical Facilitators and Learning Support staff are here to help you navigate any challenges. + +*** + +## Section 1: +You can start this section following *session 1*, but you may want to wait until you feel comfortable wtih basic SQL query writing. + +Steps to complete this part of the assignment: +- Design a logical data model +- Duplicate the logical data model and add another table to it following the instructions +- Write, within this markdown file, an answer to Prompt 3 + + +### Design a Logical Model + +#### Prompt 1 +Design a logical model for a small bookstore. 📚 + +At the minimum it should have employee, order, sales, customer, and book entities (tables). Determine sensible column and table design based on what you know about these concepts. Keep it simple, but work out sensible relationships to keep tables reasonably sized. + +Additionally, include a date table. +A date table (also called a calendar table) is a permanent table containing a list of dates and various components of those dates. Some theory, tips, and commentary can be found [here](https://www.sqlshack.com/designing-a-calendar-table/), [here](https://www.mssqltips.com/sqlservertip/4054/creating-a-date-dimension-or-calendar-table-in-sql-server/) and [here](https://sqlgeekspro.com/creating-calendar-table-sql-server/). +Remember, you don't actually need to run any of the queries in these articles, but instead understand *why* date tables in SQL make sense, and how to situate them within your logical models. + +There are several tools online you can use, I'd recommend [Draw.io](https://www.drawio.com/) or [LucidChart](https://www.lucidchart.com/pages/). + +**HINT:** You do not need to create any data for this prompt. This is a conceptual model only. + +#### Prompt 2 +We want to create employee shifts, splitting up the day into morning and evening. Add this to the ERD. + +#### Prompt 3 +The store wants to keep customer addresses. Propose two architectures for the CUSTOMER_ADDRESS table, one that will retain changes, and another that will overwrite. Which is type 1, which is type 2? + +**HINT:** search type 1 vs type 2 slowly changing dimensions. + +## Answer + +Slowly changing dimensions describe how changes in attributes such as a customer’s address are handled over time. Type 1 overwrites the existing value, meaning that only the most recent address is stored and all historical information is lost. This approach is simple and efficient but does not allow any analysis of past data (in this case addresses). In contrast, Type 2 preserves history by creating a new record (row) each time an address changes, along with fields that indicate when each version was valid. This allows the system to retain both current and past addresses for customers. +If ths goal is for a bookstore to promotional mail and link addresses to purchase patterns for analysis/insights, Type 2 is generally the better option because it allows the store to retrieve the current address for accurate delivery while also maintaining a full history of where materials were sent in the past. This supports auditing, campaign analysis, and future marketing insights, whereas Type 1 would only provide the latest address and lose all historical context. However, Type 2 would require more data storate and management needs since the addresses are constantly updating and older information is retained, so it could be more costly though it is more beneficial. + +``` +Slowly changing dimensions describe how changes in attributes such as a customer’s address are handled over time. Type 1 overwrites the existing value, meaning that only the most recent address is stored and all historical information is lost. This approach is simple and efficient but does not allow any analysis of past data (in this case addresses). In contrast, Type 2 preserves history by creating a new record (row) each time an address changes, along with fields that indicate when each version was valid. This allows the system to retain both current and past addresses for customers. +If ths goal is for a bookstore to promotional mail and link addresses to purchase patterns for analysis/insights, Type 2 is generally the better option because it allows the store to retrieve the current address for accurate delivery while also maintaining a full history of where materials were sent in the past. This supports auditing, campaign analysis, and future marketing insights, whereas Type 1 would only provide the latest address and lose all historical context. However, Type 2 would require more data storate and management needs since the addresses are constantly updating and older information is retained, so it could be more costly though it is more beneficial. +``` + +*** + +## Section 2: +You can start this section following *session 4*. + +Steps to complete this part of the assignment: +- Open the assignment2.sql file in DB Browser for SQLite: + - from [Github](./02_activities/assignments/assignment2.sql) + - or, from your local forked repository +- Complete each question, by writing responses between the QUERY # and END QUERY blocks + + +### Write SQL + +#### COALESCE +1. Our favourite manager wants a detailed long list of products, but is afraid of tables! We tell them, no problem! We can produce a list with all of the appropriate details. + +Using the following syntax you create our super cool and not at all needy manager a list: +``` +SELECT +product_name || ', ' || product_size|| ' (' || product_qty_type || ')' +FROM product +``` + +But wait! The product table has some bad data (a few NULL values). +Find the NULLs and then using COALESCE, replace the NULL with a blank for the first column with nulls, and 'unit' for the second column with nulls. + +**HINT**: keep the syntax the same, but edited the correct components with the string. The `||` values concatenate the columns into strings. Edit the appropriate columns -- you're making two edits -- and the NULL rows will be fixed. All the other rows will remain the same. + +
-
+ +#### Windowed Functions +1. Write a query that selects from the customer_purchases table and numbers each customer’s visits to the farmer’s market (labeling each market date with a different number). Each customer’s first visit is labeled 1, second visit is labeled 2, etc. + +You can either display all rows in the customer_purchases table, with the counter changing on each new market date for each customer, or select only the unique market dates per customer (without purchase details) and number those visits. + +**HINT**: One of these approaches uses ROW_NUMBER() and one uses DENSE_RANK(). + +Filter the visits to dates before April 29, 2022. + +2. Reverse the numbering of the query so each customer’s most recent visit is labeled 1, then write another query that uses this one as a subquery (or temp table) and filters the results to only the customer’s most recent visit. +**HINT**: Do not use the previous visit dates filter. + +3. Using a COUNT() window function, include a value along with each row of the customer_purchases table that indicates how many different times that customer has purchased that product_id. + +You can make this a running count by including an ORDER BY within the PARTITION BY if desired. +Filter the visits to dates before April 29, 2022. + +
-
+ +#### String manipulations +1. Some product names in the product table have descriptions like "Jar" or "Organic". These are separated from the product name with a hyphen. Create a column using SUBSTR (and a couple of other commands) that captures these, but is otherwise NULL. Remove any trailing or leading whitespaces. Don't just use a case statement for each product! + +| product_name | description | +|----------------------------|-------------| +| Habanero Peppers - Organic | Organic | + +**HINT**: you might need to use INSTR(product_name,'-') to find the hyphens. INSTR will help split the column. + +2. Filter the query to show any product_size value that contain a number with REGEXP. + +
-
+ +#### UNION +1. Using a UNION, write a query that displays the market dates with the highest and lowest total sales. + +**HINT**: There are a possibly a few ways to do this query, but if you're struggling, try the following: 1) Create a CTE/Temp Table to find sales values grouped dates; 2) Create another CTE/Temp table with a rank windowed function on the previous query to create "best day" and "worst day"; 3) Query the second temp table twice, once for the best day, once for the worst day, with a UNION binding them. + +*** + +## Section 3: +You can start this section following *session 5*. + +Steps to complete this part of the assignment: +- Open the assignment2.sql file in DB Browser for SQLite: + - from [Github](./02_activities/assignments/assignment2.sql) + - or, from your local forked repository +- Complete each question, by writing responses between the QUERY # and END QUERY blocks + +### Write SQL + +#### Cross Join +1. Suppose every vendor in the `vendor_inventory` table had 5 of each of their products to sell to **every** customer on record. How much money would each vendor make per product? Show this by vendor_name and product name, rather than using the IDs. + +**HINT**: Be sure you select only relevant columns and rows. Remember, CROSS JOIN will explode your table rows, so CROSS JOIN should likely be a subquery. Think a bit about the row counts: how many distinct vendors, product names are there (x)? How many customers are there (y). Before your final group by you should have the product of those two queries (x\*y). + +
-
+ +#### INSERT +1. Create a new table "product_units". This table will contain only products where the `product_qty_type = 'unit'`. It should use all of the columns from the product table, as well as a new column for the `CURRENT_TIMESTAMP`. Name the timestamp column `snapshot_timestamp`. + +2. Using `INSERT`, add a new row to the product_unit table (with an updated timestamp). This can be any product you desire (e.g. add another record for Apple Pie). + +
-
+ +#### DELETE +1. Delete the older record for whatever product you added. + +**HINT**: If you don't specify a WHERE clause, [you are going to have a bad time](https://imgflip.com/i/8iq872). + +
-
+ +#### UPDATE +1. We want to add the current_quantity to the product_units table. First, add a new column, `current_quantity` to the table using the following syntax. +``` +ALTER TABLE product_units +ADD current_quantity INT; +``` + +Then, using `UPDATE`, change the current_quantity equal to the **last** `quantity` value from the vendor_inventory details. + +**HINT**: This one is pretty hard. First, determine how to get the "last" quantity per product. Second, coalesce null values to 0 (if you don't have null values, figure out how to rearrange your query so you do.) Third, `SET current_quantity = (...your select statement...)`, remembering that WHERE can only accommodate one column. Finally, make sure you have a WHERE statement to update the right row, you'll need to use `product_units.product_id` to refer to the correct row within the product_units table. When you have all of these components, you can run the update statement. diff --git a/02_activities/assignments/Microcredential_Cohort/Assignment2prompt1.png b/02_activities/assignments/Microcredential_Cohort/Assignment2prompt1.png new file mode 100644 index 000000000..bfb9355d3 Binary files /dev/null and b/02_activities/assignments/Microcredential_Cohort/Assignment2prompt1.png differ diff --git a/02_activities/assignments/Microcredential_Cohort/Assignment2prompt2.png b/02_activities/assignments/Microcredential_Cohort/Assignment2prompt2.png new file mode 100644 index 000000000..0d7d8b002 Binary files /dev/null and b/02_activities/assignments/Microcredential_Cohort/Assignment2prompt2.png differ diff --git a/02_activities/assignments/Microcredential_Cohort/assignment2.sql b/02_activities/assignments/Microcredential_Cohort/assignment2.sql index 4079c18ae..cc6f655a7 100644 --- a/02_activities/assignments/Microcredential_Cohort/assignment2.sql +++ b/02_activities/assignments/Microcredential_Cohort/assignment2.sql @@ -23,7 +23,12 @@ Edit the appropriate columns -- you're making two edits -- and the NULL rows wil All the other rows will remain the same. */ --QUERY 1 - +SELECT + NULLIF(product_size,'') as product_size, + COALESCE(product_name, '') || ', ' || + COALESCE(product_size, '') || ' (' || + COALESCE(product_qty_type, 'unit') || ')' +FROM product; --END QUERY @@ -41,6 +46,17 @@ HINT: One of these approaches uses ROW_NUMBER() and one uses DENSE_RANK(). Filter the visits to dates before April 29, 2022. */ --QUERY 2 +-- version where wach visit is unique, even if occuring on same day +SELECT customer_id, +ROW_NUMBER() OVER(PARTITION BY customer_id ORDER BY market_date ASC) as Number_visits +FROM customer_purchases +WHERE market_date < '2022-04-22'; + +-- version where there are unique market dates ONLY (same date of market visit only counts once) +SELECT customer_id +,DENSE_RANK() OVER(PARTITION BY customer_id ORDER BY market_date ASC) as Number_visits +FROM customer_purchases +WHERE market_date < '2022-04-22'; @@ -53,8 +69,27 @@ only the customer’s most recent visit. HINT: Do not use the previous visit dates filter. */ --QUERY 3 +-- version using temp table +DROP TABLE IF EXISTS temp.recent_visit; + +CREATE TABLE temp.recent_visit AS + SELECT customer_id, market_date, + ROW_NUMBER() OVER(PARTITION BY customer_id ORDER BY market_date DESC) as Number_visits +FROM customer_purchases; + +SELECT customer_id, market_date + FROM temp.recent_visit +WHERE Number_visits=1; +-- version using subquery +SELECT customer_id, market_date +FROM ( +SELECT customer_id, market_date, +ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY market_date DESC) AS Number_visits +FROM customer_purchases +) +WHERE Number_visits = 1; --END QUERY @@ -66,9 +101,20 @@ You can make this a running count by including an ORDER BY within the PARTITION Filter the visits to dates before April 29, 2022. */ --QUERY 4 +-- version where wach visit is unique, even if occuring on same day +SELECT customer_id, product_id, market_date, +COUNT(*) OVER (PARTITION BY customer_id, product_id) AS product_purchase_number +FROM customer_purchases +WHERE market_date < '2022-04-29'; +-- Running count versionw with each new day as a new count added to the perevious one +SELECT customer_id, product_id, market_date, +COUNT(*) OVER (PARTITION BY customer_id, product_id ORDER BY market_date) AS running_product_count +FROM customer_purchases +WHERE market_date < '2022-04-29'; + --END QUERY @@ -85,17 +131,21 @@ Remove any trailing or leading whitespaces. Don't just use a case statement for Hint: you might need to use INSTR(product_name,'-') to find the hyphens. INSTR will help split the column. */ --QUERY 5 - - - +-- Using if/else then statements combined with whether the name has a hyphen and keeping only the text after the hyphen +SELECT product_name, +CASE WHEN INSTR(product_name, '-') > 0 THEN TRIM(SUBSTR(product_name, INSTR(product_name, '-') + 1)) +ELSE NULL +END AS description +FROM product; --END QUERY /* 2. Filter the query to show any product_size value that contain a number with REGEXP. */ --QUERY 6 - - +SELECT product_size +FROM product +WHERE product_size REGEXP '[0-9]' -- strings containing digits 0 through 9 (can be used to make any combinations of numbers) --END QUERY @@ -111,10 +161,43 @@ HINT: There are a possibly a few ways to do this query, but if you're struggling with a UNION binding them. */ --QUERY 7 +-- temp table that calculates sales in dollard (revenues) based on the market date +DROP TABLE IF EXISTS temp.new_temp; +CREATE TABLE temp.new_temp AS +SELECT * + ,(quantity * cost_to_customer_per_qty) AS total_sales, + SUM(quantity * cost_to_customer_per_qty) OVER (PARTITION BY market_date) AS total_sales_by_day +FROM customer_purchases; +-- Creating CTE named daily sales and getting the highest and lowest days from distinct/unique days, then appending the highest and lowest days using a union +WITH daily_sales AS ( + SELECT DISTINCT + market_date, + total_sales_by_day + FROM temp.new_temp +), +ranked_sales AS ( + SELECT + market_date, + total_sales_by_day, + ROW_NUMBER() OVER (ORDER BY total_sales_by_day DESC) AS highest_day, + ROW_NUMBER() OVER (ORDER BY total_sales_by_day ASC) AS lowest_day + FROM daily_sales +) ---END QUERY +SELECT market_date, total_sales_by_day +FROM ranked_sales +WHERE highest_day = 1 + +UNION + +SELECT market_date, total_sales_by_day +FROM ranked_sales +WHERE lowest_day = 1; + + +--END QUERYs @@ -132,6 +215,22 @@ How many customers are there (y). Before your final group by you should have the product of those two queries (x*y). */ --QUERY 8 +DROP TABLE IF EXISTS temp.vendor_profit; + +CREATE TEMP TABLE vendor_profit AS +SELECT + vendor_id, + product_id, + (5 * original_price) AS total_revenue_per_customer +FROM vendor_inventory; + +SELECT v.vendor_id,p.product_name,ven.vendor_name, SUM(v.total_revenue_per_customer) AS total_revenue +FROM temp.vendor_profit v +CROSS JOIN customer c +JOIN product p ON v.product_id = p.product_id +JOIN vendor ven ON v.vendor_id = ven.vendor_id +GROUP BY v.vendor_id, ven.vendor_name, p.product_name; + @@ -145,7 +244,26 @@ It should use all of the columns from the product table, as well as a new column Name the timestamp column `snapshot_timestamp`. */ --QUERY 9 +-- Creating a new table that copies the product table's columns + +DROP TABLE IF EXISTS temp.product_units; +CREATE TEMP TABLE product_units AS +SELECT * +FROM product; + +-- Deleting all data rows wherein the quantity type is NOT units and IS NULL +DELETE FROM product_units +WHERE product_qty_type != 'unit'; + +DELETE FROM product_units +WHERE product_qty_type ISNULL; + +-- Adding a new column wherein it can contain text data with the current date +ALTER TABLE temp.product_units +ADD COLUMN snapshot_timestamp TEXT; +UPDATE temp.product_units +SET snapshot_timestamp = CURRENT_TIMESTAMP; --END QUERY @@ -155,8 +273,9 @@ Name the timestamp column `snapshot_timestamp`. */ This can be any product you desire (e.g. add another record for Apple Pie). */ --QUERY 10 - - +-- Adding a new row for cheese +INSERT INTO temp.product_units +VALUES(100,'Cheese','500 grams',50,'grams', CURRENT_TIMESTAMP); --END QUERY @@ -167,8 +286,8 @@ This can be any product you desire (e.g. add another record for Apple Pie). */ HINT: If you don't specify a WHERE clause, you are going to have a bad time.*/ --QUERY 11 - - +DELETE FROM product_units +WHERE product_id = 100; --END QUERY @@ -191,7 +310,88 @@ Finally, make sure you have a WHERE statement to update the right row, When you have all of these components, you can run the update statement. */ --QUERY 12 +--- THere are some product IDs that are in product tabel but not in the vendor-inventory table + +SELECT DISTINCT product_id +FROM vendor_inventory; + +SELECT DISTINCT product_id +FROM product; + + +DROP TABLE IF EXISTS temp.Recent_quantity; +CREATE TEMP TABLE Recent_quantity AS +SELECT market_date, quantity, product_id, +ROW_NUMBER() OVER(PARTITION BY product_id ORDER BY market_date DESC) as Most_recent_num +FROM vendor_inventory; + +DELETE FROM Recent_quantity +WHERE Most_recent_num != 1 ; + +-- Has 23 products (the correct numbers based on the product ID) +DROP TABLE IF EXISTS temp.Recent_quantity_nulls; +CREATE TEMP TABLE Recent_quantity_nulls AS +SELECT p.product_id, r.market_date, r.quantity, coalesce(market_date,0) as new_market_date, coalesce(quantity,0) as new_quantity +FROM product as p +LEFT JOIN temp.Recent_quantity as r +ON p.product_id = r.product_id; + + +ALTER TABLE temp.product_units +ADD current_quantity INT; +DROP TABLE IF EXISTS temp.product_units2; +CREATE TEMP TABLE product_units2 AS +SELECT p.product_name, p.product_size, p.product_category_id, p.product_qty_type, p.snapshot_timestamp, p.current_quantity, n.product_id, n.new_quantity, n.new_market_date +FROM Recent_quantity_nulls as n +LEFT JOIN product_units as p +ON n.product_id = p.product_id; + +UPDATE temp.product_units2 +SET current_quantity = new_quantity; + + + +-- Want the most recent quantity value for each product ID + +-- All quantities (even if some are 0) +SELECT product_id, current_quantity, product_qty_type +FROM product_units2 +WHERE product_qty_type='unit' +/* +The product IDs and recent quantity values are listed: + +3 60 unit +4 30 unit +5 20 unit +6 0 unit +7 10 unit +8 10 unit +10 0 unit +12 0 unit +16 140 unit +18 0 unit +19 0 unit +20 0 unit +21 0 unit +23 0 unit + +*/ + +-- Non zero quantities +SELECT product_id, current_quantity, product_qty_type +FROM product_units2 +WHERE current_quantity > 0 and product_qty_type='unit' + +/* +The product IDs and recent quantity units are listed: +3 60 unit +4 30 unit +5 20 unit +7 10 unit +8 10 unit +16 140 unit +*/ --END QUERY