diff --git a/README.md b/README.md index 753fbe96..d46888af 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,8 @@ Teams are not left on their own to solve the challenges. Coaches work with each > We will be going through replicating and processing operational data from an Oracle database into Google Cloud in real time. You'll also figure out how to forecast future demand, and how to visualize this forecast data as it arrives. - [Hack to the Future: Data Track](./hacks/httf-data/README.md) > In this hack we'll help the fictitious Cymbal Shops e-commerce platform to modernize their tech stack, primarily focusing on the migration of the legacy application MySQL database to Cloud Spanner. From that point we'll create BigQuery analytics datasets to handle both historical and live data using federation, use advanced generative AI to enhance the existing dataset with descriptions and images, and finally implement semantic search on this dataset. +- [Disneyland Agentic Data Cloud](./hacks/disneyland-on-gcp/README.md) + > Build an end-to-end intelligent guest assistance system. You will ingest operational data into AlloyDB and generate vector embeddings natively using AlloyDB AI, configure QueryData for predictable SQL generation, and set up real-time replication to BigQuery using Datastream. In BigQuery, you will train forecasting models using BigQuery ML, build a multimodal RAG pipeline on images and PDF brochures, and map visitor movement patterns using BigQuery's native SQL Graph (GQL) capabilities. After cataloging your assets in Knowledge Catalog, you will use Conversational Analytics to query insights, sync these insights back to AlloyDB via BigQuery FDW, expose operational and analytical tools using the MCP Toolbox, and build and deploy a conversational assistant web app powered by the Agent Development Kit (ADK). ### Infrastructure diff --git a/hacks/disneyland-on-gcp/QL_OWNER b/hacks/disneyland-on-gcp/QL_OWNER new file mode 100644 index 00000000..68724db2 --- /dev/null +++ b/hacks/disneyland-on-gcp/QL_OWNER @@ -0,0 +1,3 @@ +meken@google.com +# Collaborators +ginof@google.com diff --git a/hacks/disneyland-on-gcp/README.md b/hacks/disneyland-on-gcp/README.md new file mode 100644 index 00000000..db228718 --- /dev/null +++ b/hacks/disneyland-on-gcp/README.md @@ -0,0 +1,1038 @@ +# Disneyland Agentic Data Cloud + +![Disneyland Agentic Data Cloud 10-Challenge Architecture](images/ghack_challenge_architecture.png) + +## Introduction + +Welcome, Disney Data Wizards! 🪄 + +Planning the perfect Disneyland trip is a complex optimization problem. Visitors want to maximize magic and minimize waiting. They want to know: *Which rides are best suited for them? When are the crowds thinnest? What is the optimal route through the park to avoid bottleneck queues?* + +In this gHack, your mission is to transform raw data—visitor reviews, attraction catalogs, historical wait times, park brochures, and visitor movement logs—into an end-to-end, intelligent guest assistance system. + +This gHack is designed to be highly challenging and is structured into **10 challenges** that can be parallelized across **3 key team personas** to optimize development speed: + +- **DB & Platform Engineers** will build the operational database in AlloyDB, configure Datastream replication, set up the database agentic layer, sync analytical insights, and assemble the final agent and web UI (Challenges 1, 2, 8, 9, and 10). +- **Data Scientists & Analysts** will train predictive models, perform sentiment analysis, cluster attractions in BigQuery, and design the semantic layer/Conversational Analytics agent for park managers (Challenges 3, 6, and 7). +- **AI & Graph Engineers** will construct RAG pipelines, classify multimodal images, and model/query visitor movement patterns using property graphs in BigQuery (Challenges 4 and 5). + +Get ready to build an agentic data pipeline that would make Mickey proud! Let the magic begin! ✨ + +--- + +## Learning objectives + +In this hack, you will build an end-to-end data pipeline with AI and database capabilities on Google Cloud: + +1. **AlloyDB AI**: Ingest operational data and generate vector embeddings natively. +2. **Real-time replication**: Set up CDC from AlloyDB to BigQuery using Datastream. +3. **Agentic database**: Configure predictable SQL generation in AlloyDB and expose it as tools. +4. **Predictive analytics**: Train forecasting and sentiment models using BigQuery ML. +5. **Multimodal RAG**: Build image classification and PDF search pipelines. +6. **Graph analysis**: Map visitor movements and query patterns using GQL. +7. **Context layer (Knowledge Catalog)**: Build a centralized metadata context layer, business glossary, and profiling rules. +8. **Conversational analytics**: Build a natural language assistant using BigQuery Studio. +9. **AlloyDB sync**: Copy insights back to AlloyDB using FDW for fast serving. +10. **MCP tools**: Expose database capabilities using the MCP Toolbox. +11. **App deployment**: Build a web app with the ADK and deploy it to Cloud Run. + +--- + +## Challenges + +- Challenge 1: Setting up AlloyDB and replicating data to BigQuery +- Challenge 2: Creating the agentic database layer +- Challenge 3: Sentiment and wait-time forecasting +- Challenge 4: Image classification and brochure RAG +- Challenge 5: Graph analytics and visitor flow +- Challenge 6: Preparing the Context Layer +- Challenge 7: Conversational analytics for insights +- Challenge 8: From insights to action, syncing BigQuery and AlloyDB +- Challenge 9: Exposing Database Tools via MCP +- Challenge 10: Building the guest assistant app + +--- + +## Prerequisites + +- Basic knowledge of Google Cloud services (AlloyDB, BigQuery, Datastream, Cloud Run) +- Intermediate knowledge of SQL and PostgreSQL +- Basic familiarity with Python and Agentic AI concepts (MCP, ADK) +- Access to a Google Cloud project with the necessary APIs and resources provisioned + +--- + +## Contributors + +- Matt Cornillon +- Rayhane Rezgui + +--- + +## 👥 Team Roles & Parallelization Paths + +This gHack is designed to be highly challenging but is **fully parallelizable** across different team members. Whether you are a team of 2, 5, or more, you can split the challenges to build in parallel and assemble a complete intelligent system in under 4 hours. + +### 📊 Recommended Parallelization Options + +Here is how you can divide and conquer based on your team size: + +![3-Group Parallelization Timeline](images/parallelization_timeline.png) + +#### 🧩 Option 1: The 3-Group Split (Recommended) + +- **Group A (Platform & DB Engineers):** Focuses on the core transactional and serving loop. + - *Path:* [Challenge 1](#challenge-1-setting-up-alloydb-and-replicating-data-to-bigquery) -> [Challenge 2](#challenge-2-creating-the-agentic-database-layer) -> [Challenge 8](#challenge-8-from-insights-to-action-syncing-bigquery-and-alloydb) -> [Challenge 9](#challenge-9-exposing-database-tools-via-mcp) -> [Challenge 10](#challenge-10-building-the-guest-assistant-app) +- **Group B (Data Scientists):** Focuses on analytics, forecasting, and clustering. + - *Path:* [Challenge 3](#challenge-3-sentiment-and-wait-time-forecasting) -> Assist Group C with [Challenge 7](#challenge-7-conversational-analytics-for-insights) or Group A with [Challenge 10](#challenge-10-building-the-guest-assistant-app) +- **Group C (AI & Graph Engineers):** Focuses on unstructured data, graphs, and internal conversational search. + - *Path:* [Challenge 4](#challenge-4-image-classification-and-brochure-rag) -> [Challenge 5](#challenge-5-graph-analytics-and-visitor-flow) -> [Challenge 6](#challenge-6-preparing-the-context-layer) -> [Challenge 7](#challenge-7-conversational-analytics-for-insights) + +#### 👥 Option 2: The 2-Group Split + +- **Group A (Database & Application Developers):** Focuses on database setup, agentic tool serving, and app integration. + - *Path:* [Challenge 1](#challenge-1-setting-up-alloydb-and-replicating-data-to-bigquery) -> [Challenge 2](#challenge-2-creating-the-agentic-database-layer) -> [Challenge 8](#challenge-8-from-insights-to-action-syncing-bigquery-and-alloydb) -> [Challenge 9](#challenge-9-exposing-database-tools-via-mcp) -> [Challenge 10](#challenge-10-building-the-guest-assistant-app) +- **Group B (AI, ML & Analytics Engineers):** Focuses on all BigQuery-centric analytics, ML models, RAG, and property graphs. + - *Path:* [Challenge 4](#challenge-4-image-classification-and-brochure-rag) -> [Challenge 3](#challenge-3-sentiment-and-wait-time-forecasting) -> [Challenge 5](#challenge-5-graph-analytics-and-visitor-flow) -> [Challenge 6](#challenge-6-preparing-the-context-layer) -> [Challenge 7](#challenge-7-conversational-analytics-for-insights) + +--- + +## Challenge 1: Setting up AlloyDB and replicating data to BigQuery + +**Target Persona:** Data Engineer / DBA | **Estimated Duration:** 45 minutes + +### Introduction + +AlloyDB is our operational data store, and BigQuery is the workhorse for analytical workloads. In this challenge, you will initialize the operational database in AlloyDB, load the base data from Cloud Storage, generate embeddings at the source using AlloyDB AI, and set up real-time replication to BigQuery using Datastream. Finally, you will explore the replicated data in BigQuery Studio using the new Data Canvas visual interface. + +### Description + +#### Task 1.1: Ingest Data into AlloyDB + +First, let's connect to your **AlloyDB for PostgreSQL** cluster. + +> [!TIP] +> **AlloyDB Postgres Credentials:** +> +> - **Database:** `disney` *(Note: Connect to the default `postgres` database first to create the `disney` database, then reconnect to `disney` for the rest of the hackathon)* +> - **Username:** `postgres` +> - **Password:** Use the one provided by your coach + +Connect using **AlloyDB Studio** or `psql` and perform the following tasks: + +1. **Create a Dedicated Database:** + Connect to the default `postgres` database and run the following SQL command to create a new dedicated database: + + ```sql + CREATE DATABASE disney; + ``` + + Once created, **disconnect and reconnect** to your new `disney` database. All subsequent tables, extensions, and queries in this hackathon must be run within the `disney` database. + +2. **Create the Tables:** + - **`disneyland_reviews`**: Represents visitor reviews. + - Columns: `review_id` (integer, Primary Key), `rating` (integer), `year_month` (text), `reviewer_location` (text), `review_text` (text), `branch` (text). + - **`disneyland_attractions`**: Represents the park's attractions. + - Columns: `attraction_id` (integer, Primary Key), `branch` (text), `name` (text), `description` (text). + - **`visitor_movements`**: Represents visitor movements in the park. + - Columns: `visitor_id` (integer), `from_attraction_id` (integer), `to_attraction_id` (integer), `timestamp` (timestamp). +3. **Add a Full-Text Search Vector:** + To enable high-performance hybrid search later, add a generated `tsvector` column to the attractions table: + + ```sql + ALTER TABLE disneyland_attractions + ADD COLUMN description_tsvector tsvector + GENERATED ALWAYS AS (to_tsvector('english', description)) STORED; + ``` + +4. **Import Data from Cloud Storage:** + Use the AlloyDB Import API (via UI or `gcloud`) or the tool of your choice to import the data from these public CSVs: + - `gs://ghacks-disneyland-on-gcp/reviews.csv` into `disneyland_reviews` + - `gs://ghacks-disneyland-on-gcp/attractions.csv` into `disneyland_attractions` + - `gs://ghacks-disneyland-on-gcp/visitor_movements.csv` into `visitor_movements` + +#### Task 1.2: Generate Vector Embeddings at the Source + +To support semantic searches on our attractions, we need to generate and store vector embeddings directly inside AlloyDB using **AlloyDB AI**. + +1. **Enable the Extensions:** + + ```sql + CREATE EXTENSION IF NOT EXISTS vector CASCADE; + CREATE EXTENSION IF NOT EXISTS google_ml_integration CASCADE; + ALTER DATABASE disney SET google_ml_integration.enable_preview_ai_functions = 'on'; + ``` + +2. **Add the Embedding Column:** + Add a vector column of size 3072 (the output dimension of `gemini-embedding-001`) to the attractions table: + + ```sql + ALTER TABLE disneyland_attractions ADD COLUMN embedding vector(3072); + ``` + +3. **Generate Embeddings:** + Populate the `embedding` column by calling Gemini Enterprise Agent Platform's embedding model natively from SQL: + + ```sql + -- TODO: Write an UPDATE query that populates the `embedding` column by calling + -- Gemini Enterprise Agent Platform's embedding model natively from SQL using the 'gemini-embedding-001' model. + ``` + +#### Task 1.3: Set Up Real-Time Replication with One-Click Datastream + +To stream our data from AlloyDB to BigQuery in near real-time, we will use **Google Datastream** via the **One-Click Datastream Integration**. + +**Create the Stream (One-Click Setup):** + +1. In the Google Cloud Console, navigate to **AlloyDB > Clusters**. +2. Select your cluster, and look for the **Replicate data to BigQuery**. +3. Follow the guided wizard and customize the stream. +4. Ensure to + - Create the stream in the same region as your AlloyDB cluster + - Use **Built-in database authentication** + - Select the three tables you just created + - Configure the write mode select **Merge**, staleness limit to **0 seconds** and a single datatset for all schemas named **disney**. + +The stream will be created and started automatically, do not wait until its finished. You can start challenge 2 and come back to this step later. + +#### Task 1.4: Prove the Flow with Data Canvas + +Once the stream is running, the data will begin replicating to BigQuery. + +1. Navigate to **BigQuery Studio**. +2. Open **Data Canvas** (the new visual interface for exploring data). +3. Load the replicated `disneyland_reviews` table. +4. Create a quick visualization (e.g., a bar chart showing the average rating per branch) to verify the data is flowing correctly from AlloyDB. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Verify that `disneyland_reviews` has ~20,000 rows and `disneyland_attractions` has ~73 rows in AlloyDB. +- Provide the SQL query you used to generate the embeddings natively in AlloyDB. +- Run a similarity search query in AlloyDB demonstrating the top 5 attractions similar to `'thrilling dark ride in space'`. +- Show a visualization of your BigQuery Data Canvas showing the replicated reviews. + +--- + +## Challenge 2: Creating the agentic database layer + +**Target Persona:** DBA / Database Developer | **Estimated Duration:** 40 minutes | *Prerequisite: Challenge 1* + +### Introduction + +In this challenge, you will establish the **agentic database layer** directly on top of your AlloyDB instance. This involves configuring **QueryData** to guarantee predictable SQL generation for common queries (NL2SQL), and exposing advanced database capabilities (like hybrid search and semantic filtering) as SQL functions. + +By preparing these operational capabilities now, you lay the foundation for the agent to interact securely and deterministically with your data. + +### Description + +**QueryData** is an intelligent natural language-to-SQL (NL2SQL) engine in AlloyDB that translates conversational questions into precise, secure SQL queries. Rather than just matching static "golden queries", it uses a **Context Set** (a JSON file defining your schema's business logic) to dynamically construct queries using: + +- **Query Templates:** Parameterized QA pairs (using `$1`, `$2`) that allow the engine to generalize query patterns. +- **Query Facets:** Reusable filters (e.g., mapping "highly rated" to `rating >= 4`) that can be dynamically appended. +- **Value Search Queries:** Background queries (exact, fuzzy, or semantic) that map user-typed entities to actual database values. + +For more details, see the [AlloyDB QueryData Documentation](https://cloud.google.com/alloydb/docs/ai/querydata). + +#### Task 2.1: Enable the AlloyDB Data API + +QueryData and downstream API integration require the **AlloyDB Data API** (also known as the `executesql` API) to be explicitly enabled on your AlloyDB instance. Run the following `curl` command in your Cloud Shell to enable it: + +```bash +curl -X PATCH \ + -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + -H "Content-Type: application/json" \ + "https://alloydb.googleapis.com/v1alpha/projects/\${GOOGLE_CLOUD_PROJECT}/locations//clusters/disney-cluster/instances/disney-instance?updateMask=dataApiAccess" \ + -d '{"dataApiAccess": "ENABLED"}' +``` + +*(Note: Replace `` with the region where your cluster is deployed, e.g. `europe-west1`. The project ID is automatically resolved via the `GOOGLE_CLOUD_PROJECT` environment variable in Cloud Shell.)* + +#### Task 2.2: Register IAM Users in AlloyDB + +To interact with the database, QueryData relies on **IAM Database Authentication**. This means both you (the developer) and the AI agent must be registered as database users. + +Specifically, you must register: + +1. **Your Google Account** – to manage Context Sets via the AlloyDB Console. +2. **The Agent's Service Account** – to allow the agent to connect and execute queries. + +To register these users, follow these steps: + +1. **Log in using Basic Authentication:** + - In the Google Cloud Console, navigate to **AlloyDB > Clusters** and select your cluster. + - Click **AlloyDB Studio** in the left menu. + - Sign in to the `disney` database using **Basic Authentication** (Username: `postgres`, Password: ``). + +2. **Register the IAM Users:** + Open a new query editor and run the following SQL commands to register your own Google account (so you can manage context sets) and the agent's Service Account (so the agent can connect), granting them the necessary table privileges: + + ```sql + -- 1. Register your own Google account (replace with your actual GCP login email) + CREATE USER "your-email@domain.com" WITH FLAGS ('IAMUser'); + GRANT USAGE ON SCHEMA public TO "your-email@domain.com"; + GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO "your-email@domain.com"; + + -- 2. Register the dedicated Agent Service Account (replace with your project ID) + CREATE USER "disney-agent-sa@.iam.gserviceaccount.com" WITH FLAGS ('IAMUser'); + GRANT USAGE ON SCHEMA public TO "disney-agent-sa@.iam.gserviceaccount.com"; + GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO "disney-agent-sa@.iam.gserviceaccount.com"; + + -- 3. Ensure future tables grant access automatically + ALTER DEFAULT PRIVILEGES IN SCHEMA public + GRANT SELECT, INSERT, UPDATE ON TABLES TO "your-email@domain.com"; + ALTER DEFAULT PRIVILEGES IN SCHEMA public + GRANT SELECT, INSERT, UPDATE ON TABLES TO "disney-agent-sa@.iam.gserviceaccount.com"; + ``` + + *(Note: Replace `` with your actual Google Cloud Project ID, and `your-email@domain.com` with the email you use to log into the Google Cloud Console.)* + +#### Task 2.3: Set Up the QueryData Context Set in AlloyDB + +1. **Create the JSON Context Set File:** + Create a file named `querydata_disney_context.json` on your local system. Fill in the SQL templates and queries according to the requirements: + + ```json + { + "templates": [ + { + "nlQuery": "Show available attractions in Disneyland Paris", + "sql": "/* TODO: Write SQL to select name, description from public.disneyland_attractions filtered by branch */", + "intent": "List all attractions for a specific park branch", + "manifest": "List attractions by branch", + "parameterized": { + "parameterized_intent": "Show available attractions in $1", + "parameterized_sql": "/* TODO: Write the parameterized SQL using $1 */" + } + }, + { + "nlQuery": "Find reviews with rating 5 for Space Mountain", + "sql": "/* TODO: Write SQL to join reviews and attractions on branch, filtered by attraction name and rating */", + "intent": "Get reviews with a specific rating for a named attraction", + "manifest": "Get reviews by attraction and rating", + "parameterized": { + "parameterized_intent": "Find reviews with rating $2 for $1", + "parameterized_sql": "/* TODO: Write the parameterized SQL using $1 (name) and $2 (rating) */" + } + }, + { + "nlQuery": "Average rating of attractions in California Adventure", + "sql": "/* TODO: Write SQL to select the average rating from public.disneyland_reviews filtered by branch */", + "intent": "Calculate the average review rating for a specific branch", + "manifest": "Average rating by branch", + "parameterized": { + "parameterized_intent": "Average rating of attractions in $1", + "parameterized_sql": "/* TODO: Write the parameterized SQL using $1 */" + } + } + ], + "facets": [ + { + "sql_snippet": "/* TODO: Write SQL snippet to filter rating >= 4 */", + "intent": "highly rated reviews", + "manifest": "Filter reviews by a minimum rating threshold", + "parameterized": { + "parameterized_intent": "reviews with rating greater than or equal to $1", + "parameterized_sql_snippet": "/* TODO: Write the parameterized SQL snippet using $1 */" + } + } + ], + "value_searches": [ + { + "query": "SELECT DISTINCT T.name as value, 'public.disneyland_attractions.name' as columns, 'Attraction Name' as concept_type, (1.0 - ts_rank(to_tsvector('english', T.name), plainto_tsquery('english', $value))) as distance, '{}'::text as context FROM public.disneyland_attractions T WHERE to_tsvector('english', T.name) @@ plainto_tsquery('english', $value)", + "concept_type": "Attraction Name", + "description": "Full-text search for attraction names" + }, + { + "query": "SELECT DISTINCT T.branch as value, 'public.disneyland_attractions.branch' as columns, 'Branch Name' as concept_type, (1.0 - ts_rank(to_tsvector('english', T.branch), plainto_tsquery('english', $value))) as distance, '{}'::text as context FROM public.disneyland_attractions T WHERE to_tsvector('english', T.branch) @@ plainto_tsquery('english', $value)", + "concept_type": "Branch Name", + "description": "Full-text search for park branches" + } + ] + } + ``` + +2. **Upload Context Set to AlloyDB (via IAM Authentication):** + - In the Google Cloud Console, open **AlloyDB Studio** again. + - This time, sign in using **IAM Authentication** (this will use your logged-in Google account, which you registered in Task 2.2). You can click on the "Switch user/database" button (icon with a person and database) to switch to IAM authentication. + - In the left sidebar, check for **Context sets**. + - Click **Create context set**, name it `disney-context`, and upload the `querydata_disney_context.json` file. + - Validate the setup by using the **Test context set** feature with variations of your natural language templates. + +#### Task 2.4: Expose AlloyDB AI Operators + +You will prepare SQL queries that leverage AlloyDB's advanced AI features to expose them as tools. + +> [!TIP] +> Remember to switch back to **basic authentication** (logging in as the `postgres` user) in AlloyDB Studio before executing the SQL queries in this task. + +1. **Hybrid Search (ScaNN + FTS):** + Before setting up hybrid search, you need to install the required extensions in AlloyDB. + + **What is Hybrid Search (`ai.hybrid_search`)?** + Hybrid search combines the strengths of **semantic search** (vector similarity) and **keyword search** (Full-Text Search) to deliver highly relevant results. While vector search excels at understanding the conceptual meaning of a query, keyword search ensures exact matches (like names or specific terms) aren't missed. The `ai.hybrid_search` function, provided by the `ai` extension, executes both searches in parallel and merges their results using Reciprocal Rank Fusion (RRF) to produce a single, unified relevance score. + + Run the following SQL commands in AlloyDB Studio to install the extensions and create the necessary indexes: + + ```sql + -- Install required extensions + CREATE EXTENSION IF NOT EXISTS alloydb_scann CASCADE; + CREATE EXTENSION IF NOT EXISTS rum CASCADE; + + -- Index RUM for Keyword FTS + CREATE INDEX IF NOT EXISTS attractions_tsvector_idx ON disneyland_attractions USING RUM (description_tsvector rum_tsvector_ops); + + -- Index ScaNN for Vector Cosine Similarity + CREATE INDEX IF NOT EXISTS attractions_vector_idx ON disneyland_attractions USING scann (embedding cosine) WITH (num_leaves=10); + ``` + + After creating the indexes, try running a hybrid search query using the `ai.hybrid_search` function to see it in action (try searching for a "thrilling space roller coaster"). + + This hybrid search capability will be mapped later directly to an MCP tool. Save it as a SQL file for later reference. + +2. **Semantic Filtering (`google_ml.if`):** + Create a SQL function to easily evaluate if a specific attraction is suitable or safe based on a guest's natural language profile (e.g., checking if a specific ride is safe for pregnant women or suitable for toddlers). + + **What is Semantic Filtering (`google_ml.if`)?** + Semantic filtering allows you to evaluate complex, natural language conditions directly within your SQL queries. Unlike vector similarity searches that rank results by closeness, `google_ml.if` uses a generative AI model to evaluate a prompt against your data on a row-by-row basis, returning a simple boolean (`true` or `false`). This is extremely powerful for ad-hoc safety or suitability checks where pre-defined categories or labels do not exist. + + > [!TIP] + > Before writing the reusable SQL function, it is highly recommended to build and run a standalone test query in AlloyDB Studio. This allows you to verify the behavior of `google_ml.if` on a single attraction (for example, testing if "Space Mountain" is suitable for "pregnant women") before embedding it in the function. + + Create the SQL function: + + ```sql + CREATE OR REPLACE FUNCTION check_attraction_suitability(attraction_name TEXT, suitability_profile TEXT) + RETURNS TABLE(name TEXT, description TEXT) AS $$ + SELECT name, description + FROM disneyland_attractions + WHERE name = attraction_name + AND /* TODO: Add the google_ml.if operator logic here to evaluate description against suitability_profile */; + $$ LANGUAGE SQL; + ``` + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show a screenshot or validation proof of the **QueryData** displaying a successful natural-language-to-SQL translation. +- Provide the SQL DDL definition for the `check_attraction_suitability` function in AlloyDB Studio. +- Provide the SQL query you used to run and test the hybrid search (ScaNN + FTS) in AlloyDB Studio, along with its results. +- Provide the SQL query showing how you called and tested the `check_attraction_suitability` function in AlloyDB Studio, along with its results. + +--- + +## Challenge 3: Sentiment and wait-time forecasting + +**Target Persona:** Data Scientist / Data Analyst | **Estimated Duration:** 45 minutes | *Note: Can be parallelized immediately using raw CSVs in BigQuery.* + +### Introduction + +Analyzing visitor sentiment and forecasting ride waiting times are crucial to improving the guest experience. In this challenge, you will use BigQuery ML and the new BigQuery Studio Data Science Agent to perform automated sentiment classification on reviews, train a time-series forecasting model to predict future waiting times, and build unsupervised classification and ranking models to categorize attractions by intensity. + +### Description + +#### Task 3.1: Automated Sentiment Analysis with BQ Studio Data Science Agent + +Rather than writing Python code from scratch, you will leverage the new **Data Science Agent** in BigQuery Studio to accelerate your analysis. + +> [!IMPORTANT] +> **Dependency Note:** +> This task queries the `disneyland_reviews` table in BigQuery, which is replicated from AlloyDB. This requires **Challenge 1** (specifically the Datastream replication in Task 1.3) to be completed first. + +1. Open the **Data Science Agent** panel in BigQuery Studio. +2. Using natural language, prompt the agent to write a SQL query or a Python notebook that classifies the sentiment of the reviews in `disneyland_reviews` into `Positive`, `Negative`, or `Neutral`. +3. The agent should suggest using `AI.GENERATE_TEXT` or `AI.GENERATE` with a Gemini model (e.g., `gemini-2.5-flash`) to perform the sentiment classification. +4. Run the generated query on a sample of **100 reviews** and save the results into a new table `reviews_sentiment_analysis`. + +#### Task 3.2: Time-Series Wait Time Forecasting + +We want our guest assistant to predict wait times for any hour of the day. + +1. Load the historical wait times dataset from: + `gs://ghacks-disneyland-on-gcp/waiting_times.csv` into a BigQuery table named `waiting_times`. +2. Use BigQuery ML to train a time-series forecasting model. You can choose either: + - **ARIMA_PLUS**: The classic, fast statistical forecasting model. + - **TimesFM**: Google's state-of-the-art foundation model for time-series forecasting (using `AI.FORECAST`). +3. Forecast the wait times for all attractions for the next 24 hours in 30-minute intervals, and save the results in a table named `forecasted_waiting_times`. + +#### Task 3.3: Ride Clustering (Intensity & Popularity) + +To better classify our rides, we will group attractions into logical clusters using unsupervised learning. + +1. Build a query that aggregates statistics for each attraction: average wait time, total review count, and average rating. +2. Use `AI.CLASSIFY` to categorize rides based on their descriptions into one of three magical categories: `[easy-peasy, thrilling, extreme]`. +3. Use `AI.SCORE` to compare and order attractions based on a thrill level, where Rank 10 is the most extreme and Rank 1 is the least. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show the prompt and the resulting SQL/Python code generated by the Data Science Agent. +- Show the trained forecasting model and a query displaying the forecasted wait times for the next 24 hours. +- Verify the creation and content of the following two tables: + - `thrill_class`: containing a column `class` with the extracted category. + - `thrill_score`: containing a column `rank` with the numerical thrill rank score. + +--- + +## Challenge 4: Image classification and brochure RAG + +**Target Persona:** AI Developer / Data Scientist | **Estimated Duration:** 45 minutes + +### Introduction + +Disneyland managers have a collection of visitor photos and PDF brochures. In this challenge, you will construct a pipeline to organize this unstructured data, perform image classification, and build a Vector Search-based Retrieval-Augmented Generation (RAG) pipeline to query and extract information from park brochures using natural language. + +### Description + +#### Task 4.1: Multimodal Image Classification + +You have a GCS bucket containing park photos: `gs://ghacks-disneyland-on-gcp/attraction_parc_photos/`. + +1. Create a **BigQuery Object Table** pointing to the GCS bucket. +2. Create a remote model in BigQuery pointing to a multimodal model (e.g., `gemini-2.5-flash`). +3. Use `AI.GENERATE_TEXT` to pass the image URIs to the model with a prompt asking: *"Is this image from a Disneyland park? Answer with a JSON object containing keys 'is_disneyland' (boolean) and 'reason' (string)."* +4. Save the structured results into a table `images_classification`. + +#### Task 4.2: PDF Brochure Ingestion & RAG Pipeline + +We want our assistant to answer detailed questions based on the official park brochures (PDFs) located in `gs://ghacks-disneyland-on-gcp/disneyland_brochures/`. + +1. Create an **Object Table** pointing to the brochures bucket. +2. Create a **Python UDF** in BigQuery to extract text and chunk the PDF files. You can use `pypdf` inside the UDF environment. +3. Chunk the PDFs, and generate embeddings for each text chunk using a remote BQML embedding model (`gemini-embedding-001`). +4. Store the chunks and their vector embeddings in a table `brochure_embeddings`. + +#### Task 4.3: Vector Search & RAG Validation + +1. Perform a vector search query on `brochure_embeddings` to find the most relevant brochure chunks for the question: *"Where can I find a buffet-style Tex-Mex meal?"* (or French: *"Où manger un repas tex-mex à volonté ?"*). +2. Pass the retrieved chunks as context along with the question to `gemini-2.5-flash` using `AI.GENERATE_TEXT` to generate a grounded, accurate response. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Verify the BQ Object Tables created for both photos and PDFs. +- Show the results from the `images_classification` table displaying at least 5 images, their classification, and the reason. +- Show the final SQL query performing the vector search and RAG generation, along with the grounded text response answering the Tex-Mex question. + +--- + +## Challenge 5: Graph analytics and visitor flow + +**Target Persona:** Graph Specialist / Data Analyst | **Estimated Duration:** 45 minutes + +### Introduction + +Understanding visitor movement patterns is key to optimizing park operations and recommending optimal paths to avoid long queues. In this challenge, you will define a Property Graph in BigQuery using BigQuery's new native SQL Graph capabilities over the replicated visitor movement logs, query the graph for flow patterns and multi-hop journeys, and construct a routing recommendation table. + +### Description + +#### Task 5.1: Build a Property Graph in BigQuery + +Using BigQuery's new native **SQL Graph** capabilities, you will define a property graph over the attractions and movements. + +> [!IMPORTANT] +> **Dependency Note:** +> Creating the Property Graph requires both `public_disneyland_attractions` and `public_visitor_movements` tables to exist in BigQuery. This requires **Challenge 1** (Datastream replication) to be committed first. + +1. Define the **Property Graph** schema. + - **Nodes (Vertices):** Attractions (from the replicated `public_disneyland_attractions` table). + - **Edges (Relationships):** Movements (from the replicated `public_visitor_movements` table). +2. Write the DDL to create the property graph `disney_movement_graph`. + +#### Task 5.2: Query the Graph for Patterns + +Write graph queries using `GRAPH_TABLE` and GQL match patterns to solve the following analytical questions: + +1. **Flow Analysis:** *What are the top 3 attractions visitors run to immediately after leaving "Space Mountain"?* + Write a query matching paths: `(a:Attraction {name: 'Space Mountain'}) -[e:Moved]-> (b:Attraction)`. +2. **Multi-Hop Journeys:** *Find the most common 3-ride sequences (A -> B -> C) starting from "Space Mountain" taken by the same visitor within a 2-hour window.* + Analyze paths matching: `(a:Attraction {name: 'Space Mountain'}) -[e1:Moved]-> (b:Attraction) -[e2:Moved]-> (c:Attraction)` where both movements are made by the same visitor. + +#### Task 5.3: Visitor Journey Tracking & Path Analysis + +Now, let's explore GQL's path capabilities to analyze journeys taken by visitors. + +1. **Specific Journey Tracking:** Write a graph query using quantified path patterns to find how many rides visitor `'11613'` took to get from `'Dumbo the Flying Elephant'` to `'Disneyland Railroad'`. +2. **Multi-Hop Journeys by Visitor:** Write a graph query to find all unique visitor IDs who traveled from `'Space Mountain'` to `'Indiana Jones Adventure'` through an intermediate attraction in a 2-hop journey (A -> B -> C) where both transitions are made by the same visitor. + +#### Task 5.4: Graph-Based Recommendations + +To power our intelligent guest assistant, we need to provide next-ride recommendations based on real visitor behavior. + +1. **Extract Recommendations:** Write a graph query to find the most recurrent next attraction visitors go to after visiting each specific attraction. +2. **Build the Recommendation Table:** Save the results of this query into a new BigQuery table named `graph_recommendations`. This table should include the current attraction, the recommended next attraction, and a ranking score (e.g., based on frequency). This table will be synced and used later by the agent. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show the SQL DDL statement used to define and create the Property Graph `disney_movement_graph`. +- Provide the SQL graph query for the "Flow Analysis" (top 3 rides after Space Mountain) and its corresponding output. +- Provide the SQL graph query for "Multi-Hop Journeys" starting from Space Mountain and its corresponding output. +- Provide the SQL graph queries for "Visitor Journey Tracking" and "Reachable Journeys", and display their outputs. +- Verify the creation and content of the `graph_recommendations` table showing the most recurrent next attractions. + +--- + +## Challenge 6: Preparing the Context Layer + +**Target Persona:** Data Engineer / AI Architect | **Estimated Duration:** 40 minutes + +### **Introduction** + +Before creating your conversational AI agents, you need to build a centralized context layer. This layer ensures your agents understand your business terminology, operational metadata, and data asset structures, leading to higher accuracy and fewer hallucinations. + +#### **Task 6.1: Technical Metadata Enrichment** + +1. Navigate to your BigQuery dataset. +2. Enrich your chosen data assets (such as `disneyland_reviews`) by adding schema descriptions to columns. +3. Ensure critical columns like your vector embeddings, image analysis JSON fields, and source URIs have clear technical metadata descriptions. + +#### **Task 6.2: Business Glossary Alignment** + +To align raw technical structures with organizational understanding, you must map your catalog to a standardized business vocabulary. + +1. **Glossary Creation:** Create a centralized Business Glossary for the Disneyland analytics ecosystem. +2. **Core Definitions:** Define core business terms and definitions inside the glossary (e.g., define terms like *"Rollercoaster"*, *"Premium visitor"*, or *"Buffet Dining Category"* + *An example: “Premium visitor”: A visitor who left more than 2 reviews*). +3. **Asset Mapping to BigQuery:** Link these business terms directly to their corresponding BigQuery columns to map technical metadata to business language. +4. **Asset Mapping to AlloyDB:** Try mapping some terms to AlloyDB assets as well to see how Knowledge Catalog equally integrates to operational & analytical databases. + +#### **Task 6.3: Automated profiling & quality** + +It’s very important to understand the distribution of the values in a column and quality rules in order to better discover the data. + +1. Run a data profile scan on the disneyland\_reviews table. Analyze the results. +2. Define & run an automatic data quality scan with different rules (profile-based, predefined generic, custom, etc) + +#### **Task 6.4: Automated GCS Metadata Generation** + +Set up an automated extraction pipeline to handle documentation and unstructured assets within your environment. + +> [!TIP] +> **This can be done in the BigQuery Metadata Curation tab.** + +1. Configure the pipeline to analyze your unstructured `gs://ghacks-disneyland-on-gcp/` bucket. +2. Automatically generate and attach metadata tags (such as language, document type, target audience, and revision date) to the PDF assets, use semantic inference for better results. + +#### **Task 6.5: lookup context API Integration** + +Once your technical, business, and object storage metadata are established, wire them into your execution layer for application discovery. + +1. Utilize the LookupContext API to fetch operational and structural context dynamically. Test the API against a standard BigQuery data asset and an AlloyDB transactional database table. You can use Python or a rest API +2. Verify that the API returns detailed, low-latency context maps that an LLM agent can ingest to understand the underlying database schemas and table relationships. + +### **Success Criteria** + +To validate this challenge, you must demonstrate the following: + +- Show the enriched schema descriptions for your target tables directly within the BigQuery Console. +- Provide a summary or export of the linked terms inside your centralized Disneyland Business Glossary. +- Show the successful pipeline logs or sample metadata tags generated for the PDF assets in Cloud Storage. +- Provide the API JSON response payload from a successful `LookupContext` call showing the multi-database schema mapping. + +--- + +## Challenge 7: Conversational analytics for insights + +**Target Persona:** Data Analyst / Product Manager | **Estimated Duration:** 30 minutes + +### Introduction + +Disneyland park managers need to query this complex multi-silo dataset (reviews, wait times, graph movements, classifications) without writing SQL. In this challenge, you will build a data agent using BigQuery's Conversational Analytics. You will leverage the context layer defined in the Knowledge Catalog. + +### Description + +#### Task 7.1: Initialize the Conversational Analytics Agent + +1. In BigQuery Studio, navigate to the **Agents** tab. +2. Create a new agent named `disney_park_analyst` and connect it to the table under disney dataset. You can also put the previously created BQ graph as a knowledge source. (You can either choose tables or a Graph, not both) + +#### Task 7.2: Use the Knowledge Catalog + +To prevent the agent from hallucinating, you can leverage the previously configured Knowledge Catalog. + +1. **Metadata Descriptions:** Choose sources with curated metadata. +2. **Synonyms & Vocabulary:** Make sure business terms are imported. + +#### Task 7.3: Define Golden Queries + +Train the agent's SQL generation engine by providing **Golden Queries**—pre-approved, highly accurate SQL templates that the model can reference. + +Provide golden queries for: + +- Joining the attractions table with the wait-time forecasts. +- Querying the graph routing table. + +#### Task 7.4: Execute Multi-Silo Prompts + +Once configured, test the agent in the chat interface. Ask complex, cross-dataset questions like: + +- *« Which attractions have the highest negative sentiment today, and what is the most common path visitors take after leaving them? »* + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show the Conversational Analytics agent `disney_park_analyst` configured in the BigQuery Console. +- List the synonyms and Golden Queries you defined in the agent's configuration. +- Show a screenshot or proof of the chat interface successfully answering the complex multi-silo prompt without any SQL syntax errors. + +--- + +## Challenge 8: From insights to action, syncing BigQuery and AlloyDB + +**Target Persona:** DBA / Database Developer | **Estimated Duration:** 45 minutes | *Prerequisites: Challenge 1 and 2 must be completed. Challenge 3 and 5 must be completed (or mock tables used in BigQuery).* + +### Introduction + +To serve analytical insights (like wait time forecasts and next-ride recommendations) with sub-millisecond latency and without overloading BigQuery, we will not query BigQuery directly from the agent. Instead, we will use **BigQuery Foreign Data Wrapper (FDW)** to copy the analytical insights from BigQuery into **local tables** inside AlloyDB. + +This ensures that AlloyDB remains the single, high-performance serving layer for the agent, while BigQuery is used purely for heavy analytical processing. + +### Description + +#### Task 8.1: Create Local Analytical Tables in AlloyDB + +First, you need to create the local tables in AlloyDB that will store the synced analytical data. + +1. Open **AlloyDB Studio** and connect to your `disney` database. +2. Run the following DDL statements to create the target tables: + + ```sql + CREATE TABLE IF NOT EXISTS public.forecasted_waiting_times ( + attraction_id INT, + forecasted_timestamp TIMESTAMP, + predicted_wait_time NUMERIC + ); + + CREATE TABLE IF NOT EXISTS public.graph_recommendations ( + attraction_id INT, + recommended_next_attraction_id INT, + recommendation_rank INT + ); + ``` + +#### Task 8.2: Map BigQuery Tables using the AlloyDB Studio Wizard + +Use the built-in wizard to easily map the BigQuery tables as foreign tables. + +1. In **AlloyDB Studio**, click on the **BigQuery Tables** three-dots button in the Data Explorer pane on the left-hand-side. +2. Follow the wizard to connect to your BigQuery tables. +3. Map the following tables, naming the foreign tables with a `bq_` prefix to distinguish them from your local tables: + - Map `disney.forecasted_waiting_times` to a foreign table named `bq_forecasted_waiting_times`. + - Map `disney.graph_recommendations` to a foreign table named `bq_graph_recommendations`. +4. Query the foreign tables to verify the connection. + +#### Task 8.3: Sync Data from BigQuery to AlloyDB + +Now, copy the data from the foreign tables into your local AlloyDB tables. + +1. In AlloyDB Studio, run the following SQL queries to perform the initial sync: + + ```sql + -- Sync wait time forecasts + INSERT INTO public.forecasted_waiting_times (attraction_id, forecasted_timestamp, predicted_wait_time) + SELECT attraction_id, forecasted_timestamp, predicted_wait_time + FROM public.bq_forecasted_waiting_times; + + -- Sync graph recommendations + INSERT INTO public.graph_recommendations (attraction_id, recommended_next_attraction_id, recommendation_rank) + SELECT attraction_id, recommended_next_attraction_id, recommendation_rank + FROM public.bq_graph_recommendations; + ``` + +2. Verify that the local tables `public.forecasted_waiting_times` and `public.graph_recommendations` now contain rows. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show the DDL used to create the local tables in AlloyDB. +- Provide a screenshot of AlloyDB Studio showing the local tables populated with synced data from BigQuery. + +--- + +## Challenge 9: Exposing Database Tools via MCP + +**Target Persona:** Platform Engineer / DBA | **Estimated Duration:** 30 minutes | *Prerequisites: Challenges 2, 7, and 8 must be completed.* + +### Introduction + +Now that all operational and analytical data resides locally in AlloyDB, you will expose these capabilities as tools using the **MCP Toolbox for databases**. This allows any downstream AI agent to securely and efficiently interact with the database. + +### Description + +#### Task 9.1: Install MCP Toolbox + +In your Cloud Shell terminal, download and make the toolbox executable: + +```bash +export VERSION=1.1.0 +curl -L -o toolbox https://storage.googleapis.com/mcp-toolbox-for-databases/v$VERSION/linux/amd64/toolbox +chmod +x toolbox +``` + +#### Task 9.2: Configure `tools.yaml` + +Create a `tools.yaml` file outlining all the tools. + +```yaml +# ========================================== +# SOURCES +# ========================================== + +# Source 1: Standard Postgres Connection (used for Tools 1-5) +kind: source +name: disney-db +type: alloydb-postgres +project: "[YOUR_PROJECT_ID]" +region: "europe-west1" +cluster: "[YOUR_CLUSTER]" +instance: "[YOUR_INSTANCE]" +ipType: "public" +database: "disney" +user: "postgres" +password: "buildwithgemini2026" + +--- + +# Source 2: Gemini Data Analytics API (used for QueryData) +kind: source +name: gda-api-source +type: cloud-gemini-data-analytics +projectId: "[YOUR_PROJECT_ID]" + +# ========================================== +# TOOLS +# ========================================== + +--- +# Tool 1: Hybrid Search using ScaNN and Full-Text Search +kind: tool +name: search_attractions_hybrid +type: postgres-sql +source: disney-db +description: "Performs a high-performance hybrid (vector + keyword) search on park attractions based on user interests." +parameters: + - name: vector_query + type: string + description: "Semantic search term (e.g., 'thrilling space roller coaster')" + - name: text_query + type: string + description: "Keyword search term (e.g., 'Space Mountain')" +statement: | + -- TODO: Write the hybrid search query utilizing AlloyDB's ai.hybrid_search operator, + -- combining the vector cosine similarity index and the full-text search index. + -- (Hint: Pass the vector query embedded via google_ml.embedding) +--- +# Tool 2: Semantic Filtering using AlloyDB AI operator (google_ml.if) +kind: tool +name: check_ride_suitability +type: postgres-sql +source: disney-db +description: "Evaluates if a specific attraction is safe or suitable based on a guest's profile (e.g., 'pregnant women' or 'toddlers')." +parameters: + - name: attraction_name + type: string + description: "Name of the attraction" + - name: suitability_profile + type: string + description: "Profile of the guest (e.g., 'pregnant women', 'toddlers')" +statement: | + -- TODO: Call your check_attraction_suitability function with the appropriate parameters +--- +# Tool 3: Transactional Tool to record new reviews +kind: tool +name: add_attraction_review +type: postgres-sql +source: disney-db +description: "Saves a new customer review for an attraction into the operational database." +parameters: + - name: rating + type: integer + description: "Numerical rating out of 5" + - name: review_text + type: string + description: "Customer review text" + - name: branch + type: string + description: "The park branch location" +statement: | + -- TODO: Write an INSERT statement that records a new review into the disneyland_reviews table. +--- +# Tool 4: Analytical Tool checking Wait Time Forecasts (Local Table) +kind: tool +name: get_wait_time_forecast +type: postgres-sql +source: disney-db +description: "Queries the local database to get forecasted wait times for a specific attraction." +parameters: + - name: attraction_id + type: integer + description: "Unique ID of the attraction" +statement: | + -- TODO: Query the local table public.forecasted_waiting_times for the attraction's predicted wait time +--- +# Tool 5: Analytical Tool checking Graph Recommendations (Local Table) +kind: tool +name: get_next_ride_recommendation +type: postgres-sql +source: disney-db +description: "Gets next-ride routing recommendations for a guest leaving a specific attraction to avoid queues." +parameters: + - name: attraction_id + type: integer + description: "Unique ID of the attraction" +statement: | + -- TODO: Query the local table public.graph_recommendations to retrieve recommendations +--- +# Tool 6: QueryData Natural Language Search +kind: tool +name: query_disney_data +type: cloud-gemini-data-analytics-query +source: gda-api-source +description: "Use this tool to ask natural language questions about the Disneyland reviews, attractions, and wait times. It will translate your question into SQL, execute it, and return the answer." +location: "europe-west1" +context: + datasourceReferences: + alloydb: + databaseReference: + projectId: "[YOUR_PROJECT_ID]" + region: "europe-west1" + clusterId: "[YOUR_CLUSTER]" + instanceId: "[YOUR_INSTANCE]" + databaseId: "disney" + agentContextReference: + # Tip: You can find this ID in AlloyDB Studio by clicking on 'Edit Context' + # It should be in this format: projects/[YOUR_PROJECT_ID]/locations/[LOCATION]/contextSets/[CONTEXT_NAME] + contextSetId: "projects/[YOUR_PROJECT_ID]/locations/europe-west4/contextSets/disney-context" +generationOptions: + generateQueryResult: true + generateNaturalLanguageAnswer: true + generateExplanation: true + generateDisambiguationQuestion: true + +--- +# ========================================== +# TOOLSET +# ========================================== +kind: toolset +name: disneyland_operational_tools +tools: + - search_attractions_hybrid + - check_ride_suitability + - add_attraction_review + - get_wait_time_forecast + - get_next_ride_recommendation + - query_disney_data +``` + +#### Task 9.3: Start and Validate the Server + +Before running the toolbox, you must authenticate your local environment and configure it to impersonate the dedicated Agent Service Account. This allows the toolbox to call the QueryData API using the service account's identity: + +```bash +# 1. Authenticate and set up service account impersonation +gcloud auth application-default login --impersonate-service-account=disney-agent-sa@.iam.gserviceaccount.com + +# 2. Start the toolbox +./toolbox --config tools.yaml --ui +``` + +Open the visual web interface in Cloud Shell (default port is 5000), execute each of the tools, and verify that they are pulling/pushing data successfully. + +> [!TIP] +> When using the Cloud Shell web preview, be sure to append `/ui` to the end of the URL in your browser to access the MCP toolbox interface. + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show the **MCP Toolbox UI** with the six tools (`search_attractions_hybrid`, `check_ride_suitability`, `add_attraction_review`, `get_wait_time_forecast`, `get_next_ride_recommendation`, and `query_disney_data`) defined and tested successfully (all showing a green status). + +--- + +## Challenge 10: Building the guest assistant app + +**Target Persona:** Full-Stack AI / App Developer | **Estimated Duration:** 90 minutes | *Prerequisites: Challenges 2, 7, 8, and 9 must be completed.* + +### Introduction + +This is the final integration and application challenge! Because the entire database agentic layer—including BigQuery FDW data sync, operational/analytical SQL tools, and MCP Toolbox—has already been securely structured in Challenges 7, 8, and 9, this challenge focuses exclusively on the developer's magic: constructing the conversational guest assistant, vibe-coding a premium web application, and deploying it to **Cloud Run**. + +![Challenge 10 Architecture](images/challenge10_architecture.png) + +### Description + +#### Task 10.1: Scaffold the Guest Assistant with ADK + +Using the **Agent Development Kit (ADK)**, you will construct the conversational agent that consumes your MCP tools. + +1. **Create `agent.py`:** + Set up the agent, pointing it to your local MCP server to load the toolset: + + ```python + from google.adk.agents import Agent + from toolbox_core import ToolboxSyncClient + + # 1. Connect to the MCP server running on port 5000 + # TODO: Initialize the ToolboxSyncClient pointing to your local MCP server + toolbox = ... + + # 2. Load all tools (operational + analytical) + # TODO: Load the 'disneyland_operational_tools' toolset from the MCP client + disney_tools = ... + + # 3. Define the Guest Guide Agent + # TODO: Configure your Agent named 'disney_guide_agent'. Use the 'gemini-2.5-flash' model, + # write a rich, magical system instruction guiding the assistant on its persona and tool usage, + # and link the loaded tools to the agent. + visitor_guide = Agent( + ... + ) + ``` + +#### Task 10.2: Vibe-Coding a Premium Web Application + +Rather than a generic, plain interface, you will **vibe-code a stunning, premium web application** that hooks into your ADK agent. + +**Leverage the Google AI Stack for Vibe-Coding:** + +- **Stitch:** Use Stitch to rapidly design and iterate on the premium web interface (dark modes, glassmorphism, animations) and export production-ready components. +- **Google Antigravity 2.0 & CLI:** Use the `antigravity` CLI and its Agentic IDE capabilities to autonomously scaffold and vibe-code the frontend logic, hooking it directly to your ADK agent. +- **Google AI Studio:** Prototype, experiment, and fine-tune any complex conversational interactions or multimodal prompts before integrating them into your codebase. + +#### Task 10.3: Deploy to Google Cloud Run (Optional) + +To complete the gHack and make the guest assistant publicly accessible, you will containerize and push the application to **Cloud Run**. + +1. **Write the Dockerfile:** + Create a `Dockerfile` that packages both the ADK python backend (exposing the agent API) and the built frontend assets. +2. **Deploy Command:** + Use `gcloud run deploy` to push the application in one step, making sure to associate it with the dedicated Agent Service Account so it can authenticate to AlloyDB QueryData: + + ```bash + gcloud run deploy disneyland-guest-assistant \ + --source . \ + --platform managed \ + --region europe-west1 \ + --service-account=disney-agent-sa@.iam.gserviceaccount.com \ + --allow-unauthenticated + ``` + +### Success Criteria + +To validate this challenge, you must demonstrate the following: + +- Show a screenshot or proof of the **Vibe-Coded Web App** running, showcasing a premium design with glassmorphism, animations, and a rich, responsive layout. +- Show a full conversation demonstration in your application UI where the agent uses hybrid search, checks wait times, recommends a next-ride, and records a review—all working flawlessly in one session. + +--- + +**Congratulations! You have completed the Disneyland Agentic Data Cloud gHack! 🏆** +You have built a state-of-the-art, end-to-end agentic data pipeline on Google Cloud. Have a magical day! 🪄✨ diff --git a/hacks/disneyland-on-gcp/artifacts/Makefile b/hacks/disneyland-on-gcp/artifacts/Makefile new file mode 100644 index 00000000..344010ba --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/Makefile @@ -0,0 +1,21 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +TARGET=ghacks-setup.zip +OBJS=*.tf runtime.yaml + +$(TARGET): $(OBJS) + zip -r $(TARGET) $(OBJS) + +clean: + rm -f $(TARGET) diff --git a/hacks/disneyland-on-gcp/artifacts/main.tf b/hacks/disneyland-on-gcp/artifacts/main.tf new file mode 100644 index 00000000..b5602214 --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/main.tf @@ -0,0 +1,261 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +resource "google_project_service" "default" { + project = var.gcp_project_id + for_each = toset([ + "cloudresourcemanager.googleapis.com", + "serviceusage.googleapis.com", + "iam.googleapis.com", + "storage.googleapis.com", + "compute.googleapis.com", + "alloydb.googleapis.com", + "bigquery.googleapis.com", + "bigqueryconnection.googleapis.com", + "datastream.googleapis.com", + "aiplatform.googleapis.com", + "dataplex.googleapis.com", + "artifactregistry.googleapis.com", + "run.googleapis.com", + "servicenetworking.googleapis.com", + "cloudaicompanion.googleapis.com", + "geminicloudassist.googleapis.com", + "geminidataanalytics.googleapis.com" + ]) + service = each.key + + disable_on_destroy = false +} + +# In case a default network is not present in the project, this variable needs to be set +resource "google_compute_network" "default_network_created" { + name = "default" + auto_create_subnetworks = true + count = var.create_default_network ? 1 : 0 + depends_on = [ + google_project_service.default + ] +} + +# Enabling comms between VMs for auto mode subnets (needed by Dataproc/Dataflow workers) +resource "google_compute_firewall" "fwr_allow_custom" { + name = "fwr-ingress-allow-custom" + network = google_compute_network.default_network_created[0].self_link + count = var.create_default_network ? 1 : 0 + source_ranges = ["10.128.0.0/9"] + allow { + protocol = "all" + } +} + +# Enabling Identity-Aware-Proxy for TCP forwarding (for SSH access from Console) +resource "google_compute_firewall" "fwr_allow_iap" { + name = "fwr-ingress-allow-iap" + network = google_compute_network.default_network_created[0].self_link + count = var.create_default_network ? 1 : 0 + source_ranges = ["35.235.240.0/20"] + allow { + protocol = "tcp" + ports = ["22"] + } +} + +# This piece of code makes it possible to deal with the default network the same way, +# regardless of how it has been created. Make sure to refer to the default network through +# this resource when needed. +data "google_compute_network" "default_network" { + name = "default" + depends_on = [ + google_project_service.default, + google_compute_network.default_network_created + ] +} + +# Reserve a range for Private Service Access +resource "google_compute_global_address" "private_ip_alloc" { + name = "alloydb-private-ip-alloc" + purpose = "VPC_PEERING" + address_type = "INTERNAL" + prefix_length = 16 + network = data.google_compute_network.default_network.id +} + +# Create the VPC Peering connection +resource "google_service_networking_connection" "vpc_connection" { + network = data.google_compute_network.default_network.id + service = "servicenetworking.googleapis.com" + reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name] + depends_on = [google_project_service.default] +} + +# Random password for the AlloyDB database user +resource "random_string" "password" { + length = 12 + special = false +} + +# AlloyDB Cluster +resource "google_alloydb_cluster" "default" { + cluster_id = "disney-cluster" + location = var.gcp_region + + network_config { + network = data.google_compute_network.default_network.id + } + + initial_user { + user = "postgres" + password = random_string.password.result + } + + depends_on = [ + google_project_service.default, + google_service_networking_connection.vpc_connection + ] +} + +resource "google_alloydb_instance" "default" { + cluster = google_alloydb_cluster.default.name + instance_id = "disney-instance" + instance_type = "PRIMARY" + + database_flags = { + "alloydb.logical_decoding" = "on" # needed for replication + "bigquery_fdw.enabled" = "on" # needed for connecting to bq + "alloydb.iam_authentication" = "on" + } + + machine_config { + cpu_count = 2 + } + + network_config { + enable_public_ip = true + authorized_external_networks { + cidr_range = "0.0.0.0/0" + } + } + + depends_on = [google_alloydb_cluster.default] +} + +# Storage Bucket +resource "google_storage_bucket" "disney" { + name = var.gcp_project_id + location = var.gcp_region + uniform_bucket_level_access = true +} + +# BigQuery Dataset +resource "google_bigquery_dataset" "disney" { + dataset_id = "disney" + friendly_name = "Disneyland Data" + description = "Dataset for Disneyland Data Cloud" + location = var.gcp_region + delete_contents_on_destroy = true + + depends_on = [google_project_service.default] +} + +resource "google_bigquery_reservation" "bq_edition" { + name = "disneyland-analytics" + location = var.gcp_region + edition = "ENTERPRISE" + slot_capacity = 0 + + autoscale { + max_slots = 100 + } +} + +resource "google_bigquery_reservation_assignment" "default" { + assignee = "projects/${var.gcp_project_id}" + reservation = google_bigquery_reservation.bq_edition.id + job_type = "QUERY" +} + +# BigQuery Cloud Resource Connection +resource "google_bigquery_connection" "agent_platform" { + connection_id = "conn" + location = var.gcp_region + friendly_name = "Connection to Agent Platform" + cloud_resource {} + + depends_on = [google_project_service.default] +} + +# Grant Agent Platform User and Storage Viewer to BigQuery Connection Service Account +resource "google_project_iam_member" "bq_connection_roles" { + for_each = toset([ + "roles/aiplatform.user", + "roles/storage.objectViewer" + ]) + project = var.gcp_project_id + role = each.key + member = "serviceAccount:${google_bigquery_connection.agent_platform.cloud_resource[0].service_account_id}" +} + +# AlloyDB Service Agent +resource "google_project_service_identity" "alloydb_sa" { + provider = google-beta + project = var.gcp_project_id + service = "alloydb.googleapis.com" +} + +# Grant Agent Platform User, BQ Viewer and BQ Read Session User to AlloyDB Service Agent +resource "google_project_iam_member" "alloydb_sa_roles" { + for_each = toset([ + "roles/aiplatform.user", + # "roles/bigquery.dataViewer", # these need to be granted to another service account + # "roles/bigquery.readSessionUser" + ]) + project = var.gcp_project_id + role = each.key + member = "serviceAccount:${google_project_service_identity.alloydb_sa.email}" +} + +# ========================================================================= +# Dedicated SA for the agent & QueryData +# ========================================================================= + +# 1. Create the Service Account +resource "google_service_account" "agent_sa" { + account_id = "disney-agent-sa" + display_name = "Disneyland Guest Assistant Agent Service Account" +} + +# 2. Grant required roles to the Service Account +resource "google_project_iam_member" "agent_sa_roles" { + for_each = toset([ + "roles/geminidataanalytics.queryDataUser", + "roles/alloydb.databaseUser", + "roles/serviceusage.serviceUsageConsumer", + "roles/aiplatform.user" + ]) + project = var.gcp_project_id + role = each.key + member = "serviceAccount:${google_service_account.agent_sa.email}" +} + +# 3. Allow all Project Editors and Owners to impersonate this Service Account +resource "google_service_account_iam_binding" "sa_impersonation" { + service_account_id = google_service_account.agent_sa.name + role = "roles/iam.serviceAccountTokenCreator" + + members = [ + "projectEditor:${var.gcp_project_id}", + "projectOwner:${var.gcp_project_id}" + ] +} + diff --git a/hacks/disneyland-on-gcp/artifacts/outputs.tf b/hacks/disneyland-on-gcp/artifacts/outputs.tf new file mode 100644 index 00000000..444799ae --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/outputs.tf @@ -0,0 +1,30 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "project_id" { + value = var.gcp_project_id +} + +output "alloydb_proxy_ip" { + value = google_compute_instance.gce_tcp_proxy.network_interface[0].access_config[0].nat_ip +} + +output "alloydb_usr" { + value = google_alloydb_cluster.default.initial_user[0].user +} + +output "alloydb_pwd" { + value = google_alloydb_cluster.default.initial_user[0].password + sensitive = true +} diff --git a/hacks/disneyland-on-gcp/artifacts/providers.tf b/hacks/disneyland-on-gcp/artifacts/providers.tf new file mode 100644 index 00000000..067e274c --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/providers.tf @@ -0,0 +1,37 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +terraform { + required_providers { + google = { + source = "hashicorp/google" + version = "~> 7.24.0" + } + google-beta = { + source = "hashicorp/google-beta" + version = "~> 7.24.0" + } + } +} + +provider "google" { + project = var.gcp_project_id + region = var.gcp_region + zone = var.gcp_zone +} + +provider "google-beta" { + project = var.gcp_project_id + region = var.gcp_region + zone = var.gcp_zone +} diff --git a/hacks/disneyland-on-gcp/artifacts/runtime.yaml b/hacks/disneyland-on-gcp/artifacts/runtime.yaml new file mode 100644 index 00000000..29bb0065 --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/runtime.yaml @@ -0,0 +1,15 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +runtime: terraform +version: 1.4.6 diff --git a/hacks/disneyland-on-gcp/artifacts/variables.tf b/hacks/disneyland-on-gcp/artifacts/variables.tf new file mode 100644 index 00000000..9e4d640b --- /dev/null +++ b/hacks/disneyland-on-gcp/artifacts/variables.tf @@ -0,0 +1,38 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +variable "gcp_project_id" { + type = string + description = "The GCP project ID to create resources in." +} + +# Default value passed in +variable "gcp_region" { + type = string + description = "Region to create resources in." + default = "us-central1" +} + +# Default value passed in +variable "gcp_zone" { + type = string + description = "Zone to create resources in." + default = "us-central1-a" +} + +# Relevant when running in an environment where no default network exists yet +variable "create_default_network" { + type = bool + default = false + description = "Whether to create a default network with subnets for all regions" +} diff --git a/hacks/disneyland-on-gcp/images/.gitkeep b/hacks/disneyland-on-gcp/images/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/hacks/disneyland-on-gcp/images/challenge10_architecture.png b/hacks/disneyland-on-gcp/images/challenge10_architecture.png new file mode 100644 index 00000000..5166ebd4 Binary files /dev/null and b/hacks/disneyland-on-gcp/images/challenge10_architecture.png differ diff --git a/hacks/disneyland-on-gcp/images/ghack_challenge_architecture.png b/hacks/disneyland-on-gcp/images/ghack_challenge_architecture.png new file mode 100644 index 00000000..cb2e7f26 Binary files /dev/null and b/hacks/disneyland-on-gcp/images/ghack_challenge_architecture.png differ diff --git a/hacks/disneyland-on-gcp/images/parallelization_timeline.png b/hacks/disneyland-on-gcp/images/parallelization_timeline.png new file mode 100644 index 00000000..0633cdbb Binary files /dev/null and b/hacks/disneyland-on-gcp/images/parallelization_timeline.png differ diff --git a/hacks/disneyland-on-gcp/qwiklabs.yaml b/hacks/disneyland-on-gcp/qwiklabs.yaml new file mode 100644 index 00000000..1f20af08 --- /dev/null +++ b/hacks/disneyland-on-gcp/qwiklabs.yaml @@ -0,0 +1,133 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +schema_version: 2 +default_locale: en +title: "[gHacks] Disneyland Data Cloud" +description: "In this gHack, you're building your ultimate Disneyland planning tool. We've got the data: reviews + from visitors across global branches, historical waiting times, and attendance figures. Your mission? + Transform this raw data into actionable insights leveraging the power of AI and Data on Google Cloud." +instruction: + type: md + uri: instructions/en.md +duration: 420 +max_duration: 420 +credits: 0 +level: fundamental +tags: +product_tags: +role_tags: +domain_tags: +environment: + resources: + - type: gcp_project + id: project + variant: gcp_medium_extra + startup_script: + type: qwiklabs + path: artifacts + - type: gcp_user + id: user_1 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_2 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_3 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_4 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_5 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_6 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_7 + permissions: + - project: project + roles: + - roles/owner + - type: gcp_user + id: user_8 + permissions: + - project: project + roles: + - roles/owner + + student_visible_outputs: + - label: Open Console + reference: project.console_url + - label: Username 1 + reference: user_1.username + - label: Password 1 + reference: user_1.password + - label: Username 2 + reference: user_2.username + - label: Password 2 + reference: user_2.password + - label: Username 3 + reference: user_3.username + - label: Password 3 + reference: user_3.password + - label: Username 4 + reference: user_4.username + - label: Password 4 + reference: user_4.password + - label: Username 5 + reference: user_5.username + - label: Password 5 + reference: user_5.password + - label: Username 6 + reference: user_6.username + - label: Password 6 + reference: user_6.password + - label: Username 7 + reference: user_7.username + - label: Password 7 + reference: user_7.password + - label: Username 8 + reference: user_8.username + - label: Password 8 + reference: user_8.password + + - label: Project ID + reference: project.project_id + - label: AlloyDB IP + reference: project.startup_script.alloydb_ip + - label: AlloyDB User + reference: project.startup_script.alloydb_usr + - label: AlloyDB Password + reference: project.startup_script.alloydb_pwd + + diff --git a/hacks/disneyland-on-gcp/resources/Makefile b/hacks/disneyland-on-gcp/resources/Makefile new file mode 100644 index 00000000..726b6c70 --- /dev/null +++ b/hacks/disneyland-on-gcp/resources/Makefile @@ -0,0 +1,21 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +TARGET=student-files.zip +OBJS=*.* + +$(TARGET): $(OBJS) + zip -r $(TARGET) $(OBJS) + +clean: + rm -f $(TARGET) diff --git a/hacks/disneyland-on-gcp/resources/template-lectures.pdf b/hacks/disneyland-on-gcp/resources/template-lectures.pdf new file mode 100644 index 00000000..fe6e6e91 Binary files /dev/null and b/hacks/disneyland-on-gcp/resources/template-lectures.pdf differ diff --git a/hacks/disneyland-on-gcp/solutions.md b/hacks/disneyland-on-gcp/solutions.md new file mode 100644 index 00000000..6b9f3499 --- /dev/null +++ b/hacks/disneyland-on-gcp/solutions.md @@ -0,0 +1,1121 @@ +# Disneyland Agentic Data Cloud - Coach's Guide & Solutions + +Welcome to the coach's guide for the *Disneyland Agentic Data Cloud* gHack. This guide contains the complete reference implementations, SQL queries, configurations, and code blocks for all 9 challenges of the hackathon. + +> [!NOTE] +> If you are a gHacks participant, this is the answer guide. Don't cheat yourself by looking at this guide during the hack! + +## Coach's Guides + +* Challenge 1: Setting up AlloyDB and replicating data to BigQuery +* Challenge 2: Creating the agentic database layer +* Challenge 3: Sentiment and wait-time forecasting +* Challenge 4: Image classification and brochure RAG +* Challenge 5: Graph analytics and visitor flow +* Challenge 6: Preparing the Context Layer +* Challenge 7: Conversational analytics for insights +* Challenge 8: From insights to action, syncing BigQuery and AlloyDB +* Challenge 9: Exposing Database Tools via MCP +* Challenge 10: Building the guest assistant app + +--- + +## Challenge 1: Setting up AlloyDB and replicating data to BigQuery + +### 1.1 Ingest Data into AlloyDB + +#### 1. Create a Dedicated Database + +Connect to the default `postgres` database in AlloyDB Studio or `psql`, and run: + +```sql +CREATE DATABASE disney; +``` + +Once created, **disconnect and reconnect** to the new `disney` database. All subsequent tables, extensions, and queries must be run within this `disney` database. + +#### 2. Create the Tables + +Create the two primary operational tables with primary keys: + +```sql +CREATE TABLE disneyland_reviews ( + review_id INT PRIMARY KEY, + rating INT, + year_month TEXT, + reviewer_location TEXT, + review_text TEXT, + branch TEXT +); + +CREATE TABLE disneyland_attractions ( + attraction_id INT PRIMARY KEY, + branch TEXT, + name TEXT, + description TEXT +); + +CREATE TABLE visitor_movements ( + visitor_id INT, + from_attraction_id INT, + to_attraction_id INT, + timestamp TIMESTAMP, + PRIMARY KEY (visitor_id, timestamp) +); +``` + +#### 3. Add a Full-Text Search Vector + +Add a generated `tsvector` column to the attractions table to support hybrid search: + +```sql +ALTER TABLE disneyland_attractions +ADD COLUMN description_tsvector tsvector +GENERATED ALWAYS AS (to_tsvector('english', description)) STORED; +``` + +#### 4. Import Data from Cloud Storage + +Use the AlloyDB Import API (via the Cloud Console UI or `gcloud`) to import the public CSV files: + +* Import `gs://ghacks-disneyland-on-gcp/reviews.csv` into `disneyland_reviews` +* Import `gs://ghacks-disneyland-on-gcp/attractions.csv` into `disneyland_attractions` +* Import `gs://ghacks-disneyland-on-gcp/visitor_movements.csv` into `visitor_movements` + +--- + +### 1.2 Generate Vector Embeddings at the Source + +#### 1. Enable Extensions + +```sql +CREATE EXTENSION IF NOT EXISTS vector CASCADE; +CREATE EXTENSION IF NOT EXISTS google_ml_integration CASCADE; +``` + +#### 2. Add the Embedding Column + +Add a vector column of size 3072 (corresponding to the dimension of `gemini-embedding-001`): + +```sql +ALTER TABLE disneyland_attractions ADD COLUMN embedding vector(3072); +``` + +#### 3. Generate Embeddings + +Populate the column by calling the Vertex AI embedding model natively: + +```sql +UPDATE disneyland_attractions +SET embedding = google_ml.embedding('gemini-embedding-001', description)::vector; +``` + +#### 4. Verify Similarity Search (Checkpoint Validation) + +To verify the embeddings, run a similarity search for the top 5 attractions closest to `'thrilling dark ride in space'`: + +```sql +SELECT name, description, 1 - (embedding <=> google_ml.embedding('gemini-embedding-001', 'thrilling dark ride in space')::vector) AS similarity +FROM disneyland_attractions +ORDER BY embedding <=> google_ml.embedding('gemini-embedding-001', 'thrilling dark ride in space')::vector ASC +LIMIT 5; +``` + +--- + +### 1.3 Set Up Real-Time Replication with One-Click Datastream + +1. In the Google Cloud Console, navigate to **AlloyDB > Clusters**. +2. Select your cluster and click **Replicate data to BigQuery** on the top menu. +3. Follow the wizard: + * **Region**: Same as your AlloyDB cluster (e.g., `us-central1` or `europe-west1`). + * **Tables**: Select `disneyland_reviews`, `disneyland_attractions`, and `visitor_movements`. + * **Write Mode**: **Merge**. + * **Staleness Limit**: **0 seconds** (real-time). + * **Destination Dataset**: **disney**. +4. Start the stream. + +*(Optional: If configuring logical replication manually on the database)*: + +```sql +CREATE PUBLICATION pub_disney FOR TABLE disneyland_reviews, disneyland_attractions, visitor_movements; +ALTER USER postgres WITH REPLICATION; +SELECT PG_CREATE_LOGICAL_REPLICATION_SLOT('slot_disney', 'pgoutput'); +``` + +--- + +### 1.4 Prove the Flow with Data Canvas + +1. In the Google Cloud Console, navigate to **BigQuery Studio**. +2. Click the **Data Canvas** tab. +3. Load the replicated `disney.public_disneyland_reviews` table. +4. Click **Visualize** or **Analyze** to generate a quick chart (e.g., a bar chart of the average rating per branch) to prove that the data has replicated successfully. + +--- + +## Challenge 2: Creating the agentic database layer + +### 2.1 QueryData Context Set + +Create a file named `querydata_disney_context.json` with the following configuration and upload it in the AlloyDB Console under **Context Sets** (named `disney-context`): + +```json +{ + "templates": [ + { + "nlQuery": "Show available attractions in Disneyland Paris", + "sql": "SELECT name, description FROM public.disneyland_attractions WHERE branch = 'Disneyland_Paris'", + "intent": "List all attractions for a specific park branch", + "manifest": "List attractions by branch", + "parameterized": { + "parameterized_intent": "Show available attractions in $1", + "parameterized_sql": "SELECT name, description FROM public.disneyland_attractions WHERE branch = $1" + } + }, + { + "nlQuery": "Find reviews with rating 5 for Space Mountain", + "sql": "SELECT r.review_id, r.rating, r.review_text FROM public.disneyland_reviews r INNER JOIN public.disneyland_attractions a ON r.branch = a.branch WHERE a.name = 'Space Mountain' AND r.rating = 5", + "intent": "Get reviews with a specific rating for a named attraction", + "manifest": "Get reviews by attraction and rating", + "parameterized": { + "parameterized_intent": "Find reviews with rating $2 for $1", + "parameterized_sql": "SELECT r.review_id, r.rating, r.review_text FROM public.disneyland_reviews r INNER JOIN public.disneyland_attractions a ON r.branch = a.branch WHERE a.name = $1 AND r.rating = $2" + } + }, + { + "nlQuery": "Average rating of attractions in California Adventure", + "sql": "SELECT AVG(rating) FROM public.disneyland_reviews WHERE branch = 'Disneyland_California'", + "intent": "Calculate the average review rating for a specific branch", + "manifest": "Average rating by branch", + "parameterized": { + "parameterized_intent": "Average rating of attractions in $1", + "parameterized_sql": "SELECT AVG(rating) FROM public.disneyland_reviews WHERE branch = $1" + } + } + ], + "facets": [ + { + "sql_snippet": "r.rating >= 4", + "intent": "highly rated reviews", + "manifest": "Filter reviews by a minimum rating threshold", + "parameterized": { + "parameterized_intent": "reviews with rating greater than or equal to $1", + "parameterized_sql_snippet": "r.rating >= $1" + } + } + ], + "value_searches": [ + { + "query": "SELECT DISTINCT T.name as value, 'public.disneyland_attractions.name' as columns, 'Attraction Name' as concept_type, (1.0 - ts_rank(to_tsvector('english', T.name), plainto_tsquery('english', $value))) as distance, '{}'::text as context FROM public.disneyland_attractions T WHERE to_tsvector('english', T.name) @@ plainto_tsquery('english', $value)", + "concept_type": "Attraction Name", + "description": "Full-text search for attraction names" + }, + { + "query": "SELECT DISTINCT T.branch as value, 'public.disneyland_attractions.branch' as columns, 'Branch Name' as concept_type, (1.0 - ts_rank(to_tsvector('english', T.branch), plainto_tsquery('english', $value))) as distance, '{}'::text as context FROM public.disneyland_attractions T WHERE to_tsvector('english', T.branch) @@ plainto_tsquery('english', $value)", + "concept_type": "Branch Name", + "description": "Full-text search for park branches" + } + ] +} +``` + +--- + +### 2.2 Expose AlloyDB AI Operators + +#### 1. Install Extensions and Create Indexes for Hybrid Search + +```sql +-- Install required extensions +CREATE EXTENSION IF NOT EXISTS alloydb_scann CASCADE; +CREATE EXTENSION IF NOT EXISTS rum CASCADE; +-- Index RUM for Keyword FTS +CREATE INDEX IF NOT EXISTS attractions_tsvector_idx ON disneyland_attractions USING RUM (description_tsvector rum_tsvector_ops); + +-- Index ScaNN for Vector Cosine Similarity +CREATE INDEX IF NOT EXISTS attractions_vector_idx ON disneyland_attractions USING scann (embedding cosine) WITH (num_leaves=10); +``` + +#### 2. Create the Semantic Filtering Function + +Define a custom SQL function utilizing `google_ml.if` to filter attractions semantically: + +```sql +CREATE OR REPLACE FUNCTION check_attraction_suitability(attraction_name TEXT, suitability_profile TEXT) +RETURNS TABLE(name TEXT, description TEXT) AS $$ + SELECT a.name, a.description + FROM disneyland_attractions a + WHERE a.name = attraction_name + AND google_ml.if( + prompt => 'Is this attraction ' || suitability_profile || '? Description: ' || a.description + ); +$$ LANGUAGE SQL; +``` + +#### 3. Test and Validate the Capabilities + +To verify these features in AlloyDB Studio, you can run the following test queries: + +**Hybrid Search Test:** + +```sql +ALTER DATABASE disney SET google_ml_integration.enable_preview_ai_functions = 'on'; +SELECT a.name, a.description, search_results.score +FROM disneyland_attractions a +JOIN ai.hybrid_search( + search_inputs => ARRAY[ + '{ + "data_type": "vector", + "table_name": "disneyland_attractions", + "key_column": "attraction_id", + "vec_column": "embedding", + "distance_operator": "public.<=>", + "limit": 5, + "query_vector": "ai.embedding(''gemini-embedding-001'', ''thrilling space roller coaster'')::vector" + }'::JSONB, + '{ + "data_type": "text", + "table_name": "disneyland_attractions", + "key_column": "attraction_id", + "text_column": "description_tsvector", + "limit": 5, + "ranking_function": "<=>", + "query_text_input": "thrilling space roller coaster" + }'::JSONB + ], + id_type => NULL::BIGINT +) AS search_results ON a.attraction_id = search_results.id; +``` + +**Semantic Filtering Function Test:** + +```sql +-- This should return 0 rows (Space Mountain is not safe for pregnant women) +SELECT * FROM check_attraction_suitability('Space Mountain', 'safe for pregnant women'); + +-- This should return 1 row (it's a small world is suitable for toddlers) +SELECT * FROM check_attraction_suitability('it''s a small world', 'suitable for toddlers'); +``` + +--- + +## Challenge 3: Sentiment and wait-time forecasting + +### 3.1 Automated Sentiment Analysis with BQ Studio Data Science Agent + +> [!IMPORTANT] +> **Dependency Note:** +> This task queries the `disneyland_reviews` table in BigQuery, which is replicated from AlloyDB. This requires **Challenge 1** (specifically the Datastream replication in Task 1.3) to be completed first. + +#### 1. Create the Remote Model + +```sql +CREATE OR REPLACE MODEL `disney.gemini_flash` + REMOTE WITH CONNECTION `us-central1.conn` + OPTIONS (ENDPOINT = 'gemini-2.5-flash'); +``` + +#### 2. Classify Sentiments (Sample of 100) + +Using the BigQuery Studio Data Science Agent, generate and run the following query: + +```sql +CREATE OR REPLACE TABLE `disney.reviews_sentiment_analysis` AS +SELECT + review_id, rating, year_month, reviewer_location, review_text, branch, + ml_generate_text_result AS sentiment +FROM + ML.GENERATE_TEXT( + MODEL `disney.gemini_flash`, + ( + SELECT *, + 'Classify the sentiment of this review as Positive, Negative, or Neutral. Output ONLY the single word. Review: ' || review_text AS prompt + FROM `disney.public_disneyland_reviews` + LIMIT 100 + ), + STRUCT(0.0 AS temperature, 10 AS max_output_tokens) + ); +``` + +--- + +### 3.2 Time-Series Wait Time Forecasting + +#### 1. Load Data + +```sql +LOAD DATA OVERWRITE `disney.waiting_times` +FROM FILES ( + format = 'CSV', + uris = ['gs://ghacks-disneyland-on-gcp/waiting_times.csv'] +); +``` + +#### 2. Train the ARIMA_PLUS Model + +```sql +CREATE OR REPLACE MODEL `disney.waiting_time_forecast_model` + OPTIONS( + model_type='ARIMA_PLUS', + time_series_timestamp_col='time_bucket', + time_series_data_col='avg_wait_time', + time_series_id_col='attraction', + data_frequency='AUTO_FREQUENCY' + ) AS + SELECT + TIMESTAMP_SECONDS(1800 * DIV(UNIX_SECONDS(timestamp), 1800)) AS time_bucket, -- 30-min intervals + attraction, + AVG(waiting_time) AS avg_wait_time + FROM `disney.waiting_times` + GROUP BY 1, 2; +``` + +#### 3. Generate 24-Hour Forecast (48 intervals of 30 mins) + +Join the forecast results with the attractions table to resolve the `attraction_id` for FDW compatibility: + +```sql +CREATE OR REPLACE TABLE `disney.forecasted_waiting_times` AS +SELECT + a.attraction_id, + f.forecast_timestamp AS forecasted_timestamp, + f.forecast_value AS predicted_wait_time +FROM + ML.FORECAST(MODEL `disney.waiting_time_forecast_model`, STRUCT(48 AS horizon, 0.95 AS confidence_level)) f +JOIN + `disney.public_disneyland_attractions` a ON f.attraction = a.name; +``` + +--- + +### 3.3 Ride Clustering (Intensity & Popularity) + +#### 1. Categorize Attractions using `AI.CLASSIFY` + +```sql +CREATE OR REPLACE TABLE `disney.thrill_class` AS +SELECT + attraction_id, + name, + AI.CLASSIFY( + description, + categories => ['easy-peasy', 'thrilling', 'extreme'] + ) AS class +FROM + `disney.public_disneyland_attractions`; +``` + +#### 2. Rank Attractions using `AI.SCORE` + +```sql +CREATE OR REPLACE TABLE `disney.thrill_score` AS +SELECT + attraction_id, + name, + CAST(AI.SCORE( + 'Score attractions on a thrill level from 1 to 10 based on their description. Description: ' || description + ) AS INT64) AS rank +FROM + `disney.public_disneyland_attractions`; +``` + +--- + +## Challenge 4: Image classification and brochure RAG + +### 4.1 Multimodal Image Classification + +#### 1. Create the Object Table + +```sql +CREATE OR REPLACE EXTERNAL TABLE `disney.attraction_images` +WITH CONNECTION `us-central1.conn` +OPTIONS ( + object_metadata = 'SIMPLE', + uris = ['gs://ghacks-disneyland-on-gcp/attraction_parc_photos/*'] +); +``` + +#### 2. Classify Images + +```sql +CREATE OR REPLACE TABLE `disney.images_classification` AS +SELECT + uri, + ml_generate_text_result AS classification_json +FROM + ML.GENERATE_TEXT( + MODEL `disney.gemini_flash`, + TABLE `disney.attraction_images`, + STRUCT( + 'Is this image from a Disneyland park? Answer with a JSON object containing keys ''is_disneyland'' (boolean) and ''reason'' (string).' AS prompt + ) + ); +``` + +--- + +### 4.2 PDF Brochure Ingestion & RAG Pipeline + +#### 1. Create the Object Table for PDFs + +```sql +CREATE OR REPLACE EXTERNAL TABLE `disney.brochures_pdf` +WITH CONNECTION `us-central1.conn` +OPTIONS ( + object_metadata = 'SIMPLE', + uris = ['gs://ghacks-disneyland-on-gcp/disneyland_brochures/*.pdf'] +); +``` + +#### 2. Chunk PDFs using Pre-provided UDF + +```sql +CREATE OR REPLACE TABLE disney.brochures_chunks AS +SELECT + uri, chunk +FROM + `disney.brochures_pdf`, + UNNEST( + disney.chunk_pdf( + TO_JSON_STRING( + OBJ.GET_ACCESS_URL(OBJ.MAKE_REF(uri, 'us-central1.conn'), 'r') + ), + 1000, + 100 + ) + ) AS chunk; +``` + +#### 3. Generate Embeddings + +```sql +CREATE OR REPLACE MODEL `disney.embedding_model` + REMOTE WITH CONNECTION `us-central1.conn` + OPTIONS (ENDPOINT = 'gemini-embedding-001'); + +CREATE OR REPLACE TABLE disney.brochure_embeddings AS +SELECT * +FROM + AI.GENERATE_EMBEDDING( + MODEL `disney.embedding_model`, + ( + SELECT uri, chunk AS content FROM disney.brochures_chunks + ) + ); +``` + +--- + +### 4.3 Vector Search & RAG Validation + +Run a single, end-to-end RAG query that finds the relevant chunks using vector search, aggregates them, and passes them to Gemini Flash to answer: *"Where to eat a tex-mex meal buffet-style?"*: + +```sql +WITH context_chunks AS ( + SELECT base.content AS chunk + FROM + VECTOR_SEARCH( + TABLE `disney.brochure_embeddings`, + 'embedding', + ( + SELECT embedding + FROM + AI.GENERATE_EMBEDDING( + MODEL `disney.embedding_model`, + (SELECT 'Where to eat a tex-mex meal buffet-style?' AS content) + ) + ), + top_k => 3 + ) +), +assembled_context AS ( + SELECT STRING_AGG(chunk, '\n\n') AS context FROM context_chunks +) +SELECT + ml_generate_text_result AS grounded_response +FROM + ML.GENERATE_TEXT( + MODEL `disney.gemini_flash`, + ( + SELECT + 'Answer the following question using only the provided context. If the answer cannot be found in the context, say "I don''t know".\n\nQuestion: Where to eat a tex-mex meal buffet-style?\n\nContext:\n' || context AS prompt + FROM assembled_context + ) + ); +``` + +--- + +## Challenge 5: Graph analytics and visitor flow + +### 5.1 Build a Property Graph in BigQuery + +> [!IMPORTANT] +> **Dependency Note:** +> Creating the Property Graph requires both `public_disneyland_attractions` and `public_visitor_movements` tables to exist in BigQuery. This requires **Challenge 1** (Datastream replication) to be completed first. + +Define a property graph over the attractions and movements. Specify explicit keys for the node and edge tables: + +```sql +CREATE OR REPLACE PROPERTY GRAPH disney.disney_movement_graph +NODE TABLES ( + disney.public_disneyland_attractions + KEY (attraction_id) + LABEL Attraction +) +EDGE TABLES ( + disney.public_visitor_movements + KEY (visitor_id, timestamp) + SOURCE KEY (from_attraction_id) REFERENCES public_disneyland_attractions (attraction_id) + DESTINATION KEY (to_attraction_id) REFERENCES public_disneyland_attractions (attraction_id) + LABEL Moved +); +``` + +--- + +### 5.2 Query the Graph for Patterns + +#### 1. Flow Analysis: Top 3 Rides after Space Mountain + +Identify where visitors head immediately after riding "Space Mountain": + +```sql +SELECT b_name, COUNT(*) AS transition_count +FROM GRAPH_TABLE( + disney.disney_movement_graph + MATCH (a:Attraction) -[e:Moved]-> (b:Attraction) + WHERE a.name = 'Space Mountain' + RETURN b.name AS b_name +) +GROUP BY b_name +ORDER BY transition_count DESC +LIMIT 3; +``` + +#### 2. Multi-Hop Journeys: Top 3-Ride Sequences from Space Mountain + +Identify the most common sequences of 3 rides starting from "Space Mountain" taken by the same visitor within a 2-hour window: + +```sql +SELECT mid_ride, end_ride, COUNT(*) AS journey_count +FROM GRAPH_TABLE( + disney.disney_movement_graph + MATCH (a:Attraction) -[e1:Moved]-> (b:Attraction) -[e2:Moved]-> (c:Attraction) + WHERE a.name = 'Space Mountain' + AND e1.visitor_id = e2.visitor_id + AND e1.timestamp < e2.timestamp + AND TIMESTAMP_DIFF(e2.timestamp, e1.timestamp, MINUTE) <= 120 + RETURN b.name AS mid_ride, c.name AS end_ride +) +GROUP BY mid_ride, end_ride +ORDER BY journey_count DESC +LIMIT 5; +``` + +--- + +### 5.3 Visitor Journey Tracking & Path Analysis + +#### 1. Specific Journey Tracking + +Trace the journey of visitor `'11613'` starting from `'Dumbo the Flying Elephant'` to `'Disneyland Railroad'` and count the total transitions: + +```sql +SELECT * FROM GRAPH_TABLE( + disney.disney_movement_graph + MATCH (src:Attraction) -[e:Moved {visitor_id: '11613'}]->{1,10} (dst:Attraction) + WHERE src.name = 'Dumbo the Flying Elephant' AND dst.name = 'Disneyland Railroad' + RETURN src.name AS start_ride, dst.name AS end_ride, ARRAY_LENGTH(e) AS total_hops +); +``` + +#### 2. Multi-Hop Journeys by Visitor + +Find all unique visitor IDs who traveled from `'Space Mountain'` to `'Indiana Jones Adventure'` in a 2-hop journey (Space Mountain -> intermediate ride -> Indiana Jones Adventure) where both transitions are made by the same visitor: + +```sql +SELECT DISTINCT visitor_id +FROM GRAPH_TABLE( + disney.disney_movement_graph + MATCH (a:Attraction {name: 'Space Mountain'}) -[e1:Moved]-> (b:Attraction) -[e2:Moved]-> (c:Attraction {name: 'Indiana Jones Adventure'}) + WHERE e1.visitor_id = e2.visitor_id + RETURN e1.visitor_id AS visitor_id +); +``` + +#### Task 5.4: Graph-Based Recommendations + +To extract the most frequent next rides and build the recommendation table, run the following query: + +```sql +CREATE OR REPLACE TABLE disney.graph_recommendations AS +SELECT + start_ride_id as attraction_id, + next_ride_id as recommended_next_attraction_id, + DENSE_RANK() OVER(PARTITION BY start_ride_id ORDER BY journey_count DESC) as recommendation_rank +FROM ( + SELECT + start_ride_id, + next_ride_id, + COUNT(*) as journey_count + FROM GRAPH_TABLE( + disney.disney_movement_graph + MATCH (a:Attraction) -[e:Moved]-> (b:Attraction) + RETURN a.attraction_id AS start_ride_id, b.attraction_id AS next_ride_id + ) + GROUP BY start_ride_id, next_ride_id +) +WHERE start_ride_id != next_ride_id; +``` + +--- + +## Challenge 6: Preparing the Context Layer + +This challenge is performed using the BigQuery and Dataplex console interfaces. + +### 6.1 Technical Metadata Enrichment + +Add descriptions to the columns of the `disneyland_reviews` table in the BigQuery schema editor to help the agent interpret data types and purposes (e.g., describing the vector embeddings or sentiment labels). + +### 6.2 Business Glossary Alignment + +1. Create a Business Glossary inside Dataplex. +2. Define key terms like "Rollercoaster" or "Premium visitor" (e.g., `Premium visitor: A visitor who left more than 2 reviews`). +3. Link the business terms to the target columns in BigQuery and AlloyDB tables. + +### 6.3 Automated profiling & quality + +1. Create and execute a Dataplex Data Profile scan on `disney.public_disneyland_reviews`. +2. Define Data Quality rules (e.g., checking that `rating` is between 1 and 5, or that `reviewer_location` is not null) and run a Data Quality scan. + +### 6.4 Automated GCS Metadata Generation + +1. Go to the BigQuery Metadata Curation tab. +2. Link your GCS object table pointing to brochures. +3. Run the automated tag inference to attach metadata tags to the PDF brochure files. + +### 6.5 Lookup Context API + +Verify you can query the LookupContext API. An example call: + +```bash +curl -X POST \ + -H "Authorization: Bearer \$(gcloud auth print-access-token)" \ + -H "Content-Type: application/json" \ + https://dataplex.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/YOUR_LOCATION/entryGroups/YOUR_ENTRY_GROUP/entries/YOUR_ENTRY_ID:lookup \ + -d '{"aspectTypes": ["business-context"]}' +``` + +--- + +## Challenge 7: Conversational analytics for insights + +### 7.1 Initialize Agent + +In BigQuery Studio, go to the **Agents** tab, create a new agent named `disney_park_analyst` and connect it to the `disney` dataset. + +### 7.2 Configure Knowledge Catalog + +* **Synonyms**: + * "rollercoaster", "thrill ride" -> "Space Mountain", "Big Thunder Mountain" + * "queue", "line" -> `waiting_time` + +### 7.3 Define Golden Queries + +Add these pre-approved SQL templates to train the agent's SQL generation: + +#### Golden Query 1: Attractions + Wait Time Forecasts + +```sql +SELECT + a.name AS attraction_name, + a.branch, + f.forecasted_timestamp, + f.predicted_wait_time +FROM + `disney.public_disneyland_attractions` a +JOIN + `disney.forecasted_waiting_times` f ON a.attraction_id = f.attraction_id +WHERE + a.name = 'Space Mountain' +ORDER BY + f.forecasted_timestamp ASC; +``` + +#### Golden Query 2: Graph Routing + +```sql +SELECT + a1.name AS current_attraction, + a2.name AS recommended_next_attraction, + r.recommendation_rank +FROM + `disney.graph_recommendations` r +JOIN + `disney.public_disneyland_attractions` a1 ON r.attraction_id = a1.attraction_id +JOIN + `disney.public_disneyland_attractions` a2 ON r.recommended_next_attraction_id = a2.attraction_id +WHERE + a1.name = 'Space Mountain' AND r.recommendation_rank = 1; +``` + +--- + +### 7.4 Execute Multi-Silo Prompts + +Test the agent with the complex prompt: *"Which attractions have the highest negative sentiment today, and what is the most common path visitors take after leaving them?"* + +The agent should resolve this to a SQL query similar to: + +```sql +WITH negative_attractions AS ( + SELECT + a.attraction_id, + a.name AS attraction_name, + COUNT(*) AS negative_count + FROM + `disney.reviews_sentiment_analysis` s + JOIN + `disney.public_disneyland_attractions` a ON s.branch = a.branch + WHERE + s.sentiment = 'Negative' + AND s.review_text ILIKE '%' || a.name || '%' + GROUP BY + a.attraction_id, a.name + ORDER BY + negative_count DESC + LIMIT 1 +), +next_steps AS ( + SELECT + b.name AS next_attraction_name, + COUNT(*) AS transition_count + FROM + GRAPH_TABLE( + disney.disney_movement_graph + MATCH (a:Attraction) -[e:Moved]-> (b:Attraction) + RETURN a.attraction_id AS from_id, b.name AS name + ) + WHERE + from_id = (SELECT attraction_id FROM negative_attractions) + GROUP BY + next_attraction_name + ORDER BY + transition_count DESC + LIMIT 1 +) +SELECT + (SELECT attraction_name FROM negative_attractions) AS worst_attraction, + (SELECT next_attraction_name FROM next_steps) AS most_common_next_attraction; +``` + +--- + +## Challenge 8: From insights to action, syncing BigQuery and AlloyDB + +### 8.1 Create Local Analytical Tables in AlloyDB + +Run the following DDL in AlloyDB Studio inside the `disney` database: + +```sql +CREATE TABLE IF NOT EXISTS public.forecasted_waiting_times ( + attraction_id INT, + forecasted_timestamp TIMESTAMP, + predicted_wait_time NUMERIC +); + +CREATE TABLE IF NOT EXISTS public.graph_recommendations ( + attraction_id INT, + recommended_next_attraction_id INT, + recommendation_rank INT +); +``` + +### 8.2 Grant IAM Privileges to AlloyDB + +Find your AlloyDB cluster's service account: + +```bash +gcloud beta alloydb clusters describe --region= --format="value(serviceAccountEmail)" +``` + +Grant this service account the following roles in your project: + +* **BigQuery Data Viewer** (`roles/bigquery.dataViewer`) +* **BigQuery Read Session User** (`roles/bigquery.readSessionUser`) + +### 8.3 Map BigQuery Tables using the AlloyDB Studio Wizard + +1. In the Google Cloud Console, navigate to **AlloyDB** -> **Clusters** -> select your cluster -> **AlloyDB Studio**. +2. Connect to the `disney` database. +3. Click on the **Query BigQuery** button (or look for the **External Data** / **BigQuery** integration option in the explorer pane). +4. Follow the wizard to link your BigQuery dataset `disney` and map the tables. Name the foreign tables `bq_forecasted_waiting_times` and `bq_graph_recommendations` to match the sync queries. + +*Alternative (Manual DDL):* +If you prefer to run the DDLs manually instead of using the wizard: + +```sql +CREATE EXTENSION IF NOT EXISTS bigquery_fdw; +CREATE SERVER bq_disney_server FOREIGN DATA WRAPPER bigquery_fdw; +CREATE USER MAPPING FOR postgres SERVER bq_disney_server; + +CREATE FOREIGN TABLE bq_forecasted_waiting_times ( + attraction_id INT, + forecasted_timestamp TIMESTAMP, + predicted_wait_time NUMERIC +) SERVER bq_disney_server OPTIONS ( + project '', + dataset 'disney', + table 'forecasted_waiting_times' +); + +CREATE FOREIGN TABLE bq_graph_recommendations ( + attraction_id INT, + recommended_next_attraction_id INT, + recommendation_rank INT +) SERVER bq_disney_server OPTIONS ( + project '', + dataset 'disney', + table 'graph_recommendations' +); +``` + +### 8.4 Sync Data from BigQuery to AlloyDB + +Run the following queries in AlloyDB Studio to copy the analytical insights locally: + +```sql +-- Sync wait time forecasts +INSERT INTO public.forecasted_waiting_times (attraction_id, forecasted_timestamp, predicted_wait_time) +SELECT attraction_id, forecasted_timestamp, predicted_wait_time +FROM public.bq_forecasted_waiting_times; + +-- Sync graph recommendations +INSERT INTO public.graph_recommendations (attraction_id, recommended_next_attraction_id, recommendation_rank) +SELECT attraction_id, recommended_next_attraction_id, recommendation_rank +FROM public.bq_graph_recommendations; +``` + +--- + +## Challenge 9: Exposing Database Tools via MCP + +### 9.1 Configure `tools.yaml` + +Create a `tools.yaml` file to configure the **MCP Toolbox** for database access. This file defines five tools mapping to the operational and local analytical tables (not the FDW tables directly): + +```yaml +kind: source +name: disney-db +type: alloydb-postgres +project: "[YOUR_PROJECT_ID]" +region: "europe-west1" +cluster: "[YOUR_CLUSTER]" +instance: "[YOUR_INSTANCE]" +ipType: "public" +database: "disney" +user: "postgres" +password: "buildwithgemini2026" +--- +# Source 2: Gemini Data Analytics API (used for QueryData) +kind: source +name: gda-api-source +type: cloud-gemini-data-analytics +projectId: "[YOUR_PROJECT_ID]" + +--- +# Tool 1: Hybrid Search using ScaNN and Full-Text Search +kind: tool +name: search_attractions_hybrid +type: postgres-sql +source: disney-db +description: "Performs a high-performance hybrid (vector + keyword) search on park attractions based on user interests." +parameters: + - name: vector_query + type: string + description: "Semantic search term (e.g., 'thrilling space roller coaster')" + - name: text_query + type: string + description: "Keyword search term (e.g., 'Space Mountain')" +statement: | + ALTER DATABASE disney SET google_ml_integration.enable_preview_ai_functions = 'on'; + SELECT a.name, a.description, search_results.score + FROM disneyland_attractions a + JOIN ai.hybrid_search( + search_inputs => ARRAY[ + ( '{ + "data_type": "vector", + "table_name": "disneyland_attractions", + "key_column": "attraction_id", + "vec_column": "embedding", + "distance_operator": "public.<=>", + "limit": 5, + "query_vector": "ai.embedding(''gemini-embedding-001'', ''' || $1 || ''')::vector" + }' )::JSONB, + ( '{ + "data_type": "text", + "table_name": "disneyland_attractions", + "key_column": "attraction_id", + "text_column": "description_tsvector", + "limit": 5, + "ranking_function": "<=>", + "query_text_input": "' || $2 || '" + }' )::JSONB + ], + id_type => NULL::BIGINT + ) AS search_results ON a.attraction_id = search_results.id; +--- +# Tool 2: Semantic Filtering using AlloyDB AI operator (google_ml.if) +kind: tool +name: check_ride_suitability +type: postgres-sql +source: disney-db +description: "Evaluates if a specific attraction is safe or suitable based on a guest's profile (e.g., 'pregnant women' or 'toddlers')." +parameters: + - name: attraction_name + type: string + description: "Name of the attraction" + - name: suitability_profile + type: string + description: "Profile of the guest (e.g., 'pregnant women', 'toddlers')" +statement: | + SELECT * FROM check_attraction_suitability($1, $2); +--- +# Tool 3: Transactional Tool to record new reviews +kind: tool +name: add_attraction_review +type: postgres-sql +source: disney-db +description: "Saves a new customer review for an attraction into the operational database." +parameters: + - name: rating + type: integer + description: "Numerical rating out of 5" + - name: review_text + type: string + description: "Customer review text" + - name: branch + type: string + description: "The park branch location" +statement: | + INSERT INTO public.disneyland_reviews (review_id, rating, review_text, branch, year_month) + VALUES ((SELECT COALESCE(MAX(review_id), 0) + 1 FROM public.disneyland_reviews), $1, $2, $3, TO_CHAR(CURRENT_DATE, 'YYYY-MM')) + RETURNING review_id, rating, branch; +--- +# Tool 4: Analytical Tool checking Wait Time Forecasts (Local Table) +kind: tool +name: get_wait_time_forecast +type: postgres-sql +source: disney-db +description: "Queries the local database to get forecasted wait times for a specific attraction." +parameters: + - name: attraction_id + type: integer + description: "Unique ID of the attraction" +statement: | + SELECT predicted_wait_time + FROM public.forecasted_waiting_times + WHERE attraction_id = $1 + ORDER BY forecasted_timestamp ASC LIMIT 1; +--- +# Tool 5: Analytical Tool checking Graph Recommendations (Local Table) +kind: tool +name: get_next_ride_recommendation +type: postgres-sql +source: disney-db +description: "Gets next-ride routing recommendations for a guest leaving a specific attraction to avoid queues." +parameters: + - name: attraction_id + type: integer + description: "Unique ID of the attraction" +statement: | + SELECT recommended_next_attraction_id, recommendation_rank + FROM public.graph_recommendations + WHERE attraction_id = $1; +--- +# Tool 6: QueryData Natural Language Search +kind: tool +name: query_disney_data +type: cloud-gemini-data-analytics-query +source: gda-api-source +description: "Use this tool to ask natural language questions about the Disneyland reviews, attractions, and wait times. It will translate your question into SQL, execute it, and return the answer." +location: "europe-west1" +context: + datasourceReferences: + alloydb: + databaseReference: + projectId: "[YOUR_PROJECT_ID]" + region: "europe-west1" + clusterId: "[YOUR_CLUSTER]" + instanceId: "[YOUR_INSTANCE]" + databaseId: "disney" + agentContextReference: + # Tip: You can find this ID in AlloyDB Studio by clicking on 'Edit Context' + # It should be in this format: projects/[YOUR_PROJECT_ID]/locations/[LOCATION]/contextSets/[CONTEXT_NAME] + contextSetId: "projects/[YOUR_PROJECT_ID]/locations/europe-west4/contextSets/disney-context" +generationOptions: + generateQueryResult: true + generateNaturalLanguageAnswer: true + generateExplanation: true + generateDisambiguationQuestion: true + +--- +kind: toolset +name: disneyland_operational_tools +tools: + - search_attractions_hybrid + - check_ride_suitability + - add_attraction_review + - get_wait_time_forecast + - get_next_ride_recommendation + - query_disney_data +``` + +### 9.2 Start the Server + +Run the toolbox server: + +```bash +./toolbox --config tools.yaml --ui +``` + +--- + +## Challenge 10: Building the guest assistant app + +### 10.1 Scaffold the Guest Assistant with ADK (`agent.py`) + +Scaffold the agent using the Python ADK SDK, loading the tools from the local MCP Toolbox server: + +```python +from google.adk.agents import Agent +from toolbox_core import ToolboxSyncClient + +# 1. Connect to the MCP server running on port 5000 +toolbox = ToolboxSyncClient("http://127.0.0.1:5000") + +# 2. Load all tools (operational + analytical) +disney_tools = toolbox.load_toolset('disneyland_operational_tools') + +# 3. Define the Guest Guide Agent +visitor_guide = Agent( + name='disney_guide_agent', + model="gemini-2.5-flash", + description='A helpful, friendly guide for Disneyland visitors.', + instruction=""" + You are a friendly Disneyland Guest Assistant. Your goal is to help visitors plan their day. + Use the tools at your disposal to: + - Search for attractions (using hybrid search) or filter them semantically. + - Check forecasted wait times for attractions. + - Recommend the next ride to avoid queues based on graph recommendations. + - Add customer reviews if they want to review a ride. + + Always maintain a magical, helpful, and friendly tone. + """, + tools=disney_tools, +) +```