diff --git a/README.md b/README.md index 6a77758..70f30f6 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ The first six examples can be scaffolded fresh via the `swytchcode examples` int | [**`langswytch`**](./langswytch) | Advanced LangGraph cognitive patterns | LangGraph | Multi-Agent • Orchestration • Memory | | [**`swytchcode-refund-agent-openclaw`**](./swytchcode-refund-agent-openclaw) | Automated refund processing agent | OpenClaw | Support • Refunds | | [**`star-repo-openai-agents-python`**](./star-repo-openai-agents-python) | Star a GitHub repository via Swytchcode CLI | OpenAI Agents SDK | GitHub • OpenAI • Python | +| [`ai-agent-api-reliability`](./ai-agent-api-reliability) | AI agent API reliability and controlled GitHub execution | Gemini + Swytchcode Runtime | GitHub • Gemini • Reliability • Python | | [**`github-agent-crewai-python`**](./github-agent-crewai-python) | GitHub operations agent with CrewAI | CrewAI | GitHub • CrewAI • Python | ## License diff --git a/ai-agent-api-reliability/.gitignore b/ai-agent-api-reliability/.gitignore new file mode 100644 index 0000000..099ae25 --- /dev/null +++ b/ai-agent-api-reliability/.gitignore @@ -0,0 +1,29 @@ +# Environment variables / secrets +.env +.env.* +!.env.example + +# Python virtual environments +venv/ +.venv/ +**/venv/ +**/.venv/ + +# Python cache +__pycache__/ +*.py[cod] +*$py.class + +# macOS +.DS_Store + +# IDE / editor files +.vscode/ +.idea/ + +# Swytchcode local/project state +**/.swytchcode/ + +# Logs / temporary files +*.log +*.tmp \ No newline at end of file diff --git a/ai-agent-api-reliability/README.md b/ai-agent-api-reliability/README.md new file mode 100644 index 0000000..6aaa710 --- /dev/null +++ b/ai-agent-api-reliability/README.md @@ -0,0 +1,612 @@ +# AI Agent API Reliability + +An engineering demonstration of what can go wrong when an application or AI agent interacts with a production API. + +The project starts with a deliberately fragile GitHub API integration that reports success even when the request fails. It then exposes the real API response and finishes with a Gemini-powered agent using Swytchcode as the execution layer for a controlled GitHub operation. + +The central question is not simply: + +> Can an AI agent call an API? + +It is: + +> How do we know what the agent requested, what actually executed, whether the action was allowed, what the API returned, and whether the external state actually changed? + +--- + +## Why This Project Exists + +A local success message is not proof that a production API operation succeeded. + +In a real integration, failures can come from authentication, permissions, API changes, incorrect inputs, network issues, or the API returning a result that the application does not correctly interpret. + +This project demonstrates that problem using a deliberately small GitHub starring example: + +1. `naive.py` makes a GitHub API request, hides any exception, and prints success unconditionally. +2. `naive_check.py` performs the same request but inspects GitHub's actual response, exposing the `401 Unauthorized` failure. +3. The controlled agent example gives Gemini access to an explicitly enabled GitHub action. +4. Swytchcode performs the authenticated API operation and returns the result to the agent. +5. `swy audit network` provides an execution history that can be inspected afterward. + +The goal is to make the difference between **reported success** and **verified execution** visible. + +--- + +## Demonstration Scenario + +The example uses a GitHub repository-star operation because it is a simple external side effect that can be verified independently. + +### Naive approach + +```text +Application + | + v +GitHub API + | + +----> Request fails + | + v +Exception is swallowed + | + v +"Done! Starred..." +``` + +The application reports success even though the external operation did not succeed. + +### Controlled agent approach + +```text +User instruction + | + v + Gemini + | + v + Tool request + | + v +Swytchcode runtime + | + v + GitHub API + | + v +Execution result + | + v + Gemini +``` + +Gemini decides which available action to request. The execution layer performs the authenticated API operation and returns its result. + +--- + +## Repository Structure + +```text +ai-agent-api-reliability/ +│ +├── README.md +├── .gitignore +│ +├── naive-api-failure/ +│ ├── naive.py +│ └── naive_check.py +│ +└── swytchcode-agent/ + ├── main.py + └── .env.example +``` + +The working environment also contains local-only files such as: + +```text +swytchcode-agent/ +├── .env +├── venv/ +└── .swytchcode/ +``` + +These are intentionally excluded from version control. + +- `.env` contains local secrets. +- `venv/` contains the local Python virtual environment. +- `.swytchcode/` contains local Swytchcode project/integration state. + +--- + +## Naive API Failure + +The `naive-api-failure` directory demonstrates the initial reliability problem. + +### `naive.py` + +The script constructs a PUT request to GitHub's starred-repository endpoint for: + +```text +octocat/Hello-World +``` + +The implementation intentionally: + +- uses a nonfunctional demonstration token +- makes the HTTP request +- suppresses the exception +- prints a success message regardless of the result + +The important failure is not that the API rejected the request. + +The important failure is that the application hid the rejection and reported success anyway. + +**Run** + +```bash +cd naive-api-failure +python3 naive.py +``` + +The demonstration prints a success message even though the GitHub operation did not actually succeed. + +### `naive_check.py` + +`naive_check.py` performs the same request but actually inspects GitHub's response. + +Instead of discarding the error, it reports the HTTP status and response body. + +In the demonstrated setup, the intentionally invalid credential results in: + +```text +401 Unauthorized +``` + +with a bad-credentials response from GitHub. + +**Run** + +```bash +python3 naive_check.py +``` + +This makes the difference clear: + +```text +naive.py + -> "Done!" + +naive_check.py + -> 401 Unauthorized +``` + +### Important security note + +The token value in these demonstration scripts is intentionally nonfunctional. + +It is not a real GitHub credential. + +Do not replace the demonstration value with a real credential and commit it to the repository. + +If you need to test authenticated GitHub requests locally, use a secure credential-handling approach rather than embedding a real token directly in source code. + +--- + +## Controlled AI-Agent Demonstration + +The `swytchcode-agent` directory contains the AI-agent portion of the project. + +The agent uses: + +- Gemini for reasoning and tool selection +- Swytchcode as the execution layer +- GitHub as the production API + +The important separation is: + +```text +Gemini + | + | decides what action to request + v +Swytchcode + | + | performs the authenticated operation + v +GitHub +``` + +The GitHub credential is handled by the execution layer rather than being passed to Gemini as model input. + +This repository demonstrates a constrained tool-access pattern. It is not intended to claim that any single framework provides a universal solution to AI-agent security or reliability. + +### Tool Access + +The GitHub integration contains multiple starred-repository operations. + +For this demonstration, the project explicitly enables: + +```text +github.starred.update +``` + +The available tool names can be checked with: + +```bash +swy list | grep starred +``` + +The specific action is then enabled with: + +```bash +swy add method github.starred.update +``` + +This distinction is important: + +```text +GitHub integration downloaded + | + v +Available actions discovered + | + v +Specific action explicitly enabled + | + v +Agent can request the enabled action +``` + +The agent is therefore not automatically given unrestricted access to every GitHub operation. + + + +### Prerequisites +- Python 3.10+ +- Node.js and npm +- A Gemini API key +- A GitHub account with permission to manage starred repositories +- Swytchcode CLI +- Network access to GitHub and Gemini + + +### Swytchcode Setup + +Initialize the Swytchcode project from the `swytchcode-agent` directory: + +```bash +cd swytchcode-agent +swy init +``` + +Then retrieve the GitHub integration: + +```bash +swy get github +``` + +Check which starred-repository actions are available: + +```bash +swy list | grep starred +``` + +Enable the action used by this project: + +```bash +swy add method github.starred.update +``` + +GitHub authorization is handled separately: + +```bash +swy auth connect github +``` + +Authentication is performed through GitHub's normal authorization flow. The project does not place the GitHub credential in `main.py` or in the Gemini prompt. + +### Preview the API Operation + +Before making the live request, the operation can be previewed with: + +```bash +swy exec github.starred.update \ + --param owner=octocat \ + --param repo=Hello-World \ + --explain +``` + +The preview shows the intended operation without making the real API request. + +This provides a useful inspection step before executing an external side effect. + +### Execute the GitHub Operation + +The same operation can then be executed for real: + +```bash +swy exec github.starred.update \ + --param owner=octocat \ + --param repo=Hello-World +``` + +The demonstrated successful GitHub response is: + +```text +204 +``` + +The result can then be verified independently by querying the repository's starred state or checking GitHub directly. + +For example: + +```bash +swy exec github.starred.get \ + --param owner=octocat \ + --param repo=Hello-World +``` + +### Gemini Configuration + +The agent uses a Gemini API key stored in a local `.env` file. + +The repository includes: + +```text +swytchcode-agent/.env.example +``` + +Create your local environment file: + +```bash +cd swytchcode-agent +cp .env.example .env +``` + +Then set: + +```text +GEMINI_API_KEY=your_gemini_api_key_here +``` + +Do not commit `.env`. + +The application loads the environment file using `python-dotenv`. + +### Python Environment + +Create the local virtual environment: + +```bash +cd swytchcode-agent +python3 -m venv venv +``` + +Install the dependencies used by `main.py`: + +```bash +./venv/bin/pip install swytchcode-runtime google-genai python-dotenv +``` + +The main dependencies are: + +- `swytchcode-runtime` +- `google-genai` +- `python-dotenv` + +### How `main.py` Works + +`main.py` connects Gemini's function-calling interface to the GitHub tools provided by the Swytchcode runtime. + +At a high level: + +```text +User instruction + | + v +Gemini receives the available tool definitions + | + v +Gemini requests an action + | + v +Swytchcode executes the action + | + v +GitHub returns a result + | + v +Result is returned to Gemini +``` + +One of the important execution boundaries in the code is: + +```python +result = tool.execute(args) +``` + +This is where a selected tool request is handed to the execution layer. + +The model can decide which available operation it wants to request, but the authenticated external operation is performed through the runtime. + +### Running the Agent + +From `swytchcode-agent/`: + +```bash +./venv/bin/python main.py "Star the octocat/Hello-World repo for me" +``` + +The agent can also handle another repository request: + +```bash +./venv/bin/python main.py "Can you go star facebook/react for me?" +``` + +During execution, the application shows the tool request, the execution result, and the model's final response. + +The exact model response can vary depending on the model version, repository state, credentials, and API response. + +### Resetting the Demonstration State + +For repeatable testing, the repository can be returned to a known state by removing the star and checking the result: + +```bash +swy exec github.starred.delete \ + --param owner=octocat \ + --param repo=Hello-World +``` + +Then: + +```bash +swy exec github.starred.get \ + --param owner=octocat \ + --param repo=Hello-World +``` + +A `404` response confirms that the repository is not currently starred. + +This makes it possible to verify the agent's effect before and after execution. + +### Verification and Audit + +The final demonstration step is: + +```bash +swy audit network +``` + +The audit is used to inspect the network/API execution history and distinguish requested actions from actual execution results when the information is available. + +This is particularly important for agent-based systems because the model's response alone should not be treated as proof that an external operation happened. + +The audit trail provides another source of evidence for what was actually executed. + +--- + +## Reliability Lessons + +The project illustrates several practical lessons for production API and AI-agent integrations. + +**1. Do not swallow API failures** + +An exception that disappears is an observability problem. + +An application should not print a success message simply because execution reached the end of a function. + +**2. Check the actual API response** + +A request should be considered successful only after the returned result is checked and interpreted correctly. + +**3. Separate decision-making from execution** + +An AI model can decide which action it wants to take without directly holding the production API credential or directly implementing the API call. + +**4. Restrict available actions** + +Agents should only receive access to the operations required for the task. + +In this demonstration, the project explicitly enables: + +```text +github.starred.update +``` + +rather than exposing every possible GitHub operation. + +**5. Keep credentials out of source code and model input** + +Production credentials should be handled through a secure execution environment rather than embedded directly in application source or model prompts. + +**6. Make execution observable** + +A useful system should make it possible to answer: + +- What did the agent request? +- What was actually executed? +- Was the action allowed? +- What did the API return? +- Did the external state change? +- Can the execution be inspected afterward? + +--- + +## Security Considerations + +Never commit credentials to this repository. + +The repository intentionally ignores: + +```text +.env +venv/ +.venv/ +.swytchcode/ +``` + +The demonstration token in the naive scripts is intentionally nonfunctional. + +For real integrations: + +- keep API keys and access tokens outside source control +- use the minimum permissions required +- avoid passing production credentials to an LLM +- review the operations made available to an agent +- verify externally visible side effects rather than trusting model-generated success messages + +--- + +## Key Takeaway + +The difficult part of an AI/API integration is not simply getting an AI model to call an endpoint. + +The difficult part is making the execution path understandable and verifiable. + +At the beginning of this demonstration: + +```text +API fails + | + v +error is swallowed + | + v +"Done!" +``` + +At the end: + +```text +User request + | + v +Model decision + | + v +Controlled execution + | + v +Real API result + | + v +Verification + | + v +Audit trail +``` + +The difference is the ability to distinguish what the system said happened from what actually happened. + +--- + +## Demo + +This repository accompanies a technical demonstration showing the failure mode, investigation, controlled agent execution, and final audit trail. + + + +--- + +## License + +No license is currently specified for this repository. \ No newline at end of file diff --git a/ai-agent-api-reliability/naive-api-failure/naive.py b/ai-agent-api-reliability/naive-api-failure/naive.py new file mode 100644 index 0000000..aa58ded --- /dev/null +++ b/ai-agent-api-reliability/naive-api-failure/naive.py @@ -0,0 +1,26 @@ +"""The 'works in testing' version — for the video's PROBLEM half. + +This is written the way the tutorials do it: token pasted in the code, +no status check, no log. Run it to film the failure. +""" + +import json +import urllib.request + +# The key, pasted straight into the code — like every tutorial. +# (In the video story: this is the key that "aged out" three weeks after launch.) +GITHUB_TOKEN = "ghp_thisTokenExpiredWeeksAgo0000000000" + +def star_repo(owner, repo): + req = urllib.request.Request( + f"https://api.github.com/user/starred/{owner}/{repo}", + method="PUT", + headers={"Authorization": f"token {GITHUB_TOKEN}"}, + ) + try: + urllib.request.urlopen(req) + except Exception: + pass # <-- the silent part: errors vanish here + +star_repo("octocat", "Hello-World") +print("⭐ Done! Starred octocat/Hello-World") # printed no matter what % \ No newline at end of file diff --git a/ai-agent-api-reliability/naive-api-failure/naive_check.py b/ai-agent-api-reliability/naive-api-failure/naive_check.py new file mode 100644 index 0000000..8a800ba --- /dev/null +++ b/ai-agent-api-reliability/naive-api-failure/naive_check.py @@ -0,0 +1,17 @@ +"""Same call as naive.py, but this time we actually LOOK at GitHub's answer.""" + +import urllib.request, urllib.error + +GITHUB_TOKEN = "ghp_thisTokenExpiredWeeksAgo0000000000" + +req = urllib.request.Request( + "https://api.github.com/user/starred/octocat/Hello-World", + method="PUT", + headers={"Authorization": f"token {GITHUB_TOKEN}"}, +) +try: + resp = urllib.request.urlopen(req) + print("GitHub said:", resp.status) +except urllib.error.HTTPError as e: + print("GitHub said:", e.code, e.reason) + print(e.read().decode()) diff --git a/ai-agent-api-reliability/swytchcode-agent/.env.example b/ai-agent-api-reliability/swytchcode-agent/.env.example new file mode 100644 index 0000000..4e0bb82 --- /dev/null +++ b/ai-agent-api-reliability/swytchcode-agent/.env.example @@ -0,0 +1 @@ +GEMINI_API_KEY="your-gemini-api-key" \ No newline at end of file diff --git a/ai-agent-api-reliability/swytchcode-agent/main.py b/ai-agent-api-reliability/swytchcode-agent/main.py new file mode 100644 index 0000000..a983125 --- /dev/null +++ b/ai-agent-api-reliability/swytchcode-agent/main.py @@ -0,0 +1,90 @@ +""" +A tiny Gemini-powered agent that can star a GitHub repo when you ask it to +in plain English. + +Big picture: + - Gemini decides *what* to do (which tool, which repo). + - Swytchcode actually *does* it (makes the real, authenticated GitHub call). + Gemini never sees your GitHub token; Swytchcode holds the credentials and + runs the call for you. That separation is the whole point. +""" + +import os +import sys + +from google import genai +from google.genai import types +from dotenv import load_dotenv + +from swytchcode_runtime import Swytchcode +from swytchcode_runtime.prompts import TOOL_USE_INSTRUCTIONS + + +load_dotenv() + + +instruction = " ".join(sys.argv[1:]) or "Star the octocat/Hello-World repo for me" + + +gemini = genai.Client(api_key=os.environ["GEMINI_API_KEY"]) + +swy = Swytchcode() + +neutral_tools = swy.tools.get(toolkits=["github"]) +if not neutral_tools: + sys.exit("No github tools found. Run: swy add github.starred.update") + + +tools_by_name = {t.name: t for t in neutral_tools} + +gemini_tool = types.Tool(function_declarations=[ + types.FunctionDeclaration( + name=t.name, + description=t.description, + parameters=t.input_schema, + ) + for t in neutral_tools +]) + +chat = gemini.chats.create( + model="gemini-3.6-flash", + config=types.GenerateContentConfig( + tools=[gemini_tool], + system_instruction=TOOL_USE_INSTRUCTIONS, + ), +) + +print(f"\nYou said: {instruction}\n") + + +message = instruction +while True: + response = chat.send_message(message) + + + calls = response.function_calls or [] + if not calls: + + print("Gemini:", (response.text or "").strip(), "\n") + break + + + replies = [] + for fc in calls: + args = dict(fc.args or {}) + print(f"[Gemini wants to run: {fc.name} input={args}]") + tool = tools_by_name[fc.name] + try: + result = tool.execute(args) + print(f"[executor ran it -> ok: {str(result)[:400]}]") + except Exception as e: + result = f"Error: {e}" + print(f"[executor ran it -> ERROR: {e}]") + + replies.append(types.Part.from_function_response( + name=fc.name, + response={"result": str(result)}, + )) + + + message = replies