The Market Gap Analyzer is a high-performance AI Agent built on the Apify platform. It doesn't just scrape data; it investigates your competition. By combining real-time web crawling with "Negative Sentiment Mining" on Reddit and Trustpilot, it identifies exactly where your competitors are failing and tells you what to build next.
Unlike standard scrapers that return raw JSON, this agent uses a ToolLoopAgent (Vercel AI SDK) to reason through the data. It autonomously:
- Audits your product to establish a baseline.
- Generates aggressive search queries (e.g., "why I stopped using [Competitor]") to find high-churn discussions.
- Outsources heavy lifting to specialized Apify Actors for Reddit and Trustpilot to bypass anti-bot walls.
- Prunes and Ranks results to ensure only the most impactful community feedback is processed by the LLM.
- Negative Sentiment Mining: Specifically targets "Real Talk" on Reddit to find complaints that don't appear in polished marketing materials.
- Hybrid Scraping Logic: Uses a stealth
PlaywrightCrawlerfor landing pages and specialized Sub-Actors for high-security sites (G2/Trustpilot). - Context-Aware Pruning: Automatically filters and sorts hundreds of reviews by upvotes/impact to prevent
context_length_exceedederrors while maintaining high-quality output. - Actionable Roadmaps: Outputs a prioritized P0/P1/P2 feature list and a "Sales Kill-Shot" to help you win more deals.
| Field | Type | Description |
|---|---|---|
myProductUrl |
String | Your SaaS/Product landing page or review profile. |
competitorUrls |
Array | List of competitor domains or review pages. |
modelProvider |
String | openai or google. |
Here are the added sections for your README.md. These provide the technical transparency users need for the output and the "secret sauce" tips to get the highest ROI from the agent.
The agent stores its findings in the Apify Dataset. Each run generates a structured JSON object containing the strategic report. You can export this as a JSON, CSV, or Excel file, or view the formatted Markdown directly in the Apify UI.
| Field | Type | Description |
|---|---|---|
marketIntelligenceReport |
String (Markdown) | The primary output. Includes Internal Audit, Competitor Comparison, Gaps, P0-P2 Roadmap, and Sales Kill-Shot. |
timestamp |
String | ISO 8601 timestamp of when the analysis was performed. |
sources |
Array | List of platforms successfully mined (e.g., ["Reddit", "Trustpilot", "Web"]). |
To get the most out of the Market Gap Analyzer, follow these strategic guidelines:
When providing competitorUrls, prioritize competitors that have a large, vocal user base. The more "Reddit noise" there is, the better the agent can identify high-impact feature gaps.
Instead of just providing a homepage (e.g., https://competitor.com), try providing their Trustpilot profile URL or a specific pricing page. This helps the internal scrapeProduct tool focus immediately on where users feel the most "pain" (the wallet).
- For quick daily monitoring: Use
gpt-4o-mini. It is incredibly fast and handles the "pruned" data perfectly. - For board-level strategy: Switch to
gpt-4o(orclaude-3-5-sonnet). The reasoning capabilities are superior when connecting dots between complex technical complaints on Reddit.
If your product is new or has a generic name (e.g., "Flow"), make sure to include the productName in the input if available. This prevents the Reddit scraper from bringing in irrelevant data about "water flow" or "yoga flow" and keeps the focus on your SaaS.
If you notice the agent missed a specific complaint, check the logs. The agent sorts by Upvotes/Impact. If a complaint is valid but has zero engagement on Reddit, the agent may prune it to save context window space.