Quick Start Guide

⚙️ Prerequisites

Before we begin, make sure you have:

Bun >= 1.2.15 (Install Bun)
Google Gemini API Key (Get free key from Google AI Studio)

🛠️ Installation & Setup

Step 1: Clone the Repository

git clone https://github.com/mpmeetpatel/sniffhunt-scraper.git
cd sniffhunt-scraper

Step 2: Install Dependencies

bun install

This installs all dependencies for the entire workspace including all apps.

Step 3: Configure Environment

cp .env.example .env

Edit the .env file and add your Gemini API key:

# Required
GOOGLE_GEMINI_KEY=your_actual_api_key_here

# Optional (You can provide multiple keys here to avoid rate limiting & load balancing)
GOOGLE_GEMINI_KEY1=your_alternative_key_1
GOOGLE_GEMINI_KEY2=your_alternative_key_2
GOOGLE_GEMINI_KEY3=your_alternative_key_3

# Optional (defaults shown)
PORT=8080
MAX_RETRY_COUNT=2
RETRY_DELAY=1000
PAGE_TIMEOUT=10000
CORS_ORIGIN=*

🚀 Launch Options

Choose your preferred way to use SniffHunt:

Option 1: API Server + Web Interface

Perfect for interactive use and web application integration.

Start the API Server

bun run dev:server

The server will start on http://localhost:8080

Start the Web Interface (Optional)

# In a new terminal
bun run dev:web

Open http://localhost:6001 in your browser for the beautiful web interface.

Test the Setup

# Test the API
curl -X POST http://localhost:8080/scrape-sync \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "mode": "normal"}'

Option 2: MCP Integration for AI Tools

Integrate SniffHunt directly with Claude Desktop, Cursor, or other MCP-compatible AI tools.

Build and Setup MCP Server

bun run setup:mcp

This builds the MCP server and makes it globally available.

Configure Your AI Client

Add this to your MCP client configuration (e.g., Cursor, Windsurf, VSCode, Claude Desktop):

{
  "mcpServers": {
    "sniffhunt-scraper": {
      "command": "npx",
      "args": ["-y", "sniffhunt-scraper-mcp-server"],
      "env": {
        "GOOGLE_GEMINI_KEY": "your-api-key-here"
      }
    }
  }
}

Test MCP Integration

Restart your AI client and try asking:

Scrape https://anu-vue.netlify.app/guide/components/alert.html & grab the ‘Outlined Alert Code snippets’

The AI will automatically use SniffHunt to extract the content!

Option 3: CLI Scraper

Perfect for automation, scripting, and one-off extractions.

Basic Usage

# Scrape any website
bun run cli:scraper https://anu-vue.netlify.app/guide/components/alert.html

# Output saved as:
scraped.raw.md or scraped.md # (based on mode and query automatically generated name)
scraped.html

Advanced Usage

# Use normal mode for static sites
bun run cli:scraper https://anu-vue.netlify.app/guide/components/alert.html --mode normal

# Use beast mode for complex sites
bun run cli:scraper https://anu-vue.netlify.app/guide/components/alert.html --query "Grab the Outlined Alert Code snippets" --mode beast

# Add semantic query for focused extraction & Custom output filename
bun run cli:scraper https://anu-vue.netlify.app/guide/components/alert.html --query "Grab the Outlined Alert Code snippets" --output my-content

✅ Verify Installation

Health Check

# Check if API server is running
curl http://localhost:8080/health

# Should return:
{
  "status": "healthy",
  "service": "SniffHunt Scraper API",
  "version": "1.0.0"
}

Test Extraction

API Test

curl -X POST http://localhost:8080/scrape-sync \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://anu-vue.netlify.app/guide/components/alert.html",
    "mode": "normal",
    "query": "Grab the Outlined Alert Code snippets"
  }'

CLI Test

bun run cli:scraper https://anu-vue.netlify.app/guide/components/alert.html --query "Grab the Outlined Alert Code snippets"

Web UI Test

Open http://localhost:6001
Enter URL: https://anu-vue.netlify.app/guide/components/alert.html
Select mode: “Normal”
Add query: “Grab the Outlined Alert Code snippets”
Click “Extract Content”

🔧 Troubleshooting

Common Issues

Error: “Browser not found”

Solution: Install Playwright browser dependencies (run this command in the root directory of the project):

cd apps/scraper && bunx playwright-core install --with-deps --only-shell chromium

Need Help?

🐞 Bug Reports: GitHub Issues
💬 Questions: GitHub Discussions
📧 Support: contact me on Twitter

Ready to extract some content? Choose your preferred integration method above and start scraping! 🚀