# AI Crawl Observatory β

AI に読ませる面は無料。持ち帰りだけ有料。

人間・検索・対話アシスタントは無料で閲覧・引用できます。
学習・大規模収集系 crawler には、bulk export のみ x402 を適用します。
summary / overview / policy は公開し、raw log や IP は保存しません。

## Access Model

| Surface | Audience | Status | Notes |
|---------|----------|--------|-------|
| HTML page | everyone | free | 人間も AI も読める |
| Markdown | AI agents / crawlers | free | This document |
| Summary JSON | everyone | free | 集計のみ |
| Overview JSON | everyone | free | Observability の概要 |
| Policy JSON | everyone | free | Machine-readable policy |
| Daily Export CSV | charge bots | x402 | Bulk export only |

## Live Stats (30 days)

- Total Requests: 0
- Paid Requests: 0
- Pay Rate: 0%
- Allow Bots: 0
- Charge Bots: 0
- Daily Average: 0
- Generated: 2026-04-07T15:16:38.959Z

## Free Resources

- Summary JSON: https://vnlworks.com/api/labs/ai-crawl-observatory/summary
- Overview JSON: https://vnlworks.com/api/labs/ai-crawl-observatory/overview
- Policy JSON: https://vnlworks.com/api/labs/ai-crawl-observatory/policy
- OpenAPI: https://vnlworks.com/api/labs/ai-crawl-observatory/openapi.json

## Paid Resources (x402)

- Daily Export CSV: https://vnlworks.com/api/labs/ai-crawl-observatory/export
  - Price: $0.10 USDC on Base Mainnet (eip155:8453)
  - Applies to: charge bots only (GPTBot, ClaudeBot, Amazonbot, CCBot, Bytespider, Meta-ExternalAgent, FacebookBot)

## Bot Classification

### Allow (free on all surfaces)
- OAI-SearchBot — ai-search
- ChatGPT-User — ai-assistant
- Claude-SearchBot — ai-search
- Claude-User — ai-assistant
- PerplexityBot — ai-search
- Perplexity-User — ai-assistant
- Applebot — ai-search
- Googlebot — search-engine
- Bingbot — search-engine
- Meta-ExternalFetcher — ai-assistant

### Charge (free on read, x402 on export)
- GPTBot — ai-training
- ClaudeBot — ai-training
- Amazonbot — ai-crawler
- CCBot — ai-crawler
- Bytespider — ai-crawler
- Meta-ExternalAgent — ai-crawler
- FacebookBot — ai-crawler

### Observe (free, aggregate only)
- Human traffic
- Unknown UA
- Heuristic matches

## Privacy

This observability demo stores no raw request logs, IP addresses,
full User-Agent strings, or query strings. Only daily aggregates are retained.

## Discovery

- llms.txt: https://vnlworks.com/llms.txt
- x402 manifest: https://vnlworks.com/.well-known/x402
- Agentic capabilities: https://vnlworks.com/.well-known/agentic-capabilities.json
- OpenAPI: https://vnlworks.com/api/labs/ai-crawl-observatory/openapi.json
- robots.txt: https://vnlworks.com/robots.txt
- Sitemap: https://vnlworks.com/sitemap.xml