# Role Pack: Researcher

CLI-based research: web scraping, academic paper reading, fact-checking, knowledge management, report generation.

## Overview

Three domains with different maturity levels:

- **Built-in agent tools** (`web_search`, `fetch_content`, `code_search`) — already available, no install. First option for most research.
- **Mature CLI tools** — `curl`, `xh`, `jq`, `htmlq`, `pandoc`, `duckdb`, `glow`. Battle-tested, widely used.
- **Emerging tools** — `firecrawl`, `docling`, `scholarcli`. Useful but carry risk: smaller communities, API costs, breaking changes.

## Tools

### Data Extraction

| Tool      | Install             | Purpose                     | Maturity | Link                                                           |
| --------- | ------------------- | --------------------------- | -------- | -------------------------------------------------------------- |
| **curl**  | preinstalled        | Universal HTTP              | ★★★★★    | —                                                              |
| **xh**    | `brew install xh`   | Friendly HTTP, auto-JSON    | ★★★★★    | [github.com/ducaale/xh](https://github.com/ducaale/xh)         |
| **htmlq** | `apt install htmlq` | jq-for-HTML (CSS selectors) | ★★★★☆    | [github.com/mgax/htmlq](https://github.com/mgax/htmlq)         |
| **jq**    | `apt install jq`    | JSON parsing/filtering      | ★★★★★    | —                                                              |
| **gron**  | `brew install gron` | Flatten JSON for grepping   | ★★★★☆    | [github.com/tomnomnom/gron](https://github.com/tomnomnom/gron) |

### Web Scraping

| Tool              | Install                  | Purpose                                | Maturity | Link                                                                                               |
| ----------------- | ------------------------ | -------------------------------------- | -------- | -------------------------------------------------------------------------------------------------- |
| **firecrawl-cli** | `npm i -g firecrawl-cli` | Scrape → clean markdown (JS rendering) | ★★★☆☆    | [docs.firecrawl.dev](https://docs.firecrawl.dev)                                                   |
| **markcrawl**     | `pip install markcrawl`  | Full-site crawl → markdown             | ★★☆☆☆    | [github.com/smartdataanalytics-org/markcrawl](https://github.com/smartdataanalytics-org/markcrawl) |

> **Critical:** Built-in `fetch_content` and `web_search` cover most scraping. Firecrawl adds value for JS-heavy sites and bulk, but costs money and adds a dependency.

### Academic Research

| Tool                     | Install                  | Purpose                                                                  | Maturity | Link                                                                       |
| ------------------------ | ------------------------ | ------------------------------------------------------------------------ | -------- | -------------------------------------------------------------------------- |
| **scholarcli**           | `pip install scholarcli` | Search Semantic Scholar, OpenAlex, DBLP, PubMed, CrossRef                | ★★★☆☆    | [github.com/renatocgs/scholarcli](https://github.com/renatocgs/scholarcli) |
| **docling**              | `pip install docling`    | PDF parsing (tables, formulas, layout)                                   | ★★★★☆    | [ds4sd.github.io/docling](https://ds4sd.github.io/docling)                 |
| **Semantic Scholar API** | `curl`/`xh`              | Free, no auth: `api.semanticscholar.org/graph/v1/paper/search?query=...` | ★★★★★    | [api.semanticscholar.org](https://api.semanticscholar.org)                 |
| **arXiv**                | `curl`                   | Direct PDF: `curl https://arxiv.org/pdf/<id>`                            | ★★★★★    | [arxiv.org](https://arxiv.org)                                             |

> **Critical:** `agent-papers-cli`, `zotero-cli-cc`, and `litellm` from earlier drafts are niche/uncertain. Direct API calls to Semantic Scholar are free, stable, and need no wrapper.

### Knowledge Management

| Tool       | Install               | Purpose                       | Maturity | Link                                                                   |
| ---------- | --------------------- | ----------------------------- | -------- | ---------------------------------------------------------------------- |
| **rg**     | `apt install ripgrep` | Fast text search across files | ★★★★★    | [github.com/BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep) |
| **fd**     | `apt install fd-find` | Modern file finder            | ★★★★★    | [github.com/sharkdp/fd](https://github.com/sharkdp/fd)                 |
| **duckdb** | `apt install duckdb`  | SQL on CSV/JSON/Parquet       | ★★★★★    | [duckdb.org](https://duckdb.org)                                       |
| **pandoc** | `apt install pandoc`  | Document conversion           | ★★★★★    | [pandoc.org](https://pandoc.org)                                       |
| **glow**   | `brew install glow`   | Markdown preview in terminal  | ★★★★☆    | [github.com/charmbracelet/glow](https://github.com/charmbracelet/glow) |
| **kb**     | `pip install kb`      | CLI RAG, hybrid search        | ★★☆☆☆    | [github.com/ariel-frischer/kb](https://github.com/ariel-frischer/kb)   |

> **Critical:** For most research, `rg` + directory structure (`~/research/<project>/sources/`) is more reliable than a CLI RAG tool. `kb` is interesting but early-stage.

## Workflows

### Web Research

```bash
# Built-in (preferred) — use web_search / fetch_content tools directly

# CLI fallback for bulk
xh GET "https://api.semanticscholar.org/graph/v1/paper/search?query=topic&limit=10" | jq '.data[] | {title, url, year, citationCount}'

# Scrape a page
xh GET https://example.com | htmlq '.article-body' | htmlq -t > /tmp/page.md
```

### Academic Search

```bash
# Semantic Scholar (free, no auth)
xh GET "https://api.semanticscholar.org/graph/v1/paper/search?query=quantum+error+correction&limit=20&fields=title,year,citationCount,url,abstract" | jq '.data[] | {title, year, citations: .citationCount}'

# arXiv PDF download
curl -sL "https://arxiv.org/pdf/2301.12345" -o /tmp/paper.pdf

# Extract PDF content
docling /tmp/paper.pdf -o /tmp/paper.md --format markdown
```

### Fact-Checking

```bash
# Academic corroboration
xh GET "https://api.semanticscholar.org/graph/v1/paper/search?query=<claim>&limit=10" --json | jq '.data | length'
# 0 results → weak evidence

# Cross-check angles with built-in web_search
```

### Reports

```bash
# Agent writes markdown → convert
pandoc report.md -f markdown -t pdf -o report.pdf
pandoc report.md -f markdown -t docx -o report.docx
```

### Knowledge Organization

```bash
mkdir -p ~/research/<project>/{sources,notes,reports}
rg "keyword" ~/research/<project>/ --json
duckdb -c "SELECT * FROM read_csv_auto('/tmp/data.csv')"
```

## SKILL.md

```yaml
---
name: researcher
description: >
  Researcher: web research, academic literature, fact-checking, reports.
  Always cite sources, triangulate findings, distinguish fact from opinion.
  Activate for research, investigate, analyze, review, literature, fact check,
  deep dive, competitive analysis, background.
version: "2.0.0"
---

# Researcher

Find, verify, and synthesize information into actionable insights.

## Behavior

- Every factual claim gets a source link
- Minimum 2 independent sources per claim
- When sources contradict, present both views with evidence
- Flag speculation, opinion, sponsored content
- State explicitly when evidence is weak or unavailable

## Workflows

- **Web research** — Built-in `web_search` / `fetch_content` first. Fall back to `xh` + `htmlq` for bulk
- **Academic** — Query Semantic Scholar API. Download PDFs from arXiv. Extract with `docling`
- **Fact-check** — Search 2+ angles. Check academic corroboration. Present Support/Refute/Inconclusive
- **Report** — Agent synthesizes markdown. `pandoc` converts to PDF/DOCX
- **Knowledge** — Store `~/research/<project>/{sources,notes,reports}`. Search with `rg`

## Safety

- Never fabricate citations or sources
- Never present opinion as fact
- Never cherry-pick to fit a predetermined conclusion
- Disclose conflicts of interest in sources
```

## Market Context

What's established vs. vapor:

| Project                         | Status      | Reality Check               | Link                                                                             |
| ------------------------------- | ----------- | --------------------------- | -------------------------------------------------------------------------------- |
| **OpenAI Deep Research**        | Shipping    | GPT-4o, cited reports       | [openai.com](https://openai.com/index/introducing-deep-research/)                |
| **Perplexity Pro**              | Shipping    | Multi-step, cited           | [perplexity.ai](https://www.perplexity.ai)                                       |
| **Google Gemini Deep Research** | Shipping    | 25K word reports            | [gemini.google](https://gemini.google)                                           |
| **Feynman**                     | Open source | `feynman "q"` → cited brief | [feynman.is](https://feynman.is)                                                 |
| **hyperresearch**               | Open source | Leads DeepResearch-Bench    | [github.com/dwstevens/hyperresearch](https://github.com/dwstevens/hyperresearch) |

Most agent marketplace "research skills" are 50-line SKILL.md files teaching the agent to call APIs it could already call. The differentiator is reasoning quality, not the skill file.

## Composition

- → **Executive Assistant** for sending research briefs
- → **Developer** for custom data analysis or scraping scripts

## Setup

```bash
# Core
apt install jq htmlq pandoc duckdb ripgrep fd-find
brew install glow xh gron
pip install docling

# Optional: API scraping
npm i -g firecrawl-cli

# Research directory
mkdir -p ~/research/{sources,notes,reports,shared}
```
