# OpenMagpie: full documentation for LLMs

Inlined copy of the OpenMagpie docs so an assistant can help you write a
feed.yaml + watch.yaml and the matching `magpie` CLI commands in one shot.
Generated from the repo at build time (do not edit by hand); the shorter
link index is at /llms.txt. Each source file below is wrapped in a `<doc
path="...">` tag.

<doc path="README.md">

<p align="center">
  <img src="assets/magpie-logo-on-transparent.png" alt="OpenMagpie" width="220">
</p>

<p align="center">
  Open-source social listening
</p>

<p align="center">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"></a>
</p>

<p align="center">
  <b><a href="https://www.openmagpie.ai/">openmagpie.ai</a></b> |
  <a href="#quickstart">Quickstart</a> |
  <a href="apps/cli/README.md">CLI reference</a> |
  <a href="CHANGELOG.md">Changelog</a>
</p>

---

<p align="center">
  <img src="assets/cli-tour.gif" alt="magpie CLI tour: the feed's sources, the watch's semantic filter, then the matches with scores and links" width="900">
</p>

> [!TIP]
> **CLI not for you?** A UI / hosted version is on the way. Star the repo for updates, or join the waitlist at [openmagpie.ai](https://www.openmagpie.ai/).

## What it does

You scan X/Twitter, YouTube, Reddit, Hacker News, and a few RSS feeds looking for someone talking about your product, hitting a problem it solves, or asking a question you can answer well. Getting there while the conversation is happening is how you build a brand and a community around what you know: a mention answered the day it's posted beats one found in next month's report. OpenMagpie watches the threads for you so you spend your time on engagement instead of searching.

You curate sources into a feed, write a natural-language description of what's relevant (for example, "someone frustrated with manual social monitoring and asking for alternatives"), and a local LLM run via any OpenAI-compatible runner (e.g. Ollama, vLLM, LM Studio) scores each new post against it. Matches go to a webhook or your logs (more integrations coming); everything else is dropped. You read the hits instead of the firehose.

## Where it listens

OpenMagpie listens wherever communities are having those conversations.

- **Public discussion (today):** X/Twitter, YouTube, Reddit, Hacker News, and any RSS or Atom feed (news, blogs, Substack publications, and forums that publish feeds).
- **Communities you're in (roadmap):** Slack workspaces and LinkedIn you already belong to, so you catch relevant threads in the groups where you participate, no admin or app install required.
- **Public discussion (soon to be added):** Facebook, TikTok, and Instagram.

## Quickstart

One command for your first real match (needs Docker and uv; it clones the repo and runs the quickstart for you):

```bash
curl -fsSL https://openmagpie.ai | sh
```

Once setup finishes, keep processing new posts in the background:

```bash
make up-jobs        # run the schedulers in the background
tail -f .jobs/*.log # watch them work
```

<img src="assets/quickstart.gif" alt="The quickstart from curl to first matches: prerequisites check, LLM setup prompts, personalizing the listener (which subreddits, what to flag, how strict), then the seeded backlog scoring and the ready summary" width="900">

Prefer to not generate seed data? `SKIP_DATA_SEED=1` brings up the stack without sample data:

```bash
curl -fsSL https://openmagpie.ai | SKIP_DATA_SEED=1 sh
```

Prefer to clone first?

```bash
git clone https://github.com/obris-dev/openmagpie.git
cd openmagpie
./scripts/quickstart/run.sh
```

Either way it walks you through your first listener (which subreddits to watch, what to flag in plain language, how strict), seeds it, and once an LLM is reachable runs the pipeline once so the first matches print straight to the logs, tagged `[quickstart]`. Matches show up in the terminal and the CLI activity log, not the web UI yet. Your feed and watch are saved as editable YAML in `config/quickstart/` (see [config/README.md](config/README.md) for editing and reusing them). Want more posts to start with? `DAYS=7 ./scripts/quickstart/seed.sh` backfills a week instead of a day. See [examples/README.md](examples/README.md) for ready-made starters to apply by hand. Or have an AI assistant interview you and build the config: copy the prompt in [Set it up with an AI assistant](examples/README.md#set-it-up-with-an-ai-assistant).

### Prereq: an OpenAI-compatible LLM endpoint

OpenMagpie is BYO LLM; the dev stack doesn't bundle one. Whatever you already run almost certainly works, because **Ollama, vLLM, llama.cpp, and LM Studio all expose an OpenAI-compatible `/v1` API** (so do hosted providers like OpenAI, Together, or Groq). OpenMagpie talks to that `/v1` endpoint with the standard OpenAI client, so you just point `ENGINE_BASE_URL` at it. The quickstart validates your endpoint and points you at the model of your choice.

- **Local.** Any OpenAI-compatible server on your machine works; point `ENGINE_BASE_URL` at its `/v1`: Ollama (`http://host.docker.internal:11434/v1`), vLLM (`:8000/v1`), LM Studio (`:1234/v1`), or llama.cpp (`:8080/v1`). The shipped default is Ollama's `:11434`. New here and want the quickest start? Install [Ollama](https://ollama.com/download), then `ollama pull qwen2.5:7b && ollama serve`.
- **Remote (LAN box, GPU server, cloud).** Set `ENGINE_BASE_URL=http://your-host:11434/v1` in `apps/core/.env`.
- **Hosted API.** Set `ENGINE_BASE_URL=https://api.openai.com/v1` and `ENGINE_API_KEY=...` (local servers leave the key blank).

Set `ENGINE_MODEL` to the model you want to judge with. A 7B model judges in roughly 1 to 3 seconds on Apple Silicon or a recent NVIDIA GPU; CPU-only works but is slower.

### Use it

The quickstart already built and seeded everything and put the `magpie` CLI on your `PATH`. Drive it from there:

```bash
magpie auth login                            # browser device flow
magpie feed create                           # opens $EDITOR on a feed template (sources + retention)
magpie watch create                          # opens $EDITOR on a watch template (feeds + action chain)
magpie activity summary --action <action_id> # per-state run breakdown for any action (filter, webhook, log)
```

On a headless box (a server you SSH into, no browser), skip the device flow and use a personal access token. Mint one on the server, then sign in with it on the box, the token is pasted (stdin or a hidden prompt, never the command line) and stored in `~/.magpie` at `0600`, persisting across sessions:

```bash
# on the server (the issue_cli_token management command, via the local stack):
make local-manage CMD="issue_cli_token --email you@example.com --name my-box"
# then, on the box:
magpie auth login --token   # paste the printed token at the prompt
```

For CI or an ephemeral box, set `MAGPIE_TOKEN=mgp_...` in the environment instead: it's
read on every request, takes precedence over a stored login, and is never persisted (the
`GH_TOKEN` pattern), so no login step. Manage tokens with `magpie auth token list` /
`create` / `revoke` (minting needs a browser login; a token can't mint another).

A watch's `actions:` chain typically starts with a `semantic_filter` (your natural-language criteria + threshold) followed by a `webhook` or `log` delivery. Pick a backfill window when you create the feed and the first `make local-tick` scores real posts against your criteria immediately, with no wait for the scheduler.

Full command list: the [magpie CLI reference](apps/cli/README.md). The dev loop runs through `make`: see [make/README.md](make/README.md) or `make help`.

### Running it continuously

`make local-tick` runs one pass by hand. For ongoing operation, start the background scheduler. The four pipeline stages each tick on their own cadence (poll feeds, trigger watches, drain runs, flush digests):

```bash
make up-jobs                  # start the tickers (a pid + log per stage under .jobs/)
tail -f .jobs/drain.log       # watch a stage
make down-jobs                # stop them
```

Each stage is single-flight: a pass that outruns its interval self-skips the next tick, so loops never stack. Production scheduling is then just a plain cron entry per command on the same cadences, with no flock or singleton infrastructure. Override any cadence inline, e.g. `make up-jobs DRAIN_INTERVAL=30`.

Run `make help` for the full target list (`make up` / `down`, `make logs`, `make local-test`, `make local-check`, and so on).

### Upgrading

The quickstart pins your checkout to the latest release. When a newer one ships, upgrade in place from your install directory:

```bash
make upgrade                  # or: ./scripts/upgrade.sh
```

It advances the checkout to the latest release tag, rebuilds the stack, applies migrations, and refreshes the `magpie` CLI. **Your data is preserved** (the database volume persists, migrations are additive, and it never re-seeds). For the bleeding edge instead of a release, `OPENMAGPIE_BRANCH=main make upgrade`.

## How it works

A `Feed` is a reusable, curated stream (a set of sources plus an item log). A `Watch` subscribes to one or more feeds and runs an ordered **action chain** over each new item: a `semantic_filter` gates the chain (a score below threshold stops it), and downstream `webhook` / `log` actions deliver what passes. One feed can back many watches, so you pay for source polling once.

```mermaid
graph TD
    subgraph Sources
        TWITTER[X / Twitter]
        YOUTUBE[YouTube]
        REDDIT[Reddit]
        RSS[RSS / Atom feeds]
        HN[Hacker News]
        SLACK[Slack]
        LINKEDIN[LinkedIn]
        GITHUB[GitHub]
        FACEBOOK[Facebook]
        TIKTOK[TikTok]
        INSTAGRAM[Instagram]
    end

    subgraph OpenMagpie
        FEED[Feed<br/>curated streams + item log]
        WATCH[Watch<br/>subscribes to feeds]
        FILTER[semantic_filter<br/>action]
        ENGINE[Relevance engine<br/>BYO LLM]
        DELIVER[webhook / log<br/>delivery action]
    end

    subgraph Out
        WEBHOOK[Webhook]
        LOG[Log]
        FUTURE["email / Slack (planned)"]
    end

    TWITTER --> FEED
    YOUTUBE --> FEED
    REDDIT --> FEED
    RSS --> FEED
    HN --> FEED
    SLACK -. planned .-> FEED
    LINKEDIN -. planned .-> FEED
    GITHUB -. planned .-> FEED
    FACEBOOK -. "soon to be added" .-> FEED
    TIKTOK -. "soon to be added" .-> FEED
    INSTAGRAM -. "soon to be added" .-> FEED

    FEED -- "new items" --> WATCH
    WATCH -- "action chain" --> FILTER
    FILTER --> ENGINE
    ENGINE -. "your LLM" .-> LLM["any OpenAI-compatible /v1 API<br/>Ollama | vLLM | llama.cpp | LM Studio | OpenAI"]
    FILTER -- "passes -> next action" --> DELIVER

    DELIVER --> WEBHOOK
    DELIVER --> LOG
    DELIVER -. planned .-> FUTURE
```

Delivery is **instant** (per item) or **digest** (a window of items batched into one emission). A `webhook` action POSTs (or PUTs / PATCHes) one self-describing body; instant and digest use the same shape (instant is a one-item batch):

```json
{
  "watch": {"id": "01K...", "name": "ai-webhook"},
  "action_id": "01K...",
  "delivery": "digest",
  "window": {"since": "...", "until": "..."},
  "items": [
    {
      "key": "reddit_subreddit:abc123",
      "source": {"label": "r/ClaudeAI", "kind": "reddit_subreddit"},
      "item": {"title": "...", "url": "..."}
    }
  ]
}
```

`item` is the feed item narrowed to the action's `include_fields`. Each item's `key` is `source:external_id`; delivery is at-least-once, so receivers dedup on it. Every call is recorded as a `WatchActionDelivery` you can inspect:

```bash
make local-cli ARGS="delivery list --action <webhook_action_id>"    # the list: state / HTTP / host / items / attempt
make local-cli ARGS="delivery get <delivery_id>"                    # one call in full, incl. the exact body sent
```

See [AGENTS.md](AGENTS.md) for the design conventions (char pointers, typed-blob pattern, the trigger/drain/flush execution model).

## Why self-host it

Social listening is a crowded market (Brand24, Mention, Octolens, Syften, and tools like OutX that pair monitoring with AI-drafted replies). They are all closed SaaS behind a paid plan, a trial, or a sales demo, and the few genuinely free options are basic mention notifiers, not full listening. OpenMagpie is the open, self-hostable exception: run it on your own box with your own model for the cost of the hardware.

- **Open source.** Apache 2.0, the whole stack. Read it, fork it, and extend the connectors and engines yourself.
- **Bring your own LLM.** Relevance is judged by an LLM you run (via any OpenAI-compatible backend like Ollama, vLLM, llama.cpp, LM Studio etc), so your criteria and your matches stay on your infrastructure when you self-host the model.
- **Natural-language matching.** You describe what's relevant in natural language and the model scores each new post on meaning.
- **Auditable.** Every poll, judgement, and delivery is a row you can inspect (`magpie activity summary` / `delivery list`), as a table or `--jsonl` to pipe into `jq` / an LLM, or written to a file with `-o`.

## What's shipped today

| Layer | Shipped |
|---|---|
| Connectors | X/Twitter (`twitter_search`), YouTube (`youtube_search`), Reddit (`reddit_subreddit`), Hacker News (`hn_feed`, `hn_comment`), RSS/Atom (`rss`) |
| Engines | Any OpenAI-compatible `/v1` API: Ollama, vLLM, llama.cpp, LM Studio, OpenAI, ... |
| Action kinds | `semantic_filter` (LLM-judged), `webhook`, `log` |
| Delivery modes | instant, digest |
| Webhook methods | `POST`, `PUT`, `PATCH` |
| Delivery audit | per-attempt `WatchActionDelivery` |

## What we've done

X/Twitter listening is the first connector added beyond the original Reddit / HN / RSS set. What shipped in this branch:

- **`twitter_search` source kind** — a twikit-based connector that runs X search queries (mode Top/Latest, count) and maps results to a schema-parity `NewTweetPayload`, registered alongside the existing kinds with the same feed/watch/webhook pipeline.
- **Live-cookie auth** — the connector authenticates with an existing X session cookie JSON (`TWITTER_COOKIES_JSON` → `auth_token`/`ct0`, a cookies file, or a login fallback), so no API key, proxy, or vendor API is required — the same live cookies the listening kit already used keep working.
- **Reliability fixes from live polling** — a per-call twikit client (multi-source polls no longer crash with "Event loop is closed") and X's transient empty-body 404 mapped retryable instead of "tweet deleted", with a regression test. 587 tests green; all CI gates pass.
- **Verified live end-to-end** — a real X poll through a feed → watch → webhook chain delivered 44/44 items with HTTP 200, payload matched field-for-field against the Twenty `socialEvent` intake contract (`item.handle → actorHandle`, `author → actorName`, `content → eventText`, `occurred_at → occurredAt`, `url → sourceUrl`, `key → dedupeKey`).

YouTube listening followed via yt-dlp:

- **`youtube_search` source kind** — a yt-dlp-based connector that runs YouTube search queries and maps results to a schema-parity `NewVideoPayload`, registered alongside the existing kinds with the same feed/watch/webhook pipeline.
- **No authentication required** — public YouTube search works without credentials; optional cookie file for age-restricted content.
- **Error taxonomy** — 5 error codes (`video_unavailable`, `rate_limited`, `js_runtime_missing`, `network_error`, `yt_dlp_error`) with retry semantics.
- **Watermark-based deduplication** — videos newer than the source's `last_event_at` are surfaced.
- **Metrics extraction** — views, likes, comments mapped from YouTube metadata.
- **Thumbnail media** — full thumbnail URLs attached to payloads for rich display.

Next up on the roadmap: **Facebook, TikTok, and Instagram connectors** (soon to be added), then Slack, LinkedIn, GitHub, Bluesky, and Mastodon.

## Roadmap

- **More connectors**: Facebook, TikTok, and Instagram (soon to be added), then Slack, LinkedIn, GitHub, Bluesky, and Mastodon.
- **More engines**: Anthropic, OpenAI, and a keyword engine behind the same `Engine` Protocol.
- **Learns from feedback**: thumbs up/down on past matches become few-shot examples for the next pass.
- **Run-history in the payload**: the upstream filter score and chain provenance as an opt-in webhook field.
- **Branching and parallel chains**: the data model already carries `WatchPath` and dense action ranks; multi-path and DAG branching are post-v1.
- **Retention**: pruning for `WatchActionRun` and `WatchActionDelivery` history.

## Hosted version

Self-hosting is free and stays free. A managed hosted version (no infrastructure to run) is in the works as the paid tier.

Join the waitlist at [openmagpie.ai](https://www.openmagpie.ai/).

## Project structure

uv workspace; one root `uv.lock` for everything Python.

```
apps/
  core/                       Django backend (deployable)
    common/                   BaseModel (ULID PK + timestamps), ULIDField, locks, db ceilings, /healthz
    accounts/                 User / Account / UserProfile + services + AccountScopedAPIView mixin
    auth_api/                 signup / login / logout / me + tokens/* + device-flow handshake (DRF)
    sources/                  Connectors (Reddit, Hacker News, RSS/Atom) + SourcePayload classes + registry
    feeds/                    Feed + Source + FeedItem models + poll orchestrator + item log
    engine/                   Engine Protocol + OpenAICompatEngine + registry (+ probe)
    watches/                  Watch + WatchFeed + WatchPath + WatchAction + WatchActionRun + WatchActionDelivery
    conf/                     settings (base/local), urls, wsgi
  cli/                        magpie CLI (Typer + httpx + Pydantic); distributed as a standalone wheel
packages/
  openmagpie-schema/          Pure Pydantic models shared by core + cli (configs, wire types, feed shapes)
web/                          pnpm workspace: apps/{app,marketing,email-render} (Next.js) + packages/{ui,api-utils,auth,tailwind-config}
make/                         Per-concern Makefile targets
scripts/                      quickstart installer (quickstart/{bootstrap,preflight,run,seed,tick}.sh) + dev tooling (Docker preflight, git hooks, whitespace/branch/length checks, make-help)
```

## Documentation

- [CONTRIBUTING.md](CONTRIBUTING.md): contribution flow, branch naming, running the checks.
- [CHANGELOG.md](CHANGELOG.md): notable changes per release.
- [magpie CLI reference](apps/cli/README.md): install + the full command list.
- [make/README.md](make/README.md): the important dev `make` commands (`make help` for the full list).
- [AGENTS.md](AGENTS.md): cross-cutting design conventions, plus per-area notes: [apps/core](apps/core/AGENTS.md), [apps/cli](apps/cli/AGENTS.md), [web](web/AGENTS.md).

## Telemetry

OpenMagpie ships **anonymous, opt-in** usage telemetry, **off by default**. It helps prioritize what to build (a UI? which sources next? is setup too hard?) without ever sending your content. Enable it during `quickstart`, or with `make local-manage CMD="telemetry enable"`; turn it off any time with `make local-manage CMD="telemetry disable"` or `DO_NOT_TRACK=1`. Exactly what is and isn't collected: [apps/core/TELEMETRY.md](apps/core/TELEMETRY.md).

## License

OpenMagpie is open source under the [Apache License 2.0](LICENSE), with optional enterprise directories (`**/ee/`) reserved for future commercial features.

</doc>

<doc path="config/README.md">

# Your OpenMagpie config

Your feed and watch configs live here. `templates/quickstart/` is the template
the quickstart starts from; when you run the quickstart it writes your seeded
listener to `quickstart/`. Keep configs you write from here on in this directory
too.

## Layout

- `templates/quickstart/` (checked in): the template the quickstart seeds from.
  A clean starting point to copy.
- `quickstart/` (created by the quickstart, not checked in): your seeded feed and
  watch as editable YAML, with your subreddits, your filter, and the real feed id.

## The pieces

**Feed** (`feed.yaml`): the set of sources OpenMagpie polls for new posts. A feed
only gathers posts; it does not decide what matters. A feed can mix source kinds;
each source is one `spec` entry under `sources`. Supported kinds:

- `reddit_subreddit`: a subreddit's new posts. `{kind: reddit_subreddit, subreddit: selfhosted}`
- `hn_feed`: Hacker News posts; `feed` picks the stream (`new`, `show` for Show HN,
  or `ask` for Ask HN). `{kind: hn_feed, feed: new}`
- `hn_comment`: Hacker News comments matching a keyword `query` (required, the comment
  stream is huge, so it is always keyword-filtered server-side). `{kind: hn_comment, query: "your product"}`
- `rss`: any RSS/Atom feed by URL. `{kind: rss, url: "https://example.com/feed.xml", name: "Example blog"}`

Both HN kinds accept a `query` keyword filter (optional for `hn_feed`, required for
`hn_comment`), matched server-side before anything reaches your watch:

- space-separated words must **all** match (AND): `query: "open source"`.
- `match: any` matches **any** of the words (OR): `{kind: hn_comment, query: "nomad k8s", match: any}`.
- prefix a word with `-` to **exclude** it: `query: "database -mysql"`.

See `templates/quickstart/feed.yaml` for these in context.

**Watch** (`watch.yaml`): subscribes to one or more feeds and runs an ordered
chain of actions over each new post. Its `feed_ids` is what links it to a feed.

**Actions**: the ordered steps a watch runs on each new post from its feeds.

- `semantic_filter`: your LLM rates how well each post matches your plain-language
  `instructions`, from 0 to 1, and the post passes when that relevance score is at
  least `threshold` (0 keeps everything, 1 only exact matches; higher is stricter).
  This is what makes a watch a listener instead of a firehose. When a post links
  off-site (e.g. a Hacker News link post), the linked article is fetched and judged
  alongside the title; set `fetch_external_content: false` on the action to skip it.
- `log`: prints the posts that clear the filter (the prefixed lines you see when
  the pipeline runs).
- `webhook` (optional): POSTs each match to a URL. See [examples/README.md](../examples/README.md).

## Editing and reusing

To change your listener, edit `quickstart/feed.yaml` or `quickstart/watch.yaml`
and re-apply to the existing feed/watch (the ids print in the quickstart summary,
or from `magpie feed list` / `magpie watch list`):

    magpie feed edit <feed-id> -f config/quickstart/feed.yaml
    magpie watch edit <watch-id> -f config/quickstart/watch.yaml

To build a separate listener, copy a config, change its `name`, and create it.
Create the feed first, then put its printed id into the new watch's `feed_ids`.
(If you copy the template `watch.yaml`, that means replacing its
`REPLACE_WITH_FEED_ID` placeholder, which `watch create` rejects as-is.)

    magpie feed create -f config/my-feed.yaml
    magpie watch create -f config/my-watch.yaml

A freshly created feed scores only posts that arrive after it. To backfill (score
recent posts on the first tick), set a past `last_event_at` on each source in the
feed YAML before creating it, e.g. `last_event_at: 2026-01-01T00:00:00Z`.

Inspect a watch any time:

    magpie watch list
    magpie watch action get <action-id>
    magpie activity list --action <action-id>

</doc>

<doc path="examples/README.md">

# Examples: starter feeds and watches

A starter is a ready-to-apply `feed.yaml` + `watch.yaml` pair under
`examples/starters/<name>/`. Each one wires a curated feed (a couple of
subreddits) to a watch that runs a semantic filter, then logs the matches.
Apply one by hand, or copy it as a starting point for your own.

Available starters:

- `selfhosted-opensource`: r/selfhosted + r/opensource, listening for people
  asking for an open-source or self-hostable alternative to a paid tool.
- `devtools`: r/devops + r/programming, listening for people hitting a problem
  a developer tool could solve, or asking for a tool recommendation.
- `hackernews`: HN new stories (`hn_feed`), listening for open-source project
  launches. Low-volume (~1k stories/day), safe to apply as-is.
- `hackernews-comments`: a keyword-filtered slice of HN's comment stream
  (`hn_comment`), for monitoring mentions of a project or product. Kept separate
  from `hackernews` on purpose: comments are high-volume, so **read the local-
  processing warning at the top of its `feed.yaml`** and keep the keyword tight
  before applying: a broad keyword can outrun a local engine.

Both HN starters take a `query` keyword that runs **server-side as a pre-filter**:
it narrows which items the connector pulls into the feed at all, before any watch
takes action on them, so it bounds what gets ingested (and the work that follows)
rather than filtering after the fact. It's optional for stories but required for
comments, to keep a watch on the comment stream focused. It supports operators: AND by default, `match: any` for OR, `-word` to exclude, and
`"phrase"`. The exact syntax is in each starter's `feed.yaml`.

## Applying a starter by hand

The YAML files are `magpie feed/watch create -f` inputs. They are templates, not
copy-paste-ready: the two edits below are required first (the `watch create`
rejects the `REPLACE_WITH_FEED_ID` placeholder as written).

```
magpie feed create -f examples/starters/selfhosted-opensource/feed.yaml
magpie watch create -f examples/starters/selfhosted-opensource/watch.yaml
```

Two edits to make first:

- Set a past `last_event_at` on each source in `feed.yaml`. Without it the feed
  only scores posts going forward, so to test your watch you would have to wait
  for new posts to arrive at the source.
- Put the real feed id into `feed_ids` in `watch.yaml`, replacing the
  `REPLACE_WITH_FEED_ID` placeholder. The feed id prints when you create the
  feed.

## Set it up with an AI assistant

Don't want to hand-write YAML? Let an assistant build your config by interviewing
you. This works best with a coding agent that has shell access (Claude Code,
Codex, Gemini CLI): it reads the docs, asks what you want, writes the files, runs
the commands, and fixes any validation error itself. A chat LLM (ChatGPT,
Claude.ai) works too; you just run the commands it gives you. Paste this in:

```
I want to set up OpenMagpie, an open-source social-listening tool, to watch for
something. Read its docs at https://openmagpie.ai/llms-full.txt (or the README,
config/README.md, and examples/ in this repo, if you have it cloned) for the
config schema, the source kinds (Reddit, Hacker News, RSS), and worked examples.

Interview me first: ask what I want to catch, which sources fit, how strict to be,
and where matches should go (the logs or a webhook). Then, from my answers, write
a feed.yaml and a watch.yaml, using a semantic_filter with a clear plain-language
instruction and keeping any source query tight.

If you can run shell commands, create the files and run the `magpie feed create -f`
and `magpie watch create -f` commands yourself, then fix any validation error
`create` reports and retry. Otherwise, give me the files and the commands to run.
```

If your assistant can't browse the web, paste the contents of
[openmagpie.ai/llms-full.txt](https://openmagpie.ai/llms-full.txt) into the chat
first (or run `magpie feed template` and `magpie watch template` and paste those).
`create` validates everything, so the assistant just iterates against any error.

Prefer a guided walk-through instead? `magpie quickstart` does one interactively.

## Where matches show up

- In the pipeline's terminal output: the `log` action writes one line per match,
  tagged with the watch's prefix (e.g. `[oss starter]`, `[devtools starter]`).
  Run a pass with `make local-tick` (poll, then trigger, drain, flush).
- In the CLI activity log: `magpie activity summary --action <action_id>` (run
  `magpie auth login` first).

The web UI does not show matches yet, so check the terminal or the CLI.

## Upgrading to a push

Each starter watch ends with a commented `webhook` action. Uncomment it and
point `url` at your notifier (ntfy, or a relay like a Slack/Discord webhook or
openclaw-style instance) to get pushed instead of (or alongside) the log line. A
webhook also records a delivery audit you can inspect with
`magpie delivery list --action <action_id>`.

</doc>

<doc path="examples/starters/devtools/feed.yaml">

```yaml
# OpenMagpie starter: developer-tooling listening.
# Apply by hand (this starter isn't wired into the quickstart seed script):
#   magpie feed create -f examples/starters/devtools/feed.yaml
#   magpie watch create -f examples/starters/devtools/watch.yaml   (after the edits in examples/README.md)
# Then set a past last_event_at on each source, or the first tick only sees
# brand-new posts.
# Full walkthrough: examples/README.md
name: "Developer tooling (starter)"
kind: curated
poll_interval_seconds: 300
data:
  retention_days: 30
sources:
  - spec: {kind: reddit_subreddit, subreddit: devops}
  - spec: {kind: reddit_subreddit, subreddit: programming}

# Backfill window, set by hand:
# Each source can carry a `last_event_at`: the point in time it starts reading
# from. A past UTC timestamp pulls everything posted since then on the FIRST
# tick (so you have posts to score right away); leave it off and the first tick
# only sees posts that arrive AFTER you create the feed. It is a sibling key of
# `spec` (same indent), not a field inside the spec map. A source with it set
# looks exactly like this:
#
# sources:
#   - spec: {kind: reddit_subreddit, subreddit: devops}
#     last_event_at: 2026-06-01T00:00:00Z   # read posts from 2026-06-01 up to now
#   - spec: {kind: reddit_subreddit, subreddit: programming}
#     last_event_at: 2026-06-01T00:00:00Z
```

</doc>

<doc path="examples/starters/devtools/watch.yaml">

```yaml
# Companion watch for the developer-tooling starter. See feed.yaml and
# examples/README.md. Apply with `magpie watch create -f` after creating the
# feed, and set its real id below (replacing REPLACE_WITH_FEED_ID).
name: "Dev tool opportunities (starter)"
is_active: true
feed_ids:
  - REPLACE_WITH_FEED_ID
actions:
  - kind: semantic_filter
    config:
      instructions: "Someone hitting a problem a developer tool could solve, or asking for a tool recommendation."
      threshold: 0.6
  - kind: log
    config:
      prefix: "[devtools starter]"
  # Prefer a push over a log line? Uncomment and point at your notifier (ntfy, or
  # a relay like a Slack/Discord webhook or openclaw-style instance). A webhook also
  # records a delivery audit you can inspect with `magpie delivery list --action <action_id>`:
  # - kind: webhook
  #   config:
  #     url: "https://your-notifier.example/hook"
  #     method: POST
```

</doc>

<doc path="examples/starters/hackernews/feed.yaml">

```yaml
# OpenMagpie starter: Hacker News story listening.
# Uses the `hn_feed` source kind (the new / Show HN / Ask HN story feeds).
# Stories are low-volume (~1k/day), so this is safe to apply as-is.
# Want HN COMMENTS too? That's a separate, high-volume starter on purpose:
# see examples/starters/hackernews-comments/ (read its warning first).
# Apply by hand (this starter isn't wired into the quickstart seed script):
#   magpie feed create -f examples/starters/hackernews/feed.yaml
#   magpie watch create -f examples/starters/hackernews/watch.yaml   (after the edits in examples/README.md)
# Then set a past last_event_at on each source, or the first tick only sees
# brand-new posts.
# Full walkthrough: examples/README.md
name: "Hacker News stories (starter)"
kind: curated
poll_interval_seconds: 300
data:
  retention_days: 30
sources:
  # All new HN stories. `feed` also takes `show` (Show HN) or `ask` (Ask HN).
  - spec: {kind: hn_feed, feed: new}
  # `query` is OPTIONAL here (stories are low-volume ~1k/day, so the default judges
  # every story); set it to cut LLM cost when you only care about a keyword. It
  # pre-filters server-side on the story title / url / text (not the author).
  # Operators:
  #   apple banana    every word must appear (AND, the default)
  #   match: any      any word qualifies (OR; all-word matches still rank higher)
  #   -word           exclude that word
  #   "exact phrase"  match the phrase; quote it in YAML, e.g. query: '"service mesh"'
  # e.g.  - spec: {kind: hn_feed, feed: new, query: "rust", match: any}
  # Add Show HN / Ask HN too by uncommenting:
  #   - spec: {kind: hn_feed, feed: show}
  #   - spec: {kind: hn_feed, feed: ask}

# Backfill window, set by hand:
# Each source can carry a `last_event_at`: the point in time it starts reading
# from. A past UTC timestamp pulls everything posted since then on the FIRST
# tick (so you have posts to score right away); leave it off and the first tick
# only sees posts that arrive AFTER you create the feed. It is a sibling key of
# `spec` (same indent), not a field inside the spec map. A source with it set
# looks exactly like this:
#
# sources:
#   - spec: {kind: hn_feed, feed: new}
#     last_event_at: 2026-06-01T00:00:00Z   # read posts from 2026-06-01 up to now
```

</doc>

<doc path="examples/starters/hackernews/watch.yaml">

```yaml
# Companion watch for the Hacker News stories starter. See feed.yaml and
# examples/README.md. Apply with `magpie watch create -f` after creating the
# feed, and set its real id below (replacing REPLACE_WITH_FEED_ID).
name: "Open-source launches (starter)"
is_active: true
feed_ids:
  - REPLACE_WITH_FEED_ID
actions:
  - kind: semantic_filter
    config:
      instructions: "A Hacker News post announcing the launch of an open-source project: a new tool, library, or app released under an open-source license (often a 'Show HN'). Not articles about open source in general, and not closed-source or paid products."
      threshold: 0.6
  - kind: log
    config:
      prefix: "[hn oss launch]"
  # Prefer a push over a log line? Uncomment and point at your notifier (ntfy, or
  # a relay like a Slack/Discord webhook or openclaw-style instance). A webhook also
  # records a delivery audit you can inspect with `magpie delivery list --action <action_id>`:
  # - kind: webhook
  #   config:
  #     url: "https://your-notifier.example/hook"
  #     method: POST
```

</doc>

<doc path="examples/starters/hackernews-comments/feed.yaml">

```yaml
# OpenMagpie starter: Hacker News COMMENT listening.
# Uses the `hn_comment` source kind: a keyword-filtered slice of HN's comment
# stream. This is a SEPARATE starter from `hackernews` (stories) ON PURPOSE:
# comments are high-volume, so you opt into them deliberately.
# Apply by hand (this starter isn't wired into the quickstart seed script):
#   magpie feed create -f examples/starters/hackernews-comments/feed.yaml
#   magpie watch create -f examples/starters/hackernews-comments/watch.yaml   (see examples/README.md)
# Full walkthrough: examples/README.md
#
# LOCAL PROCESSING WARNING
# Every matching comment costs ONE LLM call (the semantic_filter). HN's full
# comment stream averages ~20k/day (bursty, heavier on weekdays); a single
# local engine at ~6s/comment tops out near ~14k/day, so it CANNOT keep up with
# the unfiltered stream. The REQUIRED `query` keyword is what keeps you safe: a
# tight keyword (a product or project name) matches ~dozens to hundreds/day; a
# BROAD keyword can still outrun local processing and back the poll queue up.
# Keep it tight; widen only if you have the throughput to spare.
name: "Hacker News comments (starter)"
kind: curated
poll_interval_seconds: 300
data:
  retention_days: 30
sources:
  # `query` is REQUIRED and non-blank here, to keep a watch on the comment stream
  # focused; it pre-filters the comment body server-side (not the author / title).
  # Operators:
  #   apple banana    every word must appear (AND, the default)
  #   match: any      any word qualifies (OR; all-word matches still rank higher)
  #   -word           exclude that word
  #   "exact phrase"  match the phrase; quote it in YAML, e.g. query: '"service mesh"'
  # Keep it TIGHT: HN comments run ~20k/day and each match costs one LLM call, so a
  # broad single word can outrun a local engine. CHANGE this to your own keyword.
  - spec: {kind: hn_comment, query: "kubernetes"}

# Backfill window, set by hand:
# A `last_event_at` sibling of `spec` (same indent) pulls everything since a
# past UTC timestamp on the FIRST tick.
#
# Be conservative here: a wide backfill window combined with a broad query
# enqueues a lot of LLM work at once. A source with it set looks like this:
#
# sources:
#   - spec: {kind: hn_comment, query: "kubernetes"}
#     last_event_at: 2026-06-01T00:00:00Z
```

</doc>

<doc path="examples/starters/hackernews-comments/watch.yaml">

```yaml
# Companion watch for the Hacker News comments starter. See feed.yaml (and its
# LOCAL PROCESSING WARNING) and examples/README.md. Apply with
# `magpie watch create -f` after creating the feed, and set its real id below
# (replacing REPLACE_WITH_FEED_ID).
name: "Hacker News mentions (starter)"
is_active: true
feed_ids:
  - REPLACE_WITH_FEED_ID
actions:
  - kind: semantic_filter
    config:
      instructions: "A Hacker News comment that meaningfully discusses, asks about, or reports a problem with <your project or product>. Skip passing mentions and off-topic noise. Edit this to your own."
      threshold: 0.6
  - kind: log
    config:
      prefix: "[hn comments starter]"
  # Prefer a push over a log line? Uncomment and point at your notifier (ntfy, or
  # a relay like a Slack/Discord webhook or openclaw-style instance). A webhook also
  # records a delivery audit you can inspect with `magpie delivery list --action <action_id>`:
  # - kind: webhook
  #   config:
  #     url: "https://your-notifier.example/hook"
  #     method: POST
```

</doc>

<doc path="examples/starters/selfhosted-opensource/feed.yaml">

```yaml
# OpenMagpie starter: self-hosted / open-source listening.
# Apply with the seed flow (it sets the first-tick lookback for you):
#   ./scripts/quickstart/run.sh   (full quickstart)   or   ./scripts/quickstart/seed.sh   (seed only)
# Applying by hand with `magpie feed create -f` works too, but then YOU set a
# past last_event_at on each source or the first tick only sees brand-new posts.
# Full walkthrough: examples/README.md
name: "Self-hosted and open-source (starter)"
kind: curated
poll_interval_seconds: 300
data:
  retention_days: 30
sources:
  - spec: {kind: reddit_subreddit, subreddit: selfhosted}
  - spec: {kind: reddit_subreddit, subreddit: opensource}

# Backfill window, by hand (the seed flow sets this for you; skip it then).
# Each source can carry a `last_event_at`: the point in time it starts reading
# from. A past UTC timestamp pulls everything posted since then on the FIRST
# tick (so you have posts to score right away); leave it off and the first tick
# only sees posts that arrive AFTER you create the feed. It is a sibling key of
# `spec` (same indent), not a field inside the spec map. A source with it set
# looks exactly like this:
#
# sources:
#   - spec: {kind: reddit_subreddit, subreddit: selfhosted}
#     last_event_at: 2026-06-01T00:00:00Z   # read posts from 2026-06-01 up to now
#   - spec: {kind: reddit_subreddit, subreddit: opensource}
#     last_event_at: 2026-06-01T00:00:00Z
```

</doc>

<doc path="examples/starters/selfhosted-opensource/watch.yaml">

```yaml
# Companion watch for the self-hosted / open-source starter. See feed.yaml and
# examples/README.md. The seed wires feed_ids to the created feed automatically;
# hand-applying with `magpie watch create -f` needs the real feed id here.
name: "OSS alternative seekers (starter)"
is_active: true
feed_ids:
  - REPLACE_WITH_FEED_ID
actions:
  - kind: semantic_filter
    config:
      instructions: "Someone asking for an open-source or self-hostable alternative to a paid or SaaS tool."
      threshold: 0.6
  - kind: log
    config:
      prefix: "[oss starter]"
  # Prefer a push over a log line? Uncomment and point at your notifier (ntfy, or
  # a relay like a Slack/Discord webhook or openclaw-style instance). A webhook also
  # records a delivery audit you can inspect with `magpie delivery list --action <action_id>`:
  # - kind: webhook
  #   config:
  #     url: "https://your-notifier.example/hook"
  #     method: POST
```

</doc>

<doc path="examples/starters/youtube/feed.yaml">

```yaml
# OpenMagpie starter: YouTube brand-mention listening.
# Uses the `youtube_search` source kind (yt-dlp, public search; no API key or
# credentials needed). Each poll runs the query filtered to the last week's
# uploads and surfaces videos newer than the source's watermark.
# Apply by hand (this starter isn't wired into the quickstart seed script):
#   magpie feed create -f examples/starters/youtube/feed.yaml
#   magpie watch create -f examples/starters/youtube/watch.yaml   (after the edits in examples/README.md)
# Then set a past last_event_at on each source, or the first tick only sees
# brand-new videos. NOTE: YouTube timestamps can be day-granular, so a
# same-day item may appear once more across ticks; the item store dedups it.
# Full walkthrough: examples/README.md
name: "YouTube mentions (starter)"
kind: curated
poll_interval_seconds: 900
data:
  retention_days: 30
sources:
  # `query` is REQUIRED (the firehose guard): it pre-filters server-side on
  # YouTube's search index before any per-item LLM cost. Quote exact phrases
  # inside the YAML string, e.g. query: '"open source social listening"'.
  # `count` caps the per-poll fetch (1-50). Each result costs one full
  # YouTube page fetch per poll (metadata extraction), so the default 20 at
  # this cadence is already ~2k fetches/day for the source. Raise it only
  # when a query matches more than `count` videos in a week: results are
  # relevance-ranked within the week window, so a busy query can squeeze
  # fresh low-view mentions out of the visible slots.
  - spec: {kind: youtube_search, query: '"social listening"', count: 20}
  # Watch several phrasings by adding more sources:
  #   - spec: {kind: youtube_search, query: '"brand monitoring" open source', count: 20}
```

</doc>

<doc path="examples/starters/youtube/watch.yaml">

```yaml
# Companion watch for the YouTube mentions starter. See feed.yaml and
# examples/README.md. Apply with `magpie watch create -f` after creating the
# feed, and set its real id below (replacing REPLACE_WITH_FEED_ID).
name: "YouTube tool mentions (starter)"
is_active: true
feed_ids:
  - REPLACE_WITH_FEED_ID
actions:
  - kind: semantic_filter
    config:
      instructions: "A YouTube video that reviews, demos, or recommends social listening or brand monitoring tools: tutorials, tool roundups, or hands-on walkthroughs. The video description is the judged text. Not general marketing talk that only mentions listening in passing."
      threshold: 0.6
  - kind: log
    config:
      prefix: "[youtube mention]"
  # Prefer a push over a log line? Uncomment and point at your notifier (ntfy, or
  # a relay like a Slack/Discord webhook or openclaw-style instance). A webhook also
  # records a delivery audit you can inspect with `magpie delivery list --action <action_id>`:
  # - kind: webhook
  #   config:
  #     url: "https://your-notifier.example/hook"
  #     method: POST
```

</doc>