Building an intelligence research stack

In one sitting we built a small open-source intelligence stack in Ruby: it gathers public information on a subject, keeps the cost near zero, writes cited reports, and refuses to go where it shouldn't. This is the whole build, the numbers, the honest limits, and three worked case studies.

Everything here runs on one Linux machine. The code lives in apps/intel. Every figure below was measured on 2026-10-18; where a claim is reasoning rather than measurement, it says so. The collection tools spend no language-model tokens at all.

1. Retrieval first

The foundation is retrieval-augmented generation: answering from your own documents, with citations, and saying "I don't know" when the documents don't say. We wrote that up separately as a full how-to, with real chunking math, the retrieval metrics (hit rate, MRR, NDCG) and an honest benchmark against naive RAG: Retrieval-augmented generation on ruby.im.

The one idea that carries into everything below: a citation is only as good as what it points at, so the system keeps the source, the page and a content hash with every chunk, and a grounding step checks that each claim is actually supported before anyone sees it. Intelligence work has the same rule, with a harder twist: much of the source material is hostile, and a rumour is not a fact.

2. From RAG to intelligence analysis

RAG answers questions from a trusted corpus. Intelligence analysis asks a different thing: gather everything said about a subject, including rumours and contradictions, then reason about it without being fooled. We designed a system for that, built around three ideas:

• The claim is the unit, not the document. A model breaks each source into atomic claims, tagged fact, opinion, rumour or prediction, each tied to its earliest traceable source. Five accounts copying one post are one source, not five.
• Competing hypotheses, scored by disconfirmation. We use Heuer's Analysis of Competing Hypotheses: rate each claim against each hypothesis, then rank hypotheses by the evidence against them, not for them. The claims that split the hypotheses are the ones worth chasing next.
• Judgement, measured. The only way to know whether any of this helps is to score forecasts against outcomes that are already known. Without that, a neat matrix over weak claims just looks authoritative.

That design is deliberately cautious: it treats retrieved text as data, never as instructions, and it keeps unknowns unknown. The build below is the collection half of it, which we took all the way to working code.

3. The collection toolkit

The toolkit is a set of small Ruby command-line tools that read and write one JSON record per line, so they pipe together. Collection pulls from free, structured sources that already hand over titles, dates, authors and abstracts, so a language model never has to read the raw pile — it only ever sees a ranked short list.

The core is one HTTP fetcher built to ask each server as little as possible. A fresh cached copy is reused without a request; a stale one is revalidated with ETag and gets a bodyless 304; each host has a minimum gap between requests, and 429/5xx are retried after the delay the server asks for. The result, measured on the topic "sodium-ion" across seven sources:

Between collection and the model sit two cheap, local steps. Dedupe drops the same canonical URL (tracking parameters stripped), identical text, and near-identical text — a wire story syndicated under a different headline — using SimHash over word three-grams. Rank is Okapi BM25 over the records' title and text, in memory, so thousands of records are sorted locally and only the best few dozen cost any model tokens. Both are plain Ruby standard library.

4. Crawling and cited reports

Two outside tools slot in behind the same record format. Katana crawls a single site (public pages, at a polite rate) and its URLs flow straight into the web collector. GPT Researcher writes a cited report, and — the useful part — it searches our collected corpus first, through a small local endpoint that ranks our records with BM25, so it reads what we already gathered instead of re-fetching the web. New web sources it finds are written to a file we can fold back into the corpus.

GPT Researcher is Python (it has no Ruby equivalent), kept in its own environment and called as a subprocess; everything around it is Ruby. It has no grounding check of its own, so we added one: a small tool flags any cited link that is in neither the corpus nor the researcher's own sources. On the first test run (with a small local model) it caught a mangled citation the model had invented. That is the RAG grounding rule from section 1, applied to an agent.

5. More search capacity

To widen reach without paying per query, we run our own meta-search: SearXNG, in Docker, local only, no API keys. It asks many search engines at once and feeds both a collector and the report writer. Search engines throttle heavy use, so it is broad, not bottomless.

Collection also went parallel. The fetcher is safe to share across threads: requests to the same host still take turns with that host's gap, but different hosts run at once. Measured with an empty cache:

6. An access policy, not an access hack

The most important design decision was where to stop. Reachability is not open-or-closed; it is a ladder. A probe makes one honest request per site and labels it:

Each probe is recorded with a date, so "can we read this?" becomes a revisitable fact rather than a standing assumption — you notice the day a site opens up. A companion tool finds the feeds a forum publishes (every major forum engine has a predictable feed path), so you read what a site offers instead of scraping its pages.

The ladder climbs only as far as honest requests, sanctioned APIs and published feeds. It deliberately does not defeat a site's bot defences. Excluded by policy: TLS fingerprint impersonation, Cloudflare challenge-solvers, anti-detection browsers, CAPTCHA-solving services, and throwaway-account pools. Those exist to make a bot look like a person to a site that decided it doesn't want bots — evasion against a party that hasn't agreed, and the wrong side of both the law and our own house rules. The policy per source is api where there is an API, feed where one is published, and respect otherwise. There is no bypass tier. If a walled source matters enough, the real unlock is its API or a data agreement, not a better disguise.

7. Three case studies

Conspiracy forums: access is the real constraint

Asked to collect from conspiracy forums active today, the toolkit's first finding was about access. The large forums (Reddit, Godlike Productions, Above Top Secret, the David Icke forum) sit behind bot walls or disallow crawlers outright. We respected that, and read the open side instead: Lemmy communities, Mastodon hashtags, a skeptics' board, plus news coverage. In a minute of fetching it gathered 184 items, 160 after dedupe. The lesson that shaped the access policy above: the forums where theories start are the ones closed to polite bots, and the honest answer is to say so and record it, not to break in.

AI people: orbits from public record

Pointed at the professional networks around Sam Altman, Elon Musk, Anthropic's leadership and Andrej Karpathy, the toolkit mapped the public affiliations cleanly — who co-founded what, who left where, who co-authored with whom. OpenAlex gives the academic lineage (Karpathy's is his Stanford computer-vision era); news and search give the corporate diaspora, like the well-documented set of startups founded by OpenAI alumni. This part is pure public fact, with sources on every claim. Turning that into a ranked assessment of who is "pushing the edge" is a separate, subjective step — a judgement from public technical output, not something the collection can measure — and we kept that judgement out of this public write-up.

AI skunkworks: a sourced watchlist

The clearest demonstration: the frontier is going quiet, and the toolkit can map it from public reporting alone. A recognisable pattern recurs — a departed frontier-lab leader, a seed round the size of a late-stage round, a one-line mission, and no product. All from funding announcements and press, no private data:

The toolkit maps who and how much, because that is announced. It cannot see inside — the methods and rosters are the whole secret — and reaching them would mean exactly the circumvention and private data the access policy rules out. The honest product is a dated, sourced watchlist, refreshed as the announcements come: funding news, SEC Form D filings, a researcher going quiet, a compute deal.

8. What it can't do, and won't

Two limits, stated plainly. The first is capability: it reads only what is public and structured, so it undercounts closed labs, non-English sources, and anything that lives in private conversation rather than a filing or a feed. The second is by choice: it does not break access controls, it does not touch private individuals' personal data, and it records rumours as attributed claims, never as facts. The difference between "we couldn't get in" and "here is our access policy and the evidence it still holds" is the difference between a scraper and an intelligence practice.

9. Reproduce it

The whole stack is Ruby standard library plus two gems, Docker for SearXNG, and two installed binaries (Katana, and GPT Researcher in its own Python environment). It runs on one machine and keeps its data in open formats on local disk. The engineering rules it follows — Ruby first, surgical changes, measure before optimising, own your stack, test for confidence — are written up at ruby.im.

Provided as is, for information only, with no warranty and no liability. Not legal or professional advice. Security and radio material is for authorized, lawful use only. Full disclaimer.

Visits are logged on our own server: IP address, browser and device details, and pages viewed. We use them to understand traffic and spot attacks, and never share or sell them.