Methodology
How we analyze comment ecosystems, what signals we measure, and what we store.
Pipeline approach
Bot Chorus analyzes comment ecosystems, not individual accounts. The most reliable signal of coordinated inauthentic behavior is not any single account — it's the pattern across thousands of comments on a channel or post: duplicate text, template phrases, suspicious account ages, and repeat-commenter concentration.
Each platform has its own pipeline — a purpose-built analysis sequence that uses whatever public data that platform exposes. Pipelines are developed one at a time, optimized for the specific API constraints and signal availability of each platform.
Four pipelines are live today: YouTube (comment ecosystem analysis), Reddit (subreddit comment analysis using public archives), Bluesky (hashtag reply analysis via the AT Protocol), and an Other data-contribution path for platforms without dedicated pipelines. Each pipeline is reviewed monthly — if new data becomes available or a better signal approach is identified, the pipeline is updated.
YouTube pipeline
Input: a channel name, @handle, channel URL, or a specific video URL. The system auto-detects whether the input is a channel or a video and routes accordingly.
What gets analyzed
- 5 most recent videos from the channel (or 1 specific video)
- Up to 100 top-level comments per video, ordered by most recent
- Up to 50 unique commenter accounts — sampled for age and activity
Incremental analysis
Re-running analysis on a channel or video that has been analyzed before is efficient by design. The system stores a newest_comment_at cursor per video. On a re-run, only comments posted after the previous analysis are fetched — no duplicate data is stored, no quota is wasted re-fetching old comments.
If all 5 previously analyzed videos have no new comments, the next 5 videos from the channel are analyzed instead, advancing the offset. Comment data is stored permanently for future aggregate corpus analysis.
Signals
fires when: >15% of comments are exact or near-exact duplicates
Bots post identical or near-identical text at scale. Duplicate rate is computed per-video and averaged across all analyzed videos.
fires when: >10% of comments match known templates
Common bot-template patterns: sub4sub solicitation, generic praise (nice video, great content), self-promotion, emoji-only comments, first-comment spam.
fires when: Any single account left 5+ comments on one video
Real viewers rarely comment on the same video more than once. Repeat commenter concentration signals coordinated activity from a small number of accounts.
fires when: >10% of sampled commenters created their account within 30 days
Throwaway accounts created shortly before a coordinated campaign are a known bot network signature. Account age is fetched via channels.list in batches of 50 (1 quota unit per batch).
fires when: >15% of sampled commenters have 0 subscribers and 0 videos
Accounts that exist only to comment — no original content, no subscriptions — are a common bot characteristic. These accounts have no incentive to appear authentic beyond their commenting activity.
fires when: >20% of sampled commenters have 0 videos AND account age >90 days
A refinement of zero-activity detection: accounts that are old enough to have uploaded something — but never have. New accounts might be dormant by accident; established channels with zero content are more reliably inauthentic.
fires when: >25% of comments arrived in any single 5-minute window
Real comment sections accumulate organically over time. When more than a quarter of all fetched comments landed within a 5-minute burst, the timing pattern is inconsistent with organic discussion.
fires when: >25% of sampled commenter accounts were created within any 30-day window
Bot networks are often provisioned in batches. When a large fraction of commenters' channels were created in the same month, it suggests a coordinated provisioning event rather than organic community growth.
fires when: >8% of sampled commenters also appear in other analyzed videos in the corpus
The same accounts commenting on multiple, independently analyzed videos is a strong coordination signal — legitimate viewers rarely overlap across unrelated channels at this rate. Cross-references commenter channel IDs against the full youtube_comments corpus.
Quota budget
YouTube Data API v3 free tier: 10,000 units/day. Channel analysis: ~29 units (1 channel lookup + 1 uploads playlist + 5 comment fetches + 2 commenter age batches; channel search costs 100 units when input is not a raw channel ID — average ~29). Video analysis: ~8 units. Approximately 345 channel analyses per day on the free tier.
Reddit pipeline
Input: a subreddit name (e.g. r/politics) or a direct post URL. The system analyzes the comment ecosystem of recent posts within that subreddit — detecting bot-like patterns across accounts and comment timing.
All data is sourced from public Reddit archives with zero API cost: Arctic Shift for post listings and comment history, Pullpush.io as a fallback for very recent comments not yet indexed by Arctic Shift. No Reddit API key or authentication is required.
What gets analyzed
- Up to 20 recent posts fetched — first 5 with archived comment data are analyzed
- Up to 100 comments per post from Arctic Shift (with Pullpush.io fallback)
- Up to 25 unique commenter accounts sampled for age and karma
- Cross-post coordination checked against the full reddit_comments corpus
Incremental analysis
Re-analyzing the same subreddit stores a newest_comment_at cursor. On re-runs, the system skips posts with no new archived comments since the last analysis. Comment data is stored permanently for future aggregate corpus analysis.
Note: public Reddit archives have an indexing lag of hours to a day for very recent posts. The pipeline automatically skips brand-new posts and advances through the 20-post list to find posts with archived comment data.
Signals
fires when: >15% of comments are exact or near-exact duplicates
Identical or near-identical text posted across a subreddit at scale — the clearest automated-posting signal available. Computed as the fraction of comments whose normalized text appears more than once across all analyzed posts.
fires when: >10% of comments match known templates
Common bot-template patterns: vote-manipulation requests, reward/incentive offers, generic engagement bait, referral spam. Template library is updated as new patterns are identified.
fires when: Any single account left 5+ comments across analyzed posts
An account commenting 5+ times across a single subreddit's recent posts is a repeat-commenter flag. Real participants rarely concentrate this heavily on one topic thread cycle.
fires when: >10% of sampled commenters created their account within 30 days
Fresh accounts provisioned just before a coordinated campaign are a known bot-network signature. Account creation dates are fetched from Arctic Shift's user index.
fires when: >20% of sampled commenters have total karma under 100
Accounts with very low combined link and comment karma have almost no authentic posting history. At population scale within a single subreddit, high concentrations indicate throwaways or freshly recycled accounts.
fires when: >25% of comments arrived in any single 5-minute window
Computed via sliding window over comment created_utc timestamps. A burst exceeding 25% of all fetched comments within 5 minutes is inconsistent with organic conversation pace.
fires when: >25% of sampled commenter accounts were created within any 30-day window
Bot networks provision accounts in batches. When a large fraction of commenters on the same subreddit were created in the same month, it suggests a coordinated provisioning event.
fires when: >5% of unique commenters also appear in other analyzed subreddits in the corpus
Accounts showing up across multiple independently analyzed subreddits is a strong cross-platform coordination signal. Cross-references all unique author names against the full reddit_comments table, excluding the current posts. High weight: false positives at >5% are rare.
fires when: >20% of sampled commenters are 90+ days old with total karma under 10
Old accounts with almost no karma are persistent ghost accounts — maintained on the platform but dormant until a campaign activates them. Distinguishes from fresh throwaways, which are covered by the new account signal.
Data sources
Primary: Arctic Shift (arctic-shift.photon-reddit.com) — full Reddit post and comment archive, no authentication. Fallback: Pullpush.io — PushShift-compatible Reddit search, used when Arctic Shift's archive has not yet indexed comments on very recent posts. Both sources are free, require no API key, and are not subject to Reddit's IP-blocking of server environments.
Bluesky pipeline
The Bluesky pipeline analyzes reply ecosystems around a hashtag. Given a hashtag (e.g. #crypto), the pipeline fetches up to 5 recent posts using the AT Protocol's authenticated search, then collects all available replies to each post. Up to 25 unique repliers are sampled for account metadata — indexed date and follower count — to surface coordination patterns.
Bluesky's AT Protocol makes the full reply graph public with no authentication required for individual thread fetches. Hashtag search requires an authenticated session but no special API approval — a standard Bluesky account with an app password is sufficient.
Signals
fires when: >15% of replies are near-identical text
Identical or near-identical reply text posted across hashtag threads — the clearest automated-posting signal. Computed as the average duplicate rate across all analyzed posts.
fires when: >10% of replies match known bot phrasing patterns
Common bot-template patterns: sub4sub solicitation, generic praise, self-promotion, emoji-only replies, first-comment spam. Template library is updated as new patterns are identified.
fires when: A single account posted 5+ replies to this hashtag
Real participants rarely reply to multiple threads on the same hashtag from a single account. High repeat-poster counts indicate scripted or automated engagement.
fires when: >10% of sampled posters created their account in the last 30 days
Fresh accounts provisioned shortly before a coordinated campaign are a known bot-network signature. Account indexed date is fetched from the AT Protocol AppView.
fires when: >15% of sampled posters have 0 followers
Accounts with zero followers have no authentic social graph. At population scale within a hashtag reply set, high concentrations indicate throwaways or freshly provisioned accounts.
fires when: >25% of replies arrived in a single 5-minute window
Computed via sliding window over reply timestamps. A burst exceeding 25% of all fetched replies in 5 minutes is inconsistent with organic conversation pace.
fires when: >25% of sampled accounts were created in the same 30-day window
Bot networks provision accounts in batches. When a large fraction of repliers were indexed in the same month, it suggests a coordinated provisioning event rather than organic community growth.
fires when: >8% of posters here also appear in other analyzed threads in our DB
Accounts showing up across multiple independently analyzed hashtag threads is a strong coordination signal. Cross-references all unique handles against the full bluesky_posts corpus, excluding the current posts.
Data sources
AT Protocol AppView API (public.api.bsky.app) — official Bluesky AT Protocol endpoint. Hashtag search requires an authenticated app-password session; thread and profile fetches are fully public. No quota limits, no paid tier required.
Other — data contribution
For platforms without a dedicated pipeline, Bot Chorus accepts individual account reports via the “Other” submission path. This is not an automated analysis — it is a data contribution. The account is added to our corpus, and no probability score is computed at submission time.
The result framing is intentional: signals that fired, then a closing note that the account has been logged and aggregate analysis may surface patterns as data grows. We do not lead with a percentage for reports without a full pipeline — a number without a methodology behind it is misleading.
As data accumulates from contributions and other pipeline runs, aggregate analysis across the full corpus may surface coordination patterns that individual analysis cannot detect alone. That analysis is planned as a monthly admin review process.
What scores mean
Scores are probability estimates, not accusations. A score of 0.87 means: given all the public signals we can measure, this account has an 87% probability of inauthentic behavior. It does not mean the account is confirmed to be a bot, is controlled by a foreign government, or that any individual is doing anything illegal.
Protected speech, opinion, and analysis of public data is constitutionally protected. We never claim certainty we don't have.
Active signals
These signals fire on every analysis request today.
Handle appears in the 2,843-account IRA troll list from Twitter's 2018 Congressional transparency release. The single highest-weight signal in the system.
Handle is labeled as a bot in the TwiBot-20 academic dataset — 229K Twitter accounts annotated as human or bot by researchers at Tsinghua University (Feng et al., 2020). Match fires when label=bot.
Handle name matches a candidate or committee in FEC campaign finance records, linking the account to documented political spending activity. Full candidate and committee records are stored.
Account appears in GDELT news clusters associated with known influence operation themes — disinformation, election interference, coordinated narratives. All matching article URLs, titles, and themes are stored.
More than 10 intentional captures exist on Archive.today, suggesting researchers or watchdog groups flagged the account for monitoring. All capture timestamps and URLs are stored.
Handle appears in 1+ active /pol/ or /biz/ threads. 1 thread = +10, 3+ threads = +20; 500+ total replies = additional +10. Measures coordinated attention from the platform where Twitter influence campaigns are most frequently seeded 24–72 hours before launch.
Associated Telegram channel has 1,000+ members. 1k+ = +5, 10k+ = +12, 100k+ = +20. Channels at this scale are consistently associated with state-linked influence operations that build large broadcast audiences before amplifying to Twitter.
Evidence signals
These fire on every request and appear in the network graph, but don't move the probability score. They document cross-platform presence and coordination context.
Account has posts or comments on Reddit under this handle. Cross-platform persona continuity and content pattern analysis across communities.
Account has a Bluesky profile under this handle. IRA-style operations often replicate personas across networks simultaneously as a backup or amplification channel.
Handle appears in active /pol/ or /biz/ threads. Presence alone is evidence-only; the scored thread amplification signal (+10–30 pts) fires separately based on thread count and total replies.
Account has a YouTube channel or appears in documentation videos — journalism, research, or watchdog coverage of this handle.
Account has a public Telegram channel or group under this handle. Presence alone is evidence-only; the scored channel scale signal (+5–20 pts) fires separately based on member count.
Account has a Mastodon profile under this handle across 5 checked federated instances. Mastodon is a primary migration target for suspended Twitter accounts, and the infosec.exchange instance is heavily used by disinformation researchers.
Phase 2 signals
Defined but not yet running — require deeper Wayback analysis that runs as a background phase.
Archived snapshots show the bio changed from apolitical to political content — a known coordinated behavior pattern. Bio text from every snapshot is stored.
Posting activity shows no sustained pause aligned with any human sleep pattern across multiple days.
Content is 90%+ focused on a single political narrative with no normal personal, local, or lifestyle content.
Account was dormant for months, then reactivated with sudden high-frequency posting on a single topic.
Profile image changed between archived snapshots — a pattern associated with persona recycling.
Data sources
Queried on every analysis request:
IRA Twitter Dataset
2,843 confirmed IRA accounts from Twitter's 2018 Congressional transparency release.
Stored: Handle match, IRA category label.
IRA Tweet Content (2018)
2.9M tweets from the same 2018 Congressional transparency release — the full text of every post made by the 2,843 confirmed IRA troll accounts. Import deferred pending storage upgrade.
Stored: Table schema exists; data cleared to stay within storage limits. Will return up to 25 most recent tweets per IRA handle match once re-imported.
TwiBot-20 Academic Dataset
229K Twitter accounts annotated as human or bot by researchers at Tsinghua University (Feng et al., 2020). One of the largest academic bot benchmarks. 100 sample accounts imported; full dataset pending author access request.
Stored: Bot/human label, display name, follower/following counts, tweet count, account creation date, verified status (when matched).
GDELT Project
Global news event database updated every 15 minutes. Queried for narrative coordination signals.
Stored: All matching article URLs, titles, domains, publication dates, full GDELT theme tags.
Wayback Machine (CDX API)
Internet Archive snapshot index — up to 500 CDX entries per account.
Stored: Full CDX entry list (timestamp, URL, status code, digest). Bio text, display name, follower count, and profile pic URL from each sampled snapshot.
Archive.today
Intentional researcher captures on archive.ph. High counts signal researcher attention.
Stored: All capture timestamps and archived URLs.
Memento Protocol
Aggregates snapshot counts from 20+ web archives simultaneously via timetravel.mementoweb.org.
Stored: Full memento list (datetime, URI) from all participating archives.
URLScan.io
Security researcher scan history for account profile URLs.
Stored: All scan UUIDs, scan dates, page URLs, and verdict scores.
FEC Open Data API
Federal Election Commission candidate and committee name records.
Stored: All matching candidate and committee records (names, IDs, entity types).
Arctic Shift (Reddit posts + comments)
Full Reddit post and comment archive. For account-level analysis: content authored by the handle. For subreddit pipeline: comment ecosystems on recent posts — no auth required.
Stored: Account analysis: up to 25 posts and 25 comments, subreddit, score, timestamp, URL. Subreddit pipeline: up to 200 comments per post, commenter account ages and karma.
Pullpush.io (Reddit search)
PushShift-compatible Reddit search. Used as fallback in the subreddit pipeline for comments on posts too recent for Arctic Shift to have indexed. Also used to find mentions of a handle by other users.
Stored: Up to 30 community mentions (15 posts + 15 comments): title or body excerpt, subreddit, score, timestamp, and author. Pipeline: comment fallback up to 200 comments per post.
Bluesky AT Protocol
Bluesky social network via the fully public AT Protocol AppView API. No authentication required.
Stored: Profile existence flag, follower/following counts, follower ratio, post count, up to 100 recent posts (text, engagement metrics, timestamps), up to 50 mentions of the handle.
4chan official API (/pol/ + /biz/)
Official 4chan catalog API (a.4cdn.org). Searches active /pol/ and /biz/ threads for the account handle. Live catalog only — historical archives (4plebs) are Cloudflare-blocked from server environments.
Stored: Presence flag, matching thread count, up to 20 thread matches (board, thread number, subject, OP excerpt, reply count, timestamp, and direct thread link).
YouTube Data API v3
Checks for a YouTube channel under the same handle and searches for documentation videos — journalism, research, or watchdog coverage mentioning the account. Evidence-tier signal. Requires YOUTUBE_API_KEY.
Stored: Channel existence flag, subscriber/video/view counts (when found), up to 10 documentation videos (video ID, title, channel, publish date). ~101 quota units per score on a 10,000 unit/day free budget.
Telegram Bot API
Checks whether a public Telegram channel or group exists under the same handle via getChat. Evidence-tier signal. Requires TELEGRAM_BOT_TOKEN. No quota — Bot API is free.
Stored: Channel existence flag, channel title, type (channel/supergroup/group), username, description, and member count (when found).
Mastodon ActivityPub
Checks 5 federated instances in order — mastodon.social, infosec.exchange, hachyderm.io, masto.ai, kolektiva.social — for the same handle. Evidence-tier signal. No authentication required.
Stored: Account existence flag, instance found, profile (followers, following, post count, bio), and up to 40 recent posts (text, engagement counts, timestamps, language).
What we store
Every analysis is persisted. We store the final score, all fired signals, every Wayback snapshot we retrieve, and the full raw API responses from each source. Nothing is discarded.
This matters for two reasons. First, APIs change and go offline — GDELT's schema has shifted, Botometer's endpoint broke in 2023 after Twitter's API changes, and Archive.today's CDX format differs from Memento's. Raw storage lets us re-derive signals from original data as our models improve, without re-querying sources that may no longer be reachable.
Second, when accounts are deleted or suspended, the Wayback snapshots we already pulled become irreplaceable primary source material. A suspended account leaves no trace on the live platform — but if we captured its bio text, display names, follower counts, and profile images during analysis, that record persists in our database permanently.
What we don't do
- Access any private data, direct messages, or non-public content
- Claim to identify individuals or their employers
- Label any account as “confirmed” anything
- Publish the internal attribution list — that stays private and goes to journalists and legislators only
Scoring engine
Signals are collected in parallel from all available sources, then passed as a structured bundle to a judgment layer (JEV — Judgment Engine for Vectors) that weights them into a final probability. When live API data is unavailable, the rule-based fallback applies fixed weights per signal.
Full scoring pipeline source code is available on request for research purposes. Raw signal data is available to API subscribers — see API access.
Comment analysis availability
Analyzing comments an account leaves on other content is one of the strongest bot signals available — bots tend to post identical or templated text at scale, cluster around specific topics, and comment at inhuman frequency. Whether we can access that data depends entirely on the platform.
Live — comment analysis running now
YouTube Data API v3 — commentThreads.list
Fetches up to 100 recent top-level comments on the channel's videos (5 quota units). Detects duplicate flooding, bot-template phrases, and repeat-commenter concentration. Note: inbound analysis — comments left by others on the channel.
Arctic Shift + Pullpush.io — subreddit pipeline
Analyzes the comment ecosystem of a subreddit's most recent posts. Up to 200 comments per post, up to 5 posts analyzed per run. Detects 9 signals: duplicate flooding, bot templates, repeat commenters, new accounts, low karma, timing burst, creation clustering, cross-post coordination, and ghost accounts. No API key or authentication required.
AT Protocol AppView API — account posts
Up to 100 recent posts are already retrieved on every analysis. Detects duplicate post flooding and zero-engagement posting patterns. Fires two scored signals: post duplication (+25 pts at >15% duplicate rate) and zero-engagement pattern (+10 pts when >70% of posts have no engagement and total post count ≥10).
ActivityPub REST API — account statuses
Up to 40 recent posts are retrieved from the first Mastodon instance where the handle is found. Detects duplicate and zero-engagement patterns. Fires two scored signals: post duplication (+20 pts at >15% duplicate rate) and zero-engagement pattern (+8 pts when >70% of posts have no engagement and total ≥10). Lower weights reflect Mastodon's smaller reach relative to Bluesky.
Official 4chan API — thread and reply analysis
Active /pol/ and /biz/ threads are already fetched by the collector. Thread amplification analysis computes thread count and total reply count. A single active thread fires +10; three or more fires +20; 500+ total replies adds an additional +10. Per-author analysis is not applicable on 4chan (no persistent accounts).
Telegram Bot API — channel scale analysis
Public channel membership data is fetched via getChat. Channel scale analysis uses the member count to score platform footprint: 1,000+ members fires +5, 10,000+ fires +12, 100,000+ fires +20. Note: message content analysis (via getHistory) is accessible but not yet built.
Not accessible — platform restrictions prevent analysis
Twitter's v2 API requires a paid plan (~$100+/month) for any programmatic access to tweet content or reply threads. Free tier access was eliminated in 2023.
The Instagram Graph API requires the account owner to authorize access. There is no public API for reading comments on accounts you don't own.
Same restriction as Instagram — the Graph API only allows access to pages and accounts you administer. Public comment scraping is prohibited.
Meta launched a Threads API in 2024, but it requires account authorization and is scoped to the authenticated user's own content. No public read access.
TikTok's Research API can theoretically access comments but requires formal research institution approval. The standard Developer API does not expose comment data for third-party accounts.
Discord bots can only read messages in servers where the bot has been explicitly invited. There is no public API for reading messages from servers you are not a member of.
LinkedIn's API is restricted to company pages and official integrations. There is no public API for reading post comments or activity from other users' profiles.
Snapchat has no public API for content retrieval. Content is ephemeral by design. No comment data is accessible.