Searching Local News Without a Search Engine

Introduction

Building a collection of URLs from Google or DuckDuckGo has become an increasingly brittle endeavor. Search-result markup and access rules change, automated requests get blocked, and endpoints disappear. Commercial search APIs are more stable, but they introduce keys, usage fees, and dependence on changing terms of service.

The loss matters especially for local news. Local outlets cover school boards, county commissions, rural hospitals, water systems, fires, festivals, and development disputes that rarely enter a coherent national stream. The 2026 Local Journalist Index found severe reporting shortages in 70% of U.S. counties, with most counties producing no local education or health coverage during the period studied. The local reporting that remains is scattered across individual outlet sites and difficult to search as a collection. RSS provides a practical way to bring it together: choose the publishers, collect what they publish, and search the resulting material locally. Feed URLs also move or disappear as publishers change platforms, so the source list has to be maintained. We combine local-rags-rss, a directory of local publishers and validated feeds, with textpress for collection, processing, and search.

The directory starts with the outlet registry assembled by 3DLNews and the Local Memory Project, then validates inherited feeds, discovers replacements, and reconciles outlet identities. The result is a national set of working RSS and Atom endpoints with county and Census geography. We use it to collect a recent national news snapshot, search RSS descriptions with a BM25 index, narrow results by place, and retrieve full articles for a small RAG example.

A National Directory of Local Feeds

library(dplyr)

We begin with the validated feed directory produced by local-rags-rss.

local_rags_path <- "../packages/local-rags-rss"

local_rags <- readr::read_csv(
  file.path(local_rags_path, "output/local_rags_rss.csv"),
  col_types = readr::cols(fips = readr::col_character())
)

Each row pairs a local outlet and validated feed with county and Census geography. The feed URL tells us where to collect recent items; the geographic fields let us map coverage and filter search results by region, state, county, or metro status.

options(tigris_use_cache = TRUE)
conus <- c(state.abb, "DC") |> setdiff(c("AK", "HI"))

states_sf <- tigris::states(
  cb = TRUE,
  resolution = "20m",
  year = 2020
) |>
  filter(STUSPS %in% conus) |>
  sf::st_transform(5070) |>
  select(state_abbr = STUSPS, geometry)

division_counts <- local_rags |>
  filter(state_abbr %in% conus) |>
  count(census_division, name = "feeds")

division_sf <- states_sf |>
  left_join(
    local_rags |> distinct(state_abbr, census_division),
    by = "state_abbr"
  ) |>
  group_by(census_division) |>
  summarise(.groups = "drop") |>
  left_join(division_counts, by = "census_division")

division_colors <- c(
  "New England" = "#9467bd",
  "Middle Atlantic" = "#5ba3a0",
  "South Atlantic" = "#DD8E58",
  "East North Central" = "#708A81",
  "East South Central" = "#C2956E",
  "West North Central" = "#B08968",
  "West South Central" = "#A8757B",
  "Mountain" = "#C48A5A",
  "Pacific" = "#7A8B99"
)

ggplot2::ggplot(division_sf) +
  ggplot2::geom_sf(ggplot2::aes(fill = census_division), color = "white", linewidth = 0.25) +
  ggplot2::geom_sf_text(
    ggplot2::aes(label = paste0(census_division, "\n(", feeds, ")")),
    color = "white",
    size = 2.7,
    fontface = "bold"
  ) +
  ggplot2::scale_fill_manual(values = division_colors, guide = "none") +
  ggplot2::labs(title = "Validated RSS Feeds by Census Division") +
  ggplot2::theme_void() +
  ggplot2::theme(plot.title = ggplot2::element_text(size = 16))

conus_state_fips <- local_rags |>
  filter(state_abbr %in% conus, !is.na(fips)) |>
  transmute(STATEFP = stringr::str_sub(fips, 1, 2)) |>
  distinct() |>
  pull(STATEFP)

counties_sf <- tigris::counties(
  cb = TRUE,
  resolution = "20m",
  year = 2020
) |>
  filter(STATEFP %in% conus_state_fips) |>
  sf::st_transform(5070) |>
  select(fips = GEOID, geometry)

county_counts <- local_rags |>
  filter(!is.na(fips)) |>
  count(fips, name = "feeds")

counties_sf |>
  left_join(county_counts, by = "fips") |>
  mutate(has_feed = !is.na(feeds)) |>
  ggplot2::ggplot() +
  ggplot2::geom_sf(ggplot2::aes(fill = has_feed), color = "white", linewidth = 0.03) +
  ggplot2::geom_sf(data = states_sf, fill = NA, color = "white", linewidth = 0.3) +
  ggplot2::scale_fill_manual(
    values = c(`TRUE` = "#708A81", `FALSE` = "grey85"),
    labels = c(`TRUE` = "Has a feed", `FALSE` = "No feed"),
    name = NULL
  ) +
  ggplot2::labs(title = "County Coverage in the RSS Directory") +
  ggplot2::theme_void() +
  ggplot2::theme(plot.title = ggplot2::element_text(size = 16), legend.position = "bottom")

rucc <- readr::read_csv(
  file.path(local_rags_path, "config/rucc_2023.csv"),
  col_types = readr::cols(fips = readr::col_character())
)

local_rags <- local_rags |>
  left_join(
    rucc |> select(fips, rucc_2023, metro_status),
    by = "fips"
  )

local_rags |>
  filter(!is.na(metro_status)) |>
  summarise(
    feeds = n(),
    counties = n_distinct(fips),
    .by = metro_status
  ) |>
  DT::datatable(rownames = FALSE, options = list(dom = "t"))

Build the National RSS Snapshot

textpress::fetch_rss() retrieves recent entries from every feed in the local directory. The descriptions arrive with the feeds, so this does not scrape the linked articles or download their full text.

The collection step is shown below. The analysis uses a saved snapshot so the same set of stories is searched each time the post is rendered.

rss_items <- textpress::fetch_rss(
  local_rags$feed_url,
  cores = 4
) |>
  left_join(
    local_rags |>
      select(
        feed_url, outlet, state_abbr, census_region,
        census_division, county, fips, metro_status
      ),
    by = "feed_url"
  )

The saved snapshot contains 31,385 feed items collected on August 27, 2026.

rss_items <- rss_items |>
  mutate(
    published_at = lubridate::ymd_hms(published_at, quiet = TRUE),
    description = description |>
      stringr::str_remove_all("<[^>]+>") |>
      stringr::str_remove(stringr::regex("The post .* appeared first on .*", ignore_case = TRUE)) |>
      stringr::str_replace_all("&#[0-9]+;|&[A-Za-z]+;", " ") |>
      stringr::str_squish()
  )

A few cleaned feed items look like this.

Dates, Feed Depth, and Syndication

RSS feeds expose different amounts of history. One publisher may return ten items from this week, another thirty from the past month, and a quiet feed may return much older material. We use a common 30-day window anchored to the newest item in the snapshot.

snapshot_date <- max(as.Date(rss_items$published_at), na.rm = TRUE)

recent_items <- rss_items |>
  filter(
    as.Date(published_at) >= snapshot_date - 30,
    !is.na(description),
    description != ""
  )

Syndication presents a different problem. The same press release or wire story can appear at several outlets. We collapse exact matches after normalizing titles and descriptions, while retaining the number of outlets that carried each item. Near-duplicates remain in the corpus.

search_items <- recent_items |>
  mutate(
    title_key = title |> stringr::str_to_lower() |> stringr::str_remove_all("[^[:alnum:]]"),
    description_key = description |>
      stringr::str_to_lower() |>
      stringr::str_remove_all("[^[:alnum:]]")
  ) |>
  group_by(title_key, description_key) |>
  mutate(syndication_count = n_distinct(outlet)) |>
  slice_max(published_at, n = 1, with_ties = FALSE) |>
  ungroup() |>
  mutate(search_id = paste0("item-", row_number()))

A Local BM25 Index

textpress also supports search_regex(), search_dict(), and search_vector(). Here we use search_index() and BM25: a good middle ground for exploratory keyword searches when the wording is not known exactly but the query terms should still appear in the result. BM25 ranks documents using the query terms they contain, how often those terms occur, how unusual the terms are across the collection, and document length. Here the searchable document is only the publisher-provided RSS description.

tokens <- search_items |>
  transmute(search_id, text = description) |>
  textpress::nlp_tokenize_text(
    by = "search_id",
    id_col = "search_id",
    include_spans = FALSE
  )

index <- textpress::nlp_index_tokens(
  tokens,
  stem = TRUE
)

Each query calls textpress::search_index() directly. We join all matching descriptions to the feed metadata before filtering by geography, then keep the highest-ranked local results.

show_hits <- function(hits, caption) {
  hits |>
    mutate(
      date = as.Date(published_at),
      story = paste0(
        '<a href="', url,
        '" target="_blank" rel="noopener noreferrer">',
        title,
        "</a>"
      )
    ) |>
    select(date, outlet, state_abbr, county, metro_status, story) |>
    DT::datatable(
      rownames = FALSE,
      escape = FALSE,
      caption = caption,
      options = list(pageLength = 5, scrollX = TRUE, dom = "tip")
    )
}

What Is Local News Talking About?

A national search can surface subjects that are locally important but diffuse across thousands of publishers.

county_fair_hits <- textpress::search_index(
  index, "county fair livestock",
  n = nrow(search_items), stem = TRUE
) |>
  left_join(search_items, by = "search_id") |>
  slice_head(n = 15)

county_fair_hits |>
  show_hits("National results: county fair livestock")

The geographic metadata makes more pointed comparisons possible. The same index can ask about wildfire in western states or water across the Mountain division.

wildfire_hits <- textpress::search_index(
  index, "wildfire evacuation",
  n = nrow(search_items), stem = TRUE
) |>
  left_join(search_items, by = "search_id") |>
  filter(state_abbr %in% c("CA", "OR", "WA", "ID", "MT", "NM", "AZ", "CO")) |>
  slice_head(n = 15)

wildfire_hits |>
  show_hits("Western states: wildfire evacuation")

water_hits <- textpress::search_index(
  index, "water drought irrigation",
  n = nrow(search_items), stem = TRUE
) |>
  left_join(search_items, by = "search_id") |>
  filter(census_division == "Mountain") |>
  slice_head(n = 15)

water_hits |>
  show_hits("Mountain division: water drought irrigation")

Metro status adds another useful comparison. The spread of data centers into rural counties connects a national industry to local questions about infrastructure, land, water, taxes, and public services.

data_center_hits <- textpress::search_index(
  index, "data center",
  n = nrow(search_items), stem = TRUE
) |>
  left_join(search_items, by = "search_id") |>
  filter(
    metro_status == "Nonmetro",
    stringr::str_detect(
      paste(title, description),
      stringr::regex("\\bdata centers?\\b", ignore_case = TRUE)
    )
  ) |>
  slice_head(n = 15)

data_center_hits |>
  show_hits("Nonmetro counties: data centers")

data_center_articles <- data_center_hits |>
  textpress::read_urls(cores = 4)

rag_documents <- data_center_text |>
  arrange(doc_id, node_id) |>
  summarise(article_text = paste(text, collapse = "\n"), .by = doc_id) |>
  inner_join(
    data_center_hits |>
      select(
        doc_id, title, outlet, state_abbr, county, published_at,
        author, url, feed_url
      ),
    by = "doc_id"
  )

rag_context <- rag_documents |>
  transmute(
    source = paste0(
      "OUTLET: ", outlet, "\n",
      "TITLE: ", title, "\n",
      "LOCATION: ", county, ", ", state_abbr, "\n",
      "PUBLISHED: ", published_at, "\n",
      "AUTHOR: ", coalesce(author, "Not listed"), "\n",
      "ARTICLE URL: ", url, "\n",
      "FEED URL: ", feed_url, "\n",
      "FULL ARTICLE:\n", article_text
    )
  ) |>
  pull(source) |>
  paste(collapse = "\n\n")

chat <- ellmer::chat_openai(
  system_prompt = "Answer using the supplied local-news articles.",
  model = "gpt-5-mini-2025-08-07",
  params = ellmer::params(reasoning_effort = "low"),
  echo = "none"
)

rag_answer <- chat$chat(paste(
  "Write cohesive prose for publication. Start directly with the topic and",
  "its central tension. Organize the main patterns under short Markdown",
  "subheadings formatted as `#### Heading`, followed by developed paragraphs.",
  "Write",
  "location names naturally into sentences and use",
  "publishing outlet names for parenthetical citations.\n\n",
  rag_context
))

Summary

Together, these tools build a searchable national collection from known local publishers. local-rags-rss supplies the publishers and geography, RSS supplies a recent stream of stories, and textpress indexes their descriptions on a local machine. Full articles enter only after a geographic BM25 search has narrowed the collection to a manageable set.

The larger value is visibility. At a time when local reporting is thinning, this collection makes work scattered across hundreds of small outlets easier to find and compare without stripping away its source or geography. It keeps reporting about local institutions, conflicts, and everyday civic life in view, while making patterns across communities easier to see.