BUCKYVERSE

Summary

Of the 104 .wiki/raw/** records with fetch_status: failed, only 33 have HTTP(S) source URLs testable by agent-browser. The other 71 have zotero://select/... deep links that reference the local Zotero database rather than the web.

Testing all 33 HTTP URLs with agent-browser read:

The dominant blocking categories are dead/removed content (14 URLs, 42.4%) and bot detection (10 URLs, 30.3%). Browser rendering helps with user-agent-blocked sites but cannot recover content that no longer exists or that requires authentication.

Background

The Buckyverse Zotero ingest pipeline (zotero/ingest_to_wiki.py) fetches live URL bodies using Python urllib.request with a buckyverse-ingest/1.0 user agent and a short timeout. During the 2026-06-30 ingest, many records were written with fetch_status: failed because sites blocked non-browser user agents, required JavaScript rendering, returned paywalls or login walls, used bot protection, or were no longer available.

agent-browser is a Rust CLI that drives a real Chrome instance through the Chrome DevTools Protocol and returns agent-friendly markdown. It renders JavaScript and maintains browser state, making it a candidate for recovering content that simple HTTP clients cannot fetch.

This research was initiated as TODO-20 to determine whether agent-browser can recover failed ingest URLs and to classify the blocking mechanisms at work.

Methodology

Corpus extraction

A Python extraction script (zotero/extract_failed_fetch_urls.py) scans all markdown files under .wiki/raw/**, parses YAML frontmatter, and selects records where:

Testing protocol

Each corpus URL was tested with:

agent-browser read <url> --timeout 30000

Classification

Each URL received a status (recovered / partial / still_blocked) and a blocking_type based on automated heuristics refined by manual deep-dive review of every non-recovered URL.

Results

Status breakdown

Status Count %
recovered 4 12.1%
partial 2 6.1%
still_blocked 27 81.8%
Total 33 100%

Recovered URLs

Domain Title Chars
sondrabarrett.com Books page 2,986
flextegrity.com Homepage 8,395
www.artsy.net 4D Time World artwork 7,525
blog.cjfearnley.com Objective/Subjective article 373,383

Blocking type breakdown

Blocking Type Count % Agent-browser helps?
none 4 12.1% Yes — recovered
dead_or_removed 14 42.4% No — content gone
bot_detection 10 30.3% No — challenges unsolved
soft_paywall_or_meter 2 6.1% No — network-level block
hard_paywall 2 6.1% No — subscription required
javascript_required_unresolved 1 3.0% Partially — needs multi-step
non_html_or_binary 0 0% N/A

Note: The 2 partial results (blog.cjfearnley.com PDF, doi.org citation snippets) have blocking types non_html_or_binary and hard_paywall respectively, which are counted in the totals above.

Blocking taxonomy

dead_or_removed (14 URLs)

Content no longer exists at the URL. Causes include DNS resolution failure (tripod.com), expired SSL certificates (thenewpress.com, na2ure.org), HTTP 404/410 responses (artsy.net, loc.gov), domain parking (dmass.net), and HTTP 300 redirects to nowhere (cjfearnley.com). No browser technology can recover content that has been removed from the web.

Affected domains: artsy.net (2), stoutbooks.com (2), bobwb.tripod.com, en.wikipedia.org, www.dmass.net, black-mountain-research.com, solwaygallery.com, beaconbooks.net, www.loc.gov, na2ure.org, thenewpress.com, www.cjfearnley.com

bot_detection (10 URLs)

Sites that detect and block automated access. Cloudflare protection (worldscientific.com, normanfosterfoundation.org), CAPTCHA challenges (jstor.org), and access-denied responses (nytimes.com, fastcompany.com, mitpress.mit.edu, direct.mit.edu). agent-browser read drives a real Chrome instance but does not attempt to solve challenges or pass bot verification checks.

Affected domains: jstor.org (2), mitpress.mit.edu (2), www.nytimes.com, www.worldscientific.com, normanfosterfoundation.org, www.fastcompany.com, direct.mit.edu, theculturetrip.com

soft_paywall_or_meter (2 URLs)

Washington Post archive URLs time out at the OS level — the connection never completes, suggesting network-level blocking rather than an HTTP-layer paywall modal. Both URLs are historical archive articles.

hard_paywall (2 URLs)

DOI-resolved URLs to academic publishers (ACS Nano via doi.org/10.1021/..., SAGE via doi.org/10.1177/...). The DOI resolves and returns brief citation metadata (265–393 chars) but the full article text requires a paid subscription. These are legitimate access restrictions.

javascript_required_unresolved (1 URL)

archive.org search page returns empty content — the search UI is heavily JavaScript-dependent and agent-browser read does not produce usable output in a single pass. A multi-step interaction (navigate, wait for JS hydration, extract results) might work but is outside the scope of agent-browser read.

Remaining gaps

Browser rendering does not solve:

  1. Dead content — 42% of failures are URLs that no longer exist. The only recovery path is web archives (Wayback Machine) or alternative sources.
  2. Aggressive bot protection — 30% of failures use challenge-based protection that agent-browser read does not bypass. These sites could potentially be accessed with authenticated browser profiles, but that conflicts with the safety constraints of unauthenticated, non-invasive testing.
  3. Paid content — 12% of failures require paid subscriptions. These are intentional access restrictions, not technical failures.
  4. Non-HTTP source URLs — 71 of 104 failed records have zotero:// deep links instead of web URLs. These need the original Zotero record's web URL resolved before any browser testing is possible.

Recommendations

  1. Integrate agent-browser for retryable failures: Add an optional agent-browser read retry step to future ingest runs for URLs that fail with urllib.request. This would recover the ~12% of URLs blocked by simple user-agent filtering. Implement as a separate retry pass, not a replacement for urllib.request.

  2. Mark dead URLs for archival lookup: The 14 dead/removed URLs should be checked against the Wayback Machine (web.archive.org) or other web archives. This is a separate research task.

  3. Accept paywall and bot-detection blocks: Hard paywalls, login walls, and bot challenges are legitimate access restrictions. Mark these as permanent failures in the ingest pipeline rather than retrying.

  4. Resolve zotero:// deep links: The 71 records with zotero://select/... source URLs need their web URLs extracted from the Zotero database. This is a data quality issue in the ingest pipeline, not a browser-access issue.

  5. Consider PDF extraction for binary URLs: One URL returned raw PDF content. A PDF-to-text extraction step (e.g., pdftotext or a Python library) could recover this content without browser rendering.

  6. Use agent-browser's interactive mode for JS-heavy sites: The archive.org search page and similar JavaScript-heavy sites might yield results with agent-browser's interactive navigation rather than the single-shot read command.

Sources