New ask Hacker News story: One Wikipedia page costs your AI agent 68,000 tokens
One Wikipedia page costs your AI agent 68,000 tokens 6 by arhamislam5766 | 1 comments on Hacker News. i use claude code daily and measured what pages cost it while doing research. an average wikipedia article, for instance, is 68,240 tokens of raw html (tiktoken); nike's homepage is 353,000. claude code's built-in webfetch handles the easy case well. it summarizes wikipedia to about 950 tokens and clears cloudflare on some sites like indeed and ticketmaster. but, and there's always a but, on js-rendered and some anti-bot pages it returns nothing. quotes.toscrape.com/js gives "no quotes found"; nike.com gives a 403. your agent then dumps the raw html back into context and still fails. (note: i have also had cases where i read through the chat at the end and saw that it failed and just pulled from either training data or stale caches from other sources) so i worked on building an open-source stealth browser (recompiled chromium) that runs as an mcp for claude cod...