By The way the internet works is changing beneath our feet. For decades, websites have been built in HTML — a language designed for human eyes viewing through browsers. But today, a new kind of visitor is crawling the web: AI agents. And they don't see websites the way we do.
Enter Cloudflare's latest feature: Markdown for Agents. Announced on February 12, 2026, this seemingly simple tool automatically converts HTML pages to lightweight Markdown format when AI crawlers request them. The result? An 80% reduction in the data AI systems need to process your content.
But beneath this technical innovation lies a fundamental question that should concern every publisher, from major media houses to Kenyan tech blogs: Who controls how AI consumes our content — and who benefits?
The Old Web: Built for Humans, Inefficient for AI
To understand why this matters, we need to look at how things used to work and in many ways, still do.
When you visit a website, your browser receives HTML code packed with elements designed for human consumption: navigation bars, sidebars, advertisements, tracking scripts, CSS styling, and complex div structures. A simple "About Us" heading might look like this in HTML:
That single heading consumes 12-15 tokens when processed by an AI language model. Tokens are the basic units AI systems use to understand and generate text — and they're expensive. Every API call to systems like ChatGPT, Claude, or Gemini is charged based on tokens consumed.
Now compare that to the Markdown equivalent:
Same meaning. Just 3-4 tokens.
The Cloudflare blog post announcing this feature itself demonstrates the efficiency gap: in HTML format, it weighs in at 16,180 tokens. Convert it to Markdown? Just 3,150 tokens. That's an 80% reduction.
For AI companies crawling millions of websites daily, this adds up to massive computational and financial savings. For websites, it means AI agents can process their content faster and more efficiently.
How Things Are Changing: The Rise of AI Traffic
Here's what's fundamentally different in 2026 compared to just two years ago: AI agents are now a significant source of web traffic
According to Cloudflare's own data, popular coding agents like Claude Code and OpenCode already send "Accept: text/markdown" headers with their requests, asking for Markdown instead of HTML. This isn't a future scenario — it's happening now.
Traditional search engine optimization (SEO) was all about ranking on Google. But today, AI is becoming part of SEO itself. When someone asks ChatGPT "What are the best smartphones under KSh 30,000?", the AI doesn't just make up an answer. It searches the web, crawls tech review sites (potentially including Kenyan tech blogs), extracts the information, and synthesizes a response.
This creates what some are calling "AEO" — AI Engine Optimization. Just as websites once optimized for Google's crawlers, they now need to consider how AI agents will consume their content.
How Cloudflare's Markdown for Agents Works
The technical implementation is elegantly simple. When an AI agent visits a Cloudflare-protected website with this feature enabled, here's what happens:
1. The AI agent sends a request with an "Accept: text/markdown" header, indicating it prefers Markdown format
2. Cloudflare intercepts the request, fetches the original HTML from the origin server
3. Cloudflare converts the HTML to Markdown on-the-fly at the edge
4. The clean Markdown is served to the AI agent instead of bloated HTML
For the website owner, nothing changes. Their site remains in HTML. But AI agents get a streamlined version that's easier and cheaper to process.
Cloudflare even includes an `x-markdown-tokens` header with the response, telling the AI exactly how many tokens the content contains. This helps AI systems plan their processing efficiently.
The Catch: Limited Availability
Here's where Kenyan publishers might hit a roadblock: Markdown for Agents is currently in beta and only available for Pro, Business, and Enterprise Cloudflare plans — not the free tier.
For small Kenyan tech blogs and news sites running on free Cloudflare plans, this feature remains out of reach. The pricing gap is significant:
Free plan: $0/month (no access)
Pro plan: $20/month (access granted)
Business plan: $200/month
Enterprise: Custom pricing
This creates a two-tier web: larger publishers who can afford premium Cloudflare plans can optimize for AI consumption, while smaller publishers cannot.
Why This Matters: The SEO Paradigm Is Shifting
For decades, the internet ran on a simple compact: Google sends you traffic, you show visitors ads or convert them to subscribers, and you make money to fund more content creation. This virtuous cycle powered the entire publishing industry.
AI is breaking that cycle.
The Traffic Apocalypse
Recent data paints a grim picture for publishers globally — and there's no reason to think Kenyan sites are immune:
Global search traffic to publishers dropped by 33% in 2025, according to data from analytics firm Chartbeat
When Google's AI Overviews appear at the top of search results, click-through rates to actual websites drop by 46-79%, depending on the study
Major publishers like Business Insider saw 55% declines in organic search traffic between 2022 and 2025
Zero-click searches (where users get their answer without clicking any link) jumped from 56% to 69% between 2024 and 2025
The reason? AI answers the question directly. Why click through to techinkenya.com to read a review of the latest Samsung phone when ChatGPT or Google's AI Overview can summarize the key points right there in the search results?
The Google Problem: You Can't Say No
This brings us to one of the most controversial aspects of the AI web: Google doesn't let publishers opt out of AI features without sacrificing their search ranking entirely.
Here's the dilemma publishers face:
Google uses a single crawler called "Googlebot" for two purposes:
1. Traditional search indexing (the old Google we all know)
2. AI features like AI Overviews and training the Gemini model
If you want to block Googlebot from feeding your content to AI Overviews, your only option is to block Googlebot entirely — which means disappearing from Google Search results. For most publishers who get 20-40% of their traffic from Google, that's not a realistic option.
Google does offer "Google-Extended" as a separate crawler you can block for AI training. But here's the catch: blocking Google-Extended does NOT prevent your content from appearing in AI Overviews. It only blocks training of the Gemini model.
Court documents revealed in 2025 showed that Google internally considered separating its crawlers to give publishers a choice, then rejected the idea. The UK's Competition and Markets Authority (CMA) is now forcing Google's hand, requiring them to let publishers opt out of AI features without losing search visibility. But this ruling only applies in the UK — not globally, and certainly not in Kenya.
Will AI Actually Send Traffic Back?
Cloudflare's Markdown for Agents includes links and citations. AI Overviews show links. ChatGPT and Perplexity cite sources. So surely AI will send traffic back to publishers, right?
The data suggests otherwise:
When AI Overviews appear, only 1% of users click the cited links, according to Pew Research
ChatGPT sent 1.2 billion referrals to publisher sites between September and November 2025 — sounds impressive until you realize that traffic from all AI platforms combined represents just 1% of total publisher traffic
Meanwhile, traffic lost to Google AI Overviews alone represents a 25% median decline for many publishers
The math doesn't work. AI agents extract far more value than they return.
The Content Training Question
One of the biggest concerns publishers have is whether their content is being used to train AI models without compensation.
Cloudflare's approach tries to address this with something called "Content Signals Policy" — a framework that allows publishers to specify how their content can be used.
When Markdown for Agents serves a page, it includes this header:
This signals that the content can be used for:
AI Training: Feeding the data into models to improve them
Search: Indexing for search results
AI Input: Real-time use in AI responses (like citations)
Cloudflare says this gives publishers control. But critics point out that these are just headers — not legally binding contracts. An AI company could simply ignore them.
For Kenyan publishers creating original research, investigations, or technical reviews, the question is stark: Should international AI companies be allowed to scrape your hard work, train multi-billion-dollar models on it, then compete with you for readers — all without paying you a cent?
Some publishers are fighting back with lawsuits. The New York Times is suing OpenAI and Microsoft. Ziff Davis filed similar lawsuits. But for smaller publishers in emerging markets like Kenya, legal action against Silicon Valley giants isn't realistic.
Will People Still Visit Websites?
This might be the existential question: In a world where AI answers every question instantly, why would anyone visit a website?
The data suggests they increasingly won't:
Zero-click searches are now the majority — 69% of searches end without clicking any website
Publishers expect their search traffic to decline by an additional 43% over the next three years
Some publishers are already shutting down, unable to sustain operations without search traffic
However, there are some bright spots:
1. Original investigations and on-the-ground reporting still require visiting the source
2. Branded searches remain resilient (people searching for your site specifically)
3. Service journalism (how-to guides, buying advice) may adapt by focusing on depth AI can't replicate
4. Community and commentary provide value beyond just information
The Wikimedia Foundation (which runs Wikipedia) noted a concerning trend: fewer visits to Wikipedia means fewer volunteer contributions and donations. If AI extracts all of Wikipedia's value while contributing nothing back, who will maintain Wikipedia?
This dynamic threatens the entire information ecosystem. Content creators need revenue to sustain creation. If AI consumes content without compensation, creation will eventually slow or stop.
What This Means for Kenyan Tech Publishers
So what should techinkenya.com and other Kenyan tech sites do?
Immediate Considerations:
1. Check Your Cloudflare Plan
If you're on a paid Cloudflare plan (Pro or higher), you can enable Markdown for Agents now. Go to your zone settings, find Quick Actions, and toggle the feature on. This ensures AI agents can efficiently process your content, potentially improving your visibility in AI-generated responses.
2. Audit Your robots.txt
Review which AI crawlers you're allowing or blocking. Consider:
GPTBot (OpenAI/ChatGPT)
Claudebot (Anthropic/Claude)
Googlebot (Google Search + AI)
Google-Extended (Gemini training)
-PerplexityBot
-Anthropic-AI
3. Monitor AI Referral Traffic
Start tracking which AI platforms (if any) are sending you traffic. Cloudflare Radar now shows content type insights for AI bot traffic. Understanding where your AI traffic comes from (or doesn't) helps inform strategy.
Long-Term Strategy:
1. Diversify Traffic Sources
Relying solely on Google Search is increasingly risky. Consider:
Social media (YouTube, X, Facebook)
Email newsletters (owned audience)
Direct traffic (building brand recognition)
Community forums (Reddit, Telegram groups)
2. Create AI-Hard Content
Focus on content that AI struggles to replicate:
Local, on-the-ground reporting about Kenyan tech scene
Original interviews with founders and executives
Product testing with real hands-on experience
Deep analysis that requires expertise and context
Community-building through comments and discussion
3. Consider AI Licensing
Some publishers are striking licensing deals with AI companies. OpenAI has deals with News Corp, The Atlantic, and others. These deals typically pay for access to content for training and citations.
For Kenyan publishers, collective bargaining might make sense — a coalition of East African tech publishers negotiating together would have more leverage than individual sites.
4. Build Direct Relationships
The future may favor publishers who build direct relationships with readers:
Newsletters you control
Memberships for exclusive content
Events and workshops
Consulting and services
5. Stay Informed
The AI landscape is evolving rapidly. What's true today might change tomorrow. UK regulators just forced Google to provide opt-out controls. US antitrust cases are ongoing. New AI platforms emerge constantly.
The Bigger Picture: A New Web Order
Cloudflare's Markdown for Agents isn't just a technical feature — it's a symptom of a fundamental shift in how the web works.
For 30 years, the web was built for humans. Search engines indexed it, ranked it, and sent traffic to it. Publishers monetized that traffic. This economic model, while imperfect, sustained a rich ecosystem of content creation.
AI is rewriting the rules:
Content is consumed by machines, not just humans
Value is extracted without proportional compensation
Traffic is retained by AI platforms rather than sent to sources
Creation incentives are misaligned with consumption
For Kenyan publishers specifically, there's an additional concern: Most AI training data is English-language content from Western sources. African perspectives, contexts, and priorities are underrepresented. If Kenyan sites can't sustain themselves, this representation gap only widens.
What Happens Next?
We're in uncharted territory. The web's economic foundations are shifting in real-time. No one knows exactly how this will shake out.
Some possible futures:
Scenario 1: New Compensation Models Emerge
AI companies start paying publishers for content, either through licensing deals or micropayments per citation. This could work, but requires regulatory pressure or collective publisher action.
Scenario 2: Publishers Adapt
Sites pivot to content and business models that work in an AI-first world — memberships, services, events, community. Traffic declines but isn't fatal.
Scenario 3: The Great Content Collapse
Without sustainable business models, many publishers shut down. The web becomes less rich and diverse. AI models, trained on increasingly stale data, become less valuable. Everyone loses.
Scenario 4: Platform Dominance
A few giant platforms (Google, OpenAI, Meta) consolidate control over information access. Publishers become either their licensed partners or irrelevant.
Scenario 5: Regional Fragmentation
Different jurisdictions impose different rules. The EU mandates compensation. The US allows free scraping. China builds a walled garden. Kenya and other developing markets are left to navigate between these powers.
Which future we get depends partly on publishers standing up for their rights — and partly on whether users still value direct access to sources over AI-mediated summaries.
Conclusion: Questions Worth Asking
If you run a blog, or any Kenyan tech publication, here are the questions worth grappling with:
1. Do we enable Markdown for Agents?If you're on a paid Cloudflare plan, probably yes — it can't hurt and might help AI cite you. But it's worth discussing with your team.
2. What's our AI crawler policy? Blanket blocking loses potential citations. Blanket allowing means free training data for billion-dollar companies. What's the right middle ground?
3. How do we measure success? If traffic declines are inevitable, what metrics matter? Engagement? Conversions? Revenue per visitor? Community size?
4. What's our unique value? What can techinkenya.com offer that an AI chatbot cannot? Double down on that.
5. Are we building owned channels?Email lists, memberships, apps — anything that doesn't depend on Google or AI platforms.
The AI revolution in web content isn't coming — it's here. Publishers who understand the dynamics and adapt strategically have the best chance of surviving and thriving.
Those who ignore it risk becoming footnotes in datasets that train the models that replace them.
Sources:
Cloudflare's Official Announcement
Cloudflare Developer Documentation
Technical Note for Developers:
If you're on a Pro/Business/Enterprise Cloudflare plan, here's how to check if this feature is right for you:
1. Log into Cloudflare Dashboard
2. Select your zone (techinkenya.com)
3. Look for "Quick Actions"
4. Find "Markdown for Agents" toggle
5. Enable it to start serving Markdown to AI agents
You can test it yourself:
If enabled, you'll get Markdown instead of HTML in the response.
The feature includes automatic Content Signals headers indicating your content preferences for AI training, search indexing, and AI input. You can customize these policies later as the feature matures.
No changes needed to your actual website code — Cloudflare handles conversion at the edge automatically.
Comments