feat: add llms.txt discovery as default agent behavior

Add automatic llms.txt awareness so agents check for /llms.txt or
/.well-known/llms.txt when exploring new domains.

Changes:
- System prompt: new 'llms.txt Discovery' section (full mode only,
  when web_fetch is available) instructing agents to check for llms.txt
  files when visiting new domains
- web_fetch tool: updated description to mention llms.txt discovery

llms.txt is an emerging standard (like robots.txt for AI) that helps
site owners describe how AI agents should interact with their content.
Making this a default behavior helps the ecosystem adopt agent-native
web experiences.

Ref: https://llmstxt.org
This commit is contained in:
Ocean Vael
2026-02-16 15:20:42 +00:00
committed by Peter Steinberger
parent 4df970d711
commit e368c36503
2 changed files with 19 additions and 1 deletions

View File

@@ -767,7 +767,7 @@ export function createWebFetchTool(options?: {
label: "Web Fetch",
name: "web_fetch",
description:
"Fetch and extract readable content from a URL (HTML → markdown/text). Use for lightweight page access without browser automation.",
"Fetch and extract readable content from a URL (HTML → markdown/text). Use for lightweight page access without browser automation. When exploring a new domain, also check for /llms.txt or /.well-known/llms.txt — these files describe how AI agents should interact with the site.",
parameters: WebFetchSchema,
execute: async (_toolCallId, args) => {
const params = args as Record<string, unknown>;