mirror of
https://github.com/openclaw/openclaw.git
synced 2026-05-08 04:41:25 +00:00
feat: add llms.txt discovery as default agent behavior
Add automatic llms.txt awareness so agents check for /llms.txt or /.well-known/llms.txt when exploring new domains. Changes: - System prompt: new 'llms.txt Discovery' section (full mode only, when web_fetch is available) instructing agents to check for llms.txt files when visiting new domains - web_fetch tool: updated description to mention llms.txt discovery llms.txt is an emerging standard (like robots.txt for AI) that helps site owners describe how AI agents should interact with their content. Making this a default behavior helps the ecosystem adopt agent-native web experiences. Ref: https://llmstxt.org
This commit is contained in:
committed by
Peter Steinberger
parent
4df970d711
commit
e368c36503
@@ -767,7 +767,7 @@ export function createWebFetchTool(options?: {
|
||||
label: "Web Fetch",
|
||||
name: "web_fetch",
|
||||
description:
|
||||
"Fetch and extract readable content from a URL (HTML → markdown/text). Use for lightweight page access without browser automation.",
|
||||
"Fetch and extract readable content from a URL (HTML → markdown/text). Use for lightweight page access without browser automation. When exploring a new domain, also check for /llms.txt or /.well-known/llms.txt — these files describe how AI agents should interact with the site.",
|
||||
parameters: WebFetchSchema,
|
||||
execute: async (_toolCallId, args) => {
|
||||
const params = args as Record<string, unknown>;
|
||||
|
||||
Reference in New Issue
Block a user