Linkup bots

Linkup operates two bots. Each identifies itself in its HTTP User-Agent header and sends requests from its own published IP ranges.

• LinkupBot is Linkup’s web crawler. It discovers and retrieves publicly accessible webpages to build and maintain the Linkup web search index, which powers search and web grounding for AI applications.
• Linkup-User fetches individual pages on demand, when an application built on the Linkup API requests the content of a specific URL. It does not crawl the web or follow links on its own.

LinkupBot

Requests from LinkupBot contain the following user-agent token:

LinkupBot

An example complete user-agent is:

LinkupBot/1.0 (LinkupBot for web indexing; https://www.linkup.so/bot; bot@linkup.so)

LinkupBot requests originate from the CIDR ranges published at: https://www.linkup.so/linkupbot-ips.txt

Linkup-User

Requests from Linkup-User contain the following user-agent token:

Linkup-User

An example complete user-agent is:

Linkup-User/1.0 (+https://www.linkup.so/bot; bot@linkup.so)

Linkup-User requests originate from the CIDR ranges published at: https://www.linkup.so/linkup-user-ips.txt

Verifying Linkup bots

Because user-agent headers can be spoofed, verify traffic from a Linkup bot using both:

• A source IP contained in that bot’s published CIDR list.
• A user-agent containing that bot’s token.

The IP lists may change. Systems that automate allowlisting should retrieve them periodically.

Controlling Linkup bots with robots.txt

LinkupBot respects applicable robots.txt rules.

Place robots.txt at the root of each host you want to control, for example:

https://example.com/robots.txt

Rules apply only to the corresponding host, protocol, and port. Separate subdomains may need their own robots.txt.

Block the entire site

User-agent: LinkupBot
Disallow: /

Block specific content

User-agent: LinkupBot
Disallow: /private/
Disallow: /drafts/
Disallow: /internal-document.pdf

Block everything except selected content

User-agent: LinkupBot
Disallow: /
Allow: /public/

Explicitly allow LinkupBot

Crawling is allowed by default when no applicable rule prohibits it. To allow LinkupBot while blocking other crawlers:

User-agent: *
Disallow: /
User-agent: LinkupBot
Allow: /

Allowing LinkupBot through a firewall or WAF does not override robots.txt. Both the network policy and the applicable robots rules must permit access.

robots.txt is not an access-control mechanism. Protect confidential content with authentication or other server-side controls.

Allowlisting Linkup bots

To allow a Linkup bot through a firewall, CDN, or WAF:

1. Allow source addresses from its IP list: linkupbot-ips.txt for LinkupBot, linkup-user-ips.txt for Linkup-User.
2. Require the user-agent to contain its token: LinkupBot or Linkup-User.
3. Refresh the CIDR lists periodically.
4. Avoid matching the complete user-agent string, as its version may change.

For the strongest verification, require both the published source IP and user-agent conditions.

Contact

Questions or concerns about Linkup bots can be sent to bot@linkup.so.

Please include the requested URL, timestamp and timezone, source IP, complete user-agent, and relevant server-log entries.