About the Iframely Preview Agent

If you see this agent fetching your URLs, it’s because a person shared that URL on an application built by one of our customers. Our customers use Iframely as a technical intermediary to generate URL previews or embed rich media content. We act on the customer’s behalf, not on our own. Iframely does not crawl or follow links. We take a single URL as input.

Iframely runs three separate agents, each scoped to what our customers use it for:

• Preview Agent (this one) — used for link previews and embeds triggered by a human-shared URL. Does not allow automated processing or AI training under this identity.
• Processing Agent — used for automated processing on a customer’s behalf, including AI-related use such as indexing, search, and AI-assisted retrieval.
• Training Agent — used for AI training and dataset use on a customer’s behalf, where a publisher has permitted it. By default, it accesses only declared metadata and links; anything beyond that requires the publisher’s explicit permission.

If you’re looking for AI-access controls, the Processing and Training docs are the ones you want.

What we do

Iframely is a software-as-a-service, built for social app developers and news media publishers. This agent responds to links shared socially by end-users with the information we can find about your URL — title, description, thumbnail image, and similar metadata.

That data is used by our customers to build engaging user experiences. For news and media customers, it powers content management systems that let editors curate and embed third-party media inside their own posts.

In other words, the Preview Agent is a distribution and promotion vehicle for your web content. It extends your reach to thousands of news sites, blogs, and social apps.

URL unfurling example

Here’s an example of how we present your URLs, and the information we look for:

User-Agent string

Iframely/1.3.1 (+https://iframely.com/docs/about)

Version and optional app name extension may and will change. The name extension, when present, identifies the specific customer who triggered the request, as they’ve entered it with us. This is the identity referred to as the “Iframely Preview Agent” elsewhere in our documentation and policies.

What we access

The Preview Agent reads only your page’s metadata — Open Graph tags, Twitter Cards, oEmbed data, Google’s structured data, and Iframely’s own rich media discovery protocol. It never reads your page body, and it never performs automated processing of any kind.

If you have your website optimized for social sharing already, you’re well presented on Iframely too. To check exactly what we see for a given URL, use our URL debugger.

Minimizing your traffic

We try to fetch as little of the page as possible to extract the metadata we need. In most cases, we leave the page as soon as we’ve read the <meta> portion — we don’t wait for or look into the full <body>.

If a page’s tags reference an image, video, or audio file, we’ll fetch that file too, to check its validity — using HEAD requests or HTTP Range headers to grab only the first few bytes, not the entire file.

If a customer uses our URL cards service to present a summary of your URL to their end-users, we’ll fetch the thumbnail image once and cache it on our CDN for an extended period, reducing repeat traffic to your site.

We don’t consult your robots.txt for these requests. The Preview Agent isn’t crawling your site — it never decides which URLs to fetch. It only ever fetches the one URL a person explicitly shared on a customer’s application; that’s where the request starts and ends. robots.txt governs crawl and automated-access permissions, which apply to the Processing and Training Agents instead, described in their own docs.

Refreshing the content

Provided your URL keeps being active on our network, we may re-fetch fresh URLs as often as once per hour on their first day, tapering off significantly as content ages — down to about once a month.

As per our GDPR guidelines and privacy policy, we purge all cached content from our network after three months of inactivity, though it’s often evicted much earlier.

Having a problem with the Preview Agent?

Although our parsers are battle-hardened from years in use, it’s entirely possible — the internet being what it is — that this agent is doing something it shouldn’t. We’re happy to add you to (or remove you from) our ignorelist, work with you to improve how we parse your content, discuss alternatives to how we access your site, or just answer questions. Contact us at support@iframely.com.

Allow the Preview Agent on your network

If you have bot protection in place, please add the Preview Agent to your allowlist. Your CDN provider may already recognize it and offer controls to enable it.

The Preview Agent supports both IP/reverse-DNS verification and Web Bot Authentication, and every request carries headers identifying the specific customer or application behind it — this works identically across all three Iframely agents. See Allowlisting Iframely for IP lists, reverse DNS, Web Bot Authentication, transitive trust, and the full request header reference.