About Iframely robot

If you see Iframely parsers fetching your URLs, it’s because an actual user has shared that URL on one of the apps that use Iframely as their back-end. We are not a crawler: we don’t follow links and only act on behalf of a human.

Iframely is a software-as-a-service for social app developers and news media publishers. Our robots respond to links shared socially by end-users with the information we can find about your URL: title, description, thumbnail image, etc.

This data is then used by our customers to build engaging user experiences. For news & media customers, we power up their content management systems with the ability to curate 3rd party media and embed it inside the news posts.

In other words, Iframely is a distribution and promotion vehicle for your web content. We extend your reach to thousands of news sites, blogs and social apps.

Iframely’s robots fetch URLs with the following user agent (version and optional name extension may and will change):

Iframely/1.3.1 (+https://iframely.com/docs/about)

You are probably here because you found us in your access logs and you have questions or are curious. We are here to help and give more information.

URL unfurling example

Here’s an example of how we present your URLs and info that we look for:

URL unfurled

Or in a compact form

What our parsers see

Our parsers understand Facebook’s Open Graph, Twitter Cards, oEmbed, Google’s structured data and Iframely’s own rich media discovery protocol.

If you have your website optimized for social sharing already — you’re well presented on Iframely too.

To check how Iframely and other apps see your website, visit our debug tool:

Iframely URL debugger

Minimizing your traffic

Our parsers try and fetch as little of the page as we can to extract meta tags about the content. In most cases, they will leave the page as soon as they get the <meta> portion of it, without waiting or looking into full <body>.

If a page’s tags refer to an image, video, or audio file, we will fetch that file as well to check its validity. We use head requests or HTTP Range headers to only grab first few bytes of such content, not downloading entire files during validations.

If customer uses our URL cards service to present your URLs’ summary to their end-users, we will fetch the thumbnail images and cache it inside our CDN for extended period of time, reducing traffic to your site.

Refreshing the content

Provided that your URL keeps being active on our network, our parsers may hit any fresh URLs as often as once per hour on the first day, significantly reducing traffic as the content becomes older, to about a once per month.

As per our GDPR guidelines and privacy policy, we purge all content’s cache off our networks after three months of inactivity.

Having a problem with Iframely?

Although our parsers are battle-hardened over the years in use, it is entirely possible, as the Internet is constantly evolving, that the robots are doing something they should not. We are more than happy to add you to or remove from our ignorelist, work with you to improve parsing of your content, discuss alternatives to how we are accessing your site, or just answer any questions you have. Please contact us at support at iframely.com.

Allow Iframely robot on your network

If you have bot-protection in place, please add Iframely to your allowlist. Your CDN provider might already recognize us and provide controls to enable Iframely. If not, or to allow Iframely on your own servers, we suggest two options:

Allow all of our public IP addresses as listed here iframely.com/ips-v4. Please refresh the list periodically to keep it up-to-date.
Alternatively, we can also re-route the traffic to your website via a subset of our IPs that resolve reverse DNS lookups to a subdomain of iframely.com. This would require manual adjustment on our side as well. Please contact support at iframely.com.

For both options, you may debug your website on Iframely using our URL Debugger tool.

Again, our user-agent string is Iframely/1.3.1 (+https://iframely.com/docs/about), perhaps followed by the app name extension. The version number may and will change.