Introducing the Iframely Data API for automated processing
Iframely has been connecting app developers with rich media providers and web publishers since 2012. Our core use case has been rich media embeds and URL previews for end users, when URLs are shared by people.
Automated processing of web content — including AI — is on the rise. Not always in a positive way. While our terms have required URLs to originate from a human user, we've seen an increasing number of support requests from customers who want to automate URL processing.
Today we're launching the Iframely Data API, our new solution for responsible automated processing. It builds on years of infrastructure and experience working with developers and publishers.
Applications can now initiate URL processing themselves under the Iframely access framework, which is based on processing modes and content types.
The new Data API is separate from our existing endpoints, which will now be known as Preview APIs. It comes with its own terms and its own web robot identities, so publishers can treat automated processing separately.
Most importantly, Iframely keeps its publisher-first principle: respecting publisher rights and giving publishers controls to manage how we handle their content, including through our new Publisher Console. At the other end, applications receive access decision data so developers can understand what they received and why.
Take control of your domains
First things first: you are most likely a publisher yourself.
If you're an Iframely developer customer, you may not think of yourself as a publisher. But you or your company also have a website that can be accessed and processed by Iframely on behalf of other customers.
So now is the time to claim your domains in Iframely and see how we handle your content.
Simple email verification gives you read-only access. DNS verification gives you admin rights.
If you already have an account with us, go to Manage Domains and follow the prompts. Otherwise, create a free publisher account first at iframely.com/signup. You, or your colleagues, can also email us at support@iframely.com and we’ll be happy to update the permissions for you.
Please also see our documentation for what is available for different processing modes, and adjust your publisher settings and signals where needed. See our new publisher policy for non-technical description.
Content Access for Automated Processing
New Data API for developers
Iframely's new Data API gives applications structured access to the content behind a URL. It builds on the metadata and media links already available through our Preview APIs, while adding new content types for automated processing.
The Data API is not a programmable scraper. It works with specific content pieces that we call content types. In addition to preview data, the Data API can provide structured entities, excerpts, and full text when available and permitted.
What an application receives depends on its declared processing mode and the publisher policies applicable to that mode:
assist— help a person work with a URLdiscover— categorize content for recommendation to other peoplesearch— index and retrieve content for searchai-input— provide content as input to an AI system, except for trainingai-train— use content for AI training
See Iframely default processing policy for the content available for each mode by default, before publisher-specific signals and settings are applied. By default, ai-train is limited to the equivalent of preview data.
Along with the content, Iframely provides an Access object that describes how we arrived at the access decision.
See the developer documentation:
New web robot identities
In addition to the existing Iframely web agent, now known as the Iframely Preview Agent, we introduce two more web identities:
- Iframely Processing Agent — for automated processing using
assist,discover,search, andai-inputmodes - Iframely Training Agent — for
ai-trainprocessing
The agents carry a Web Bot Auth signature and can also pass information about the requesting customer application in HTTP headers. This gives publishers more context when evaluating automated requests and helps carry the application's declared use through the processing chain.
If you use a firewall, bot management system, or other access controls, you can allowlist the new agents.
See Allowlisting Iframely Bots for more information.
Updating our terms
Automated processing requires a change to our terms of use.
Our existing terms cover URL processing initiated by a human user. The new Data API allows applications to initiate URL processing themselves, so automated processing is now covered by separate Automated Processing Terms.
As before, Iframely is a technical intermediary and does not grant customers rights to third-party content.
The updated terms take effect with the Data API launch.
See Automated Processing Terms.
Making URLs travel
URLs remain what people, publishers, and applications have in common.
Iframely has been helping this exchange work reliably for more than a decade, handling billions of requests for thousands of customers across the web.
The Data API extends that infrastructure to a new generation of applications — while keeping publishers in the loop.
If you publish content and already use Iframely, claim your domains and review your content policies.
If you're building an application that needs to process web content automatically, try the Data API.
And, as always, we're interested in your questions and feedback.