2026-09-17, SERP API · Engineering · Fundamentals

Web scraping vs API: when to build a scraper and when to buy data

Scrape when the data is simple, stable, low-volume and clearly permitted; buy an API when the pages fight back, change often, or feed something customers depend on. Most of the decision is counting the right costs. A scraper’s first version is cheap. Keeping it correct for two years is not, and the bill arrives as on-call pages, silent data gaps and engineers who were hired to build something else.

Legal points below are general, sourced and not legal advice. Nothing here covers getting around bot protection; a plan that depends on it is a plan to drop.

Definitions

• Web scraping: fetching pages designed for human visitors and extracting data from their HTML or rendered DOM with your own code.
• First-party API: an interface the site owner publishes for programmatic access, with documented fields, quotas and terms.
• Data API from a vendor: a hosted service that collects data from public pages and returns it in a stable, documented schema. You integrate with the vendor’s contract, not with the target site’s markup.

The last category exists largely because first-party APIs are often missing, limited, or discontinued.

First-party options are shrinking for search data

If the data you need comes from search engines, the official routes have narrowed (both checked 2026-09-17):

• Google’s Custom Search JSON API “is closed to new customers,” and existing customers “have until January 1, 2027 to transition to an alternative solution.”
• Microsoft’s Bing Search APIs were retired on August 11, 2025, with customers pointed to Grounding with Bing Search in Azure AI Agents.

AI assistants such as ChatGPT, Gemini, Perplexity, Copilot and Grok offer model APIs, but a model API call is not the same as the answer a user sees in the consumer product, which adds live web search, citations, shopping cards and ads. If you need what users see, you need the rendered product surface. See provider APIs compared for that distinction.

The decision framework

Score each factor for your target. Three or more in the right column usually means buy.

Cost model for a scraper

The fetch itself is the smallest line. Fill in this model with your own estimates instead of trusting anyone’s generic numbers, including ours.

Two points get missed most often.

Silent failures are the expensive ones. A scraper that crashes gets fixed. A scraper that keeps returning plausible JSON after the page moved a feature into a new container quietly corrupts months of data. Budget for validation: known-answer test queries, field-level null-rate alerts, and periodic human spot checks.

Maintenance scales with the number of surfaces rather than requests. Five target sites with five parsers need five times the attention, whatever the volume. That is where vendor APIs usually win for multi-surface products.

For a fuller treatment of this trade-off for AI answer data, see build vs buy and AI visibility: build vs buy.

Terms of service, robots.txt and the law

Everything in this section is general information with sources. Laws differ by country, facts matter, and courts continue to decide cases. Ask a lawyer about your specific collection.

robots.txt is a convention, not a permission system

RFC 9309, the Robots Exclusion Protocol standard published in September 2022, says of robots.txt rules: “These rules are not a form of access authorization.” It is a way for site owners to ask crawlers to behave. Respecting it is good practice and some terms of service make it binding; ignoring it is a signal that you are collecting against the owner’s wishes.

Terms of service can prohibit automated access

Many sites address automated collection in their terms. Google’s Terms of Service (effective July 30, 2026, checked 2026-09-17), for example, list as abuse “using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities),” and separately prohibit “bypassing our systems or protective measures.” Read the terms of every site you plan to collect from, including whether you accepted them by creating an account.

Computer misuse laws

In the United States, the Supreme Court in Van Buren v. United States (2021) held that a person “exceeds authorized access” under the Computer Fraud and Abuse Act when they access a computer with authorization but obtain information “located in particular areas of the computer—such as files, folders, or databases—that are off-limits to him.” In footnote 8 the Court expressly did not decide whether that inquiry turns only on technological limitations or also on limits in contracts or policies. Other countries have their own computer misuse statutes. The safe engineering posture is simple: do not access anything behind authentication you were not given, and do not circumvent technical access controls.

Public does not mean free of privacy law

In an August 2023 joint statement on data scraping (checked 2026-09-17), data protection authorities from a group of countries including Canada, the United Kingdom and Australia wrote that “personal information that is publicly accessible is still subject to data protection and privacy laws in most jurisdictions.” If the pages you collect contain personal data, privacy law applies to your collection, storage and use regardless of whether you scrape or buy.

Copyright and database rights

Extracting facts differs from copying expression, and some jurisdictions protect databases themselves. How much of a page you store and republish matters. This is a question for counsel, not for a blog post.

Buying does not outsource your obligations

A vendor handles collection infrastructure, and you should ask vendors how they operate. But you still decide what data you request, how you store it, who sees it, and what you do with it. Privacy, retention and use obligations stay with you.

Questions to put to legal before you start

1. What exactly will we collect, from which sites, and does any of it identify people?
2. What do those sites’ terms say about automated access, and have we accepted them?
3. Will we access anything behind a login or other access control?
4. How long will we keep the data, and will we republish any of it?
5. If we use a vendor, what does its contract say about collection practices and our permitted use?

Hybrid patterns that work

Most mature teams do not pick one approach for everything.

What buying looks like in practice

For search engines and AI assistants, a vendor API replaces your collection and extraction code with one request. With this API, a Google search returns structured JSON:

curl -X POST https://api.answerline.dev/v1/monitor/google \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "query": "project management software", "country": "US", "device": "mobile" }'

The response contains result.organicResults[], result.ads[], result.peopleAlsoAsk[], result.relatedSearches[], result.localResults[], result.knowledgeGraph and, when requested with include.aioverview, result.aioverview with its text and cited sources. Locale and device are request fields (country, hl, location, device) instead of infrastructure you run.

An AI assistant answer, as a user sees it, is the same shape of call:

import os, requests

r = requests.post(
    "https://api.answerline.dev/v1/monitor/chatgpt",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    json={"prompt": "What are the best project management tools for small teams?", "country": "US"},
    timeout=360,
)
r.raise_for_status()
answer = r.json()["result"]
print(answer["text"][:300])
print([s["url"] for s in answer["sources"]])

For scheduled volume, submit async tasks in batches of up to 500 and receive results by webhook; see sync, async and webhooks. Credits per request are listed on the credits page and plans on pricing. The free tier includes 500 credits a month, enough to compare the API’s output against your scraper on your hardest queries before deciding.

What building responsibly looks like

If the framework says build, build like a good citizen:

import time, urllib.robotparser
from urllib.parse import urlsplit
import requests

UA = "ExampleCorpBot/1.0 (+https://example.com/bot; [email protected])"
_robots = {}

def allowed(url):
    root = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    if root not in _robots:
        rp = urllib.robotparser.RobotFileParser(root + "/robots.txt")
        rp.read()
        _robots[root] = rp
    return _robots[root].can_fetch(UA, url)

def polite_get(url, delay=5.0):
    if not allowed(url):
        return None  # respect the owner's instructions
    time.sleep(delay)
    return requests.get(url, headers={"User-Agent": UA}, timeout=30)

The principles behind it:

• Identify your crawler honestly with a contact URL.
• Honour robots.txt and any crawl-delay the owner asks for.
• Keep request rates low enough that the owner would not notice load.
• Stop when you are blocked or challenged. A block is an answer.
• Collect only the fields you need, and avoid personal data unless you have a lawful basis.
• Cache, so you never fetch the same page twice without reason.

A one-page decision checklist

1. Is there a first-party API that covers the data? Use it.
2. Do we have permission, or clearly acceptable terms, for automated collection? If not, stop or buy.
3. Does the target use active bot protection? If yes, buy; do not plan around evasion.
4. How many surfaces, locales and devices? More than a few favours buying.
5. What happens if the data is wrong for a week without anyone noticing? If that is serious, buy or budget heavily for validation.
6. Have legal and privacy reviewed what we collect and keep, whichever route we take?

Related

• Best SERP APIs compares search data vendors.
• Structured SERP data API explains what parsed results contain.
• Google dorking for defenders and content theft detection are examples of search data used through an API rather than a scraper.

Try the comparison yourself with the quickstart.

Keep reading