Blog · Topic
Engineering
Building on the API: idempotency, retries, webhook delivery, batching, rate limits and storing results so a pipeline survives its own failure modes.
Articles
API errors and retries: a policy that never double-charges
Which errors to retry, which to fix, and which are final: 429s, 5xx, timeouts and batch item errors, with retry wrappers in Python and TypeScript.
Idempotency keys explained: safe retries for async tasks
How idempotencyKey prevents duplicate tasks and double charges: key design, 409 handling, crash-safe batch resubmission, and what keys do not dedupe.
Verify webhook signatures: HMAC-SHA256 in Python, Node, Go
Verify a webhook's HMAC-SHA256 signature on the raw body, check the timestamp, handle secret rotation. Tested code for Flask, FastAPI, Express and Go.
Webhook receiver design: retries, duplicates and ordering
Build a webhook receiver that loses nothing: verify, write to an inbox, ack fast, dedupe by task.id, handle retries and out-of-order results, reconcile.
AI crawlers: GPTBot, PerplexityBot, ClaudeBot and more
Every major AI crawler's user agent and stated purpose from vendor docs, robots.txt recipes that separate search from training, and how to measure it.
Enterprise rank tracking: millions of keywords, reliably
Rank tracking at millions of keywords: tiered schedules, shared collection, batching, throughput math, queue feeding, storage and data quality checks.
Google dorking for defenders: find what your site exposes
Defensive search operator queries to find exposed files, staging hosts and error pages on domains you own, run them on a schedule and remove what leaks.
Online reputation monitoring for search, news and AI
Monitor what Google, its AI Overview, news and AI assistants show about your brand: query set, schedule, snapshot diffs, alert rules and human review.
OSINT tools for defenders: map your own attack surface
theHarvester, SpiderFoot, Maltego, Shodan, crt.sh, urlscan.io, HIBP and dnstwist for defenders, search results as a source, and the ethical limits.
Perpetual KYC and adverse media: event-driven screening
How perpetual KYC uses adverse media as a trigger: what public guidance says, delta screening, risk-tier cadence, re-scoring rules and an audit trail.
Programmatic AI visibility tracking: system design
System design for AI visibility tracking at thousands of prompts: prompt catalog, planner, run accounting, raw storage, observation tables and metrics.
SERP API reliability: retries and idempotency
Engineer SERP collection that finishes: a completeness ledger, retry budgets, idempotent submission, billing on failure, data-quality checks and alerts.
SERP and AI answer data in BigQuery: schema and queries
Load SERP and AI answer results into BigQuery: a partitioned schema with nested results and JSON, free batch loads, streaming costs and example SQL.
Third-party risk monitoring with news and search data
Build continuous vendor monitoring from Google News and search results: signal taxonomy, query templates, tiered cadence, triage rules and cost per vendor.
Typosquatting detection: find lookalike domains
Detect lookalike domains impersonating your brand in Google results and ads, generate permutation candidates, check certificate logs, triage and report.
Web scraping vs API: when to build and when to buy
A decision framework for scraping versus buying an API: total cost, maintenance, data shape, scale, and the terms-of-service and legal questions to raise.
Sync calls, async tasks or webhooks: which to use
When to wait for an answer, when to queue a task and poll, and when to let results come to you by webhook: retries and failure modes.