Back to Prompts
Coding Assistant

Smart Scraping with Fallbacks

For scraping apps that need reliability From the Smart Abstraction AI Coding Building Block.

Prompt
Create a smart scraping function that tries multiple methods with automatic fallback.

Language: [Python, JavaScript, etc.]

Fallback chain:
1. Try scraping with [BeautifulSoup/Cheerio/Puppeteer]: free, fast
2. If blocked or failed, try [ScraperAPI/Apify/Browserless]: paid but reliable
3. If all scraping fails, return cached data from previous successful scrape

Requirements:
1. The main function should just take a URL and return the data
2. All the fallback logic should be hidden inside the function
3. Log which method worked (helps with debugging and cost tracking)
4. Handle common failures: timeouts, 403 forbidden, rate limits, CAPTCHAs
5. Cache successful responses so we have a fallback for next time

This is for scraping [describe what you're scraping: product prices, articles, etc.]

Show me the pattern so I can adapt it. Explain why each fallback makes sense.

I'm learning, so keep explanations simple.
1 views 0 copies