Repository navigation
feat(mrscraper): add MrScraper integration for page scraping, Google search, and account usage - #9277
riandradiva18 wants to merge 2 commits into
Conversation
- send HTML-only and screenshot-only scrapes as GET with the target URL in the query; the fetch endpoint replays the request method, so a POST fails on sites that reject it - read raw text/html responses and take token usage from response headers - send screenshot=full and detect PNG vs JPEG from the returned image - keep the API key out of the query string - report the requested search page; the SERP API returns current_page 1 for every page - show retry and token-cap fields only for Markdown requests - use defaultValue for switch defaults - regenerate the integration catalog, tool metadata, block names, and docs manifest
|
@waleedlatif1 this is ready for a look when you have time. I tested every operation against the live MrScraper API with a real key. Each call went through the tool's own request builder (
The API key never appeared in a request URL in any of these calls. Sample outputs (long strings trimmed)Get Account (name and email removed) {
"subscriptionStatus": "active",
"enterprise": false,
"tokenUsage": 5242300,
"tokenLimit": 10000000,
"tokenRemaining": 4757700,
"usagePercent": 52.42,
"rateLimit": 0,
"rateTtl": 0,
"autoRenew": false,
"endsAt": "2027-10-08T17:00:00.000Z",
"user": { "name": "…", "email": "…", "verified": true }
}Google Search, {
"searchResults": [
{
"title": "15 Cafe di Jakarta Barat yang Wajib Masuk Wishlist Kamu!",
"link": "https://blog.cove.id/cafe-di-jakarta-barat/",
"description": "28 Feb 2024 — 2. Satu Meja · 3. Upstairs Coffee Signature · …",
"source": "Cove Blog",
"displayLink": "https://blog.cove.id › cafe-di-jakart...",
"rank": 1,
"globalRank": 1
}
],
"html": null,
"general": {
"query": "best coffee jakarta",
"resultsCount": 10,
"searchTime": 10.7,
"language": "en",
"location": "id",
"pageTitle": "best coffee jakarta - Google Search",
"timestamp": "2026-10-21T09:33:19.013Z"
},
"pagination": {
"currentPage": 2,
"nextPage": 3,
"nextPageStart": 20,
"nextPageLink": "https://www.google.com/search?q=best+coffee+jakarta&…",
"pages": [{ "page": 1, "start": 0, "link": "…" }, { "page": 3, "start": 20, "link": "…" }]
},
"tokenUsage": 1
}Scrape Page, block defaults (HTML only) {
"html": "<!DOCTYPE html>\n<!--[if lt IE 7]> <html lang=\"en-us\" class=\"no-js lt-ie9 …",
"markdown": null,
"tokenUsage": 1,
"runtime": null,
"retryCount": null,
"targetStatusCode": null,
"error": null
}Scrape Page, HTML + Markdown {
"html": "<!DOCTYPE html><html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title> …",
"markdown": "# [Quotes to Scrape](/)\n\n[Login](/login)\n\n“The world as we have created it is a process of our thinking. …",
"tokenUsage": 3,
"runtime": 5.95,
"retryCount": 0,
"targetStatusCode": 200,
"error": null
}Scrape Page, Markdown + screenshot {
"html": null,
"markdown": "[Books to Scrape](index.html) We love being scraped! …",
"screenshot": { "name": "screenshot.jpg", "mimeType": "image/jpeg", "size": 272776 },
"tokenUsage": 6,
"runtime": 11.33,
"retryCount": 0,
"targetStatusCode": 200,
"error": null
}Locally: |
|
@riandradiva18 is attempting to deploy a commit to the Sim Team on Vercel. A member of the Team first needs to authorize it. |
 [Medium impact] The PR adds a new, bounded web-scraping and search workflow integration. The PR appears safe to merge, with a non-blocking fix needed for searches containing ordinary URLs. Findings
SummaryAdds MrScraper tools for page retrieval, Google search, and account usage.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (/^https?:\/\//i.test(value)) { | ||
| const url = new URL(value) | ||
| query = url.searchParams.get('q')?.trim() || '' | ||
| if (!query) throw new Error('A Google search URL must contain a q parameter.') |
There was a problem hiding this comment.
This branch treats every HTTP URL as a Google search URL. Searching for https://example.com throws because it has no q parameter. A non-Google URL containing q instead searches for that parameter and drops the supplied URL. Only extract parameters from Google search URLs; pass other URLs through as query text.
This branch has not been deployed
1 participant
Summary
Adds a MrScraper block with three operations:
q,gl,hl,startare picked up). Returns organic results, search metadata, and pagination, or the raw results page HTML.Also includes the icon, docs page, registry entries, and regenerated tool metadata, integration catalog, deployment config, and block names.
Some MrScraper API behavior that shaped the scrape tool:
GETwith the target URL in the query. That endpoint replays the incoming method against the target site, so aPOSTfails on sites that reject it (books.toscrape.com answers 405). Markdown requests go to the browser scraper as aPOSTwith a JSON body.text/htmland reportstoken_usagein a response header; the browser scraper returns JSON. The transform handles both and reads metrics from headers as a fallback.maxRetriesandtokenCaponly apply to the browser scraper, so those fields only show when Markdown is on.current_page: 1for every page, so the tool reports the page that was requested instead.Type of Change
Testing
Ran every operation against the live MrScraper API with a real key. Each call went through the tool's own request builder (via
prepareToolRequest) andtransformResponse, with block defaults mapped through the block'stools.config.params. All 11 live calls passed, and calls with an invalid key fail with a 401 as expected. The full results and sample responses are in the comment below.Locally:
bun run lint,bun run type-check,bun run check:audits, andcheck-block-registry.tsagainst staging.I didn't add unit tests. Following the repo's testing guidance, I tested against the real API instead.
Checklist