Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]

steps:
- name: Checkout repository
Expand All @@ -36,7 +36,7 @@ jobs:
API_KEY: ${{ secrets.API_KEY }}

- name: Run example tests
if: matrix.python-version == '3.14'
if: matrix.python-version == '3.15'
run: pytest tests/example_search_*_test.py
env:
API_KEY: ${{ secrets.API_KEY }}
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
steps:
- uses: actions/checkout@v6

Expand All @@ -43,7 +43,7 @@ jobs:
API_KEY: ${{ secrets.API_KEY }}

- name: Run engine and documentation examples
if: matrix.python-version == '3.14'
if: matrix.python-version == '3.15'
continue-on-error: ${{ inputs.allow_example_test_failures == true }}
run: pytest --ignore=tests/test_docs_publishing.py -k "example" --require-docs-key
env:
Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Bug reports and pull requests are welcome on GitHub. Run commands from the repos

## Development setup

Use Python 3.13 to work on the package and its documentation. The SDK CI also tests Python 3.8 through 3.14. Create a virtual environment and install the package with its test and documentation dependencies:
The SDK CI tests Python 3.8 through 3.15. Create a virtual environment and install the package with its test and documentation dependencies:

```sh
uv venv --python 3.13
Expand All @@ -31,10 +31,10 @@ The commands below use `python` from that environment.

### Checks that need no API key

These tests cover response parsing, image uploads with simulated responses, timeouts, exceptions, documentation syntax, and the example runner:
These tests cover response parsing, image uploads and extraction with simulated responses, timeouts, exceptions, documentation syntax, and the example runner:

```sh
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_timeout.py tests/test_exceptions.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -q
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_extract.py tests/test_timeout.py tests/test_exceptions.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -q
```

The file list limits this command to tests that make no external requests. Applying `-k 'not live'` to the entire suite is not enough to run offline: some SDK integration tests do not have `live` in their names.
Expand All @@ -61,7 +61,7 @@ This includes live account, location, search, and pagination checks. To run ever
python -m pytest -q
```

Both commands make real SerpApi requests. The full suite can use more searches than a test run limited to the files you changed. PR CI also runs the standalone engine examples on Python 3.14. Documentation tests run in the package and documentation release workflows on release tags, and in manual documentation runs on `master`.
Both commands make real SerpApi requests. The full suite can use more searches than a test run limited to the files you changed. PR CI also runs the standalone engine examples on Python 3.15. Documentation tests run in the package and documentation release workflows on release tags, and in manual documentation runs on `master`.

### Testing documentation examples

Expand Down
69 changes: 64 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,50 @@ Environment variables are a secure, safe, and easy way to manage secrets.
Set `export SERPAPI_KEY=<secret_serpapi_key>` in your shell.
Python accesses these variables from `os.environ["SERPAPI_KEY"]`.

### Extract content from a URL

The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. It uses `/search` with `engine=extract` automatically.

```python
import os
import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
results = client.extract(url="https://example.com/", include_markdown=True)
print(results["page_information"]["title"])
print(results["text_content"]["markdown"])

markdown = client.extract(url="https://example.com/", output="md")
html = client.extract(url="https://example.com/", output="html")
```

JSON responses return `SerpResults`. HTML and Markdown responses return strings. `include_markdown=True` adds the page's Markdown to `text_content["markdown"]` in a JSON response. The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.

To return only the page's Markdown in JSON, combine `include_markdown=True` with `json_restrictor="text_content.markdown"`. This omits all other response sections:

```python
results = client.extract(
url="https://example.com/",
include_markdown=True,
json_restrictor="text_content.markdown",
)
print(results["text_content"]["markdown"])
```

Use `mode="map"` to discover URLs or `mode="crawl"` to extract multiple pages:

```python
urls = client.extract(url="https://example.com/", mode="map")
pages = client.extract(
url="https://example.com/", mode="crawl", max_pages=2, max_depth=1
)
```

Crawl mode can consume one search credit per successfully extracted page, including the starting page. Set `max_pages` to limit the number of pages.


See the [Extract API guide](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md) for response examples and more options.

### Error handling

Unsuccessful requests raise `serpapi.HTTPError` or `serpapi.TimeoutError` exceptions. The returned status code will reflect the sort of error that occurred, please refer to [Status and Error Codes Documentation](https://serpapi.com/api-status-and-error-codes) for more details.
Expand Down Expand Up @@ -267,15 +311,30 @@ results = client.search({
- API Documentation: [serpapi.com/google-autocomplete-api](https://serpapi.com/google-autocomplete-api)

### Search Google Immersive Product

Get a current `page_token` from Google Shopping before requesting product details:

```python
import os
import serpapi

client = serpapi.Client(api_key=os.getenv("API_KEY"))
results = client.search({
'engine': 'google_immersive_product',
'page_token': 'eyJlaSI6Im5ZVmxaOXVVTDY2X3A4NFBqTnZELUFjIiwicHJvZHVjdGlkIjoiIiwiY2F0YWxvZ2lkIjoiNTE1NDU2NTc1NTc5MzcxMDY3NSIsImhlYWRsaW5lT2ZmZXJEb2NpZCI6IjI1MDkyMjcwMDUzMjk2NzQwODMiLCJpbWFnZURvY2lkIjoiMTYzOTg5MjU0MDcwMDU4MDA1NTQiLCJyZHMiOiJQQ18zNDg4MDE0MTg3ODgxNzc5NjU0fFBST0RfUENfMzQ4ODAxNDE4Nzg4MTc3OTY1NCIsInF1ZXJ5IjoibGcrdHYiLCJncGNpZCI6IjM0ODgwMTQxODc4ODE3Nzk2NTQiLCJtaWQiOiI1NzY0NjI3ODM3Nzc5MTUzMTMiLCJwdnQiOiJoZyIsInV1bGUiOm51bGx9=',
})
shopping = client.search(
engine="google_shopping",
q="lg tv",
gl="us",
hl="en",
no_cache=True,
)
page_token = next(
product["immersive_product_page_token"]
for product in shopping["shopping_results"]
if product.get("immersive_product_page_token")
)
results = client.search(
engine="google_immersive_product",
page_token=page_token,
)
```
- API Documentation: [serpapi.com/google-immersive-product-api](https://serpapi.com/google-immersive-product-api)

Expand All @@ -287,7 +346,7 @@ import serpapi
client = serpapi.Client(api_key=os.getenv("API_KEY"))
results = client.search({
'engine': 'google_reverse_image',
'image_url': 'https://i.imgur.com/5bGzZi7.jpg',
'image_url': 'https://i.imgur.com/HBrB8p0.png',
'max_results': '1',
})
```
Expand Down
2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ Use the same search parameter names as the [SerpApi API documentation](https://s
- [Getting Started](user_guide/getting-started.md) covers installation and your first search. [Client Usage](user_guide/client-usage.md) explains how to make requests and read responses.
- [AI Agents](ai-agents.md) covers search tools, Markdown results, and API references for agents.
- [Output Formats](user_guide/output-formats.md) explains when to use JSON, Markdown, or HTML.
- [Extract API](user_guide/extract.md) covers page content, URL discovery, and crawling, including how to return only a page's Markdown in JSON.
- [Migration Guide](user_guide/migrating-from-google-search-results.md) shows how to replace `google-search-results` with this package.
- [Parameters and Engines](user_guide/parameters-and-engines.md) explains search parameters and how to test them in the [SerpApi Playground](https://serpapi.com/playground).
- For scripts that collect results, see [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).
Expand All @@ -113,6 +114,7 @@ user_guide/client-usage
User Guide Overview <user_guide/index>
user_guide/parameters-and-engines
user_guide/output-formats
user_guide/extract
user_guide/pagination
user_guide/errors-and-timeouts
user_guide/account-and-locations
Expand Down
125 changes: 125 additions & 0 deletions docs/user_guide/extract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
---
title: "Extract API"
description: "Extract page content, discover URLs, and crawl websites with the Python client."
---

# Extract API

Use `client.extract()` to read content from a public URL. It supports single-page scraping, URL discovery, and crawling. Pass a parameter dictionary or keyword arguments, as you would with `client.search()`. The client sets `engine="extract"` automatically.

See the [Extract API documentation](https://serpapi.com/extract-api) for all parameters and response fields.

## Get Only the Page Content as JSON

Set `include_markdown=True` and `json_restrictor="text_content.markdown"` to get the page's Markdown content in a JSON response. This removes metadata and all other response sections.

After [setting your API key](getting-started.md#installation), run:

```python
import os
import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
results = client.extract(
url="https://example.com/",
output="json",
include_markdown=True,
json_restrictor="text_content.markdown",
)

markdown = results["text_content"]["markdown"]
print(markdown)
```

The response has this structure, with the extracted page content in `markdown`:

```json
{
"text_content": {
"markdown": "Page content in Markdown..."
}
}
```

The SDK returns a dictionary-like `SerpResults` object. Read `results["text_content"]["markdown"]` to get the plain Markdown string, as shown above. You can pass that string to an AI agent or save it to a file.

See [JSON Restrictor](json-restrictor.md) for other ways to select response fields.

## Get the Full Extract Response

Leave out `json_restrictor` to receive the available extracted sections, such as page information, headings, links, and text content:

```python
results = client.extract(
url="https://example.com/",
include_markdown=True,
)

print(results["page_information"]["title"])
print(results["text_content"]["markdown"])
```

JSON is the default output format. `include_markdown=True` adds Markdown under `text_content`; the rest of the response remains JSON. The sections returned depend on the target page.

The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.

## Markdown and HTML Responses

Use `output="md"` or `output="html"` to receive a Python string:

```python
markdown_response = client.extract(url="https://example.com/", output="md")
html = client.extract(url="https://example.com/", output="html")
```

For a scraped page, `output="md"` includes search metadata before the page content. Use the [content-only JSON example](#get-only-the-page-content-as-json) when you want the page's Markdown without those extra sections. `output="html"` returns the retrieved HTML.

## Discover URLs

Set `mode="map"` to discover URLs on a website:

```python
results = client.extract(
url="https://www.python.org/",
mode="map",
include_sitemap=False,
include_subdomains=False,
)

for entry in results["urls"]:
print(entry["url"])
```

Map mode includes sitemap URLs by default. Set `include_sitemap=False` to discover URLs from the page's links only. Set `include_subdomains=True` to include subdomains.

## Crawl Multiple Pages

Set `mode="crawl"` to follow links and extract pages:

```python
results = client.extract(
url="https://www.python.org/",
mode="crawl",
max_pages=2,
max_depth=1,
timeout=180,
)

for page in results["pages"]:
print(page["url"])
```

The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish. For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).

## Timeouts and Content Filters

The `timeout=` keyword sets the SDK's HTTP timeout in seconds. Set it above the API fetch timeout, and allow time for all pages when crawling. Put the Extract API's fetch timeout in the parameter dictionary to send it in milliseconds:

```python
results = client.extract(
{"url": "https://example.com/", "timeout": 30000},
timeout=60,
)
```

Use `include_tags` and `exclude_tags` to keep or remove content with CSS selectors, such as `include_tags="article, main"` or `exclude_tags="nav, footer"`. See the [Extract API parameters](https://serpapi.com/extract-api) for details and [Errors and Timeouts](errors-and-timeouts.md) for exception handling.
1 change: 1 addition & 0 deletions docs/user_guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Start with [Getting Started](getting-started.md) to install the package and run

- [Parameters and Engines](parameters-and-engines.md)
- [Output Formats](output-formats.md)
- [Extract API](extract.md)
- [Pagination](pagination.md)
- [Errors and Timeouts](errors-and-timeouts.md)
- [Account and Locations](account-and-locations.md)
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ classifiers = [
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.15",
"Programming Language :: Python :: Implementation :: CPython",
"Natural Language :: English",
"Topic :: Utilities",
Expand Down
24 changes: 24 additions & 0 deletions serpapi/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,29 @@ def search(self, params: dict = None, **kwargs):

return SerpResults.from_http_response(r, client=self)

def extract(self, params: dict = None, **kwargs):
"""Extract content from a public URL using the Extract API.

Accepts a parameter dictionary or keyword arguments, like :meth:`search`.
Sets ``engine=extract`` and returns a :class:`SerpResults <serpapi.SerpResults>` object for JSON, or text for HTML and Markdown.

.. code-block:: python

results = client.extract(url="https://example.com/")
markdown = client.extract(url="https://example.com/", output="md")
results = client.extract({"url": "https://example.com/", "timeout": 30000}, timeout=60)

:param params: Optional mapping of Extract API parameters, including the required ``url``. Put ``timeout`` here to set the API fetch timeout in milliseconds.
:param kwargs: Additional API parameters or HTTP request options, as in :meth:`search`. The ``timeout`` keyword controls the HTTP client timeout in seconds.

``mode`` accepts ``scrape`` (default), ``map``, or ``crawl``. ``output`` accepts ``json`` (default), ``html``, or ``md``.

Crawl mode can consume one search credit per successfully extracted page. Use ``max_pages`` to limit the number of pages.
"""
params = dict(params) if params is not None else {}
kwargs["engine"] = "extract"
return self.search(params, **kwargs)

def search_archive(self, params: dict = None, **kwargs):
"""Get a result from the SerpApi Search Archive API.

Expand Down Expand Up @@ -190,6 +213,7 @@ def account(self, params: dict = None, **kwargs):
# An un-authenticated client instance.
_client = Client()
search = _client.search
extract = _client.extract
search_archive = _client.search_archive
upload_image = _client.upload_image
locations = _client.locations
Expand Down
Loading
Loading