Skip to content

CLI Web Scraper that scrapes URL, title and snippets #499

Description

@Sarthacker

Is your feature request related to a problem? Please describe.
Currently, the web scraper only extracts the titles of blog posts using BeautifulSoup. This limits the amount of useful data we can retrieve and doesn’t provide flexibility for richer information or filtering. It also depends heavily on static HTML parsing, which can break easily if the website’s structure changes.

Describe the solution you'd like
I’d like to enhance the scraper by integrating a Custom Search Engine ID–based approach using the Google Search API. This method would allow us to dynamically search for blogs or articles by keyword, retrieve multiple metadata fields (like title, URL, snippet/description), and handle dynamic websites more reliably.

Describe alternatives you've considered

  • Expanding the current BeautifulSoup logic to extract more fields, but that would still be prone to site structure changes.
  • Using Selenium for dynamic content, but that’s slower as compared to using the Custom Search Engine ID.
  • Also any of these options would not be able to by-pass the capcha that may often occur during google search.

Additional context
This improvement would make the scraper more robust, scalable, and adaptable to different sources. It would also enable easier data analysis since we can collect richer and more structured information from multiple web pages.

I would like this issue to be assigned to me, under hacktoberfest label. Please assign this to me.

Activity

  1. github-actions commented on Oct 18, 2025

    @github-actions

    Thank you for raising this issue! Our team will review it soon.

    For more queries or discussions or approval of requests, please join the Discord server for further discussion: Discord Link

  2. barnavo05 commented on Dec 24, 2025

    @barnavo05

    Hi! I’d like to work on this issue. Could you please confirm it’s OK to proceed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions