You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CLI Web Scraper that scrapes URL, title and snippets #499
Is your feature request related to a problem? Please describe.
Currently, the web scraper only extracts the titles of blog posts using BeautifulSoup. This limits the amount of useful data we can retrieve and doesn’t provide flexibility for richer information or filtering. It also depends heavily on static HTML parsing, which can break easily if the website’s structure changes.
Describe the solution you'd like
I’d like to enhance the scraper by integrating a Custom Search Engine ID–based approach using the Google Search API. This method would allow us to dynamically search for blogs or articles by keyword, retrieve multiple metadata fields (like title, URL, snippet/description), and handle dynamic websites more reliably.
Describe alternatives you've considered
Expanding the current BeautifulSoup logic to extract more fields, but that would still be prone to site structure changes.
Using Selenium for dynamic content, but that’s slower as compared to using the Custom Search Engine ID.
Also any of these options would not be able to by-pass the capcha that may often occur during google search.
Additional context
This improvement would make the scraper more robust, scalable, and adaptable to different sources. It would also enable easier data analysis since we can collect richer and more structured information from multiple web pages.
I would like this issue to be assigned to me, under hacktoberfest label. Please assign this to me.
Activity
github-actions commentedon Oct 18, 2025
Thank you for raising this issue! Our team will review it soon.
For more queries or discussions or approval of requests, please join the Discord server for further discussion: Discord Link
barnavo05 commentedon Dec 24, 2025
Hi! I’d like to work on this issue. Could you please confirm it’s OK to proceed?