Your Majesty - @harini @Shams @Gaurav Chadha ...
@harini @Shams @Gaurav Chadha I want to build a bot to scrape a website.
But I want it to only scrape the website for new updates, that is, when information is updated on the website, my bot will scrape it and give me the updated information for me to send to telegram.
Can anyone show me how I can achieve this?
Because the nodes on buildship currently just scrape the entire website. Please Anyone with an idea of a workflow to achieve this? @everyone
3 Replies
1) You would need to have a database where you store the information
2) If it is a specific thing you want to track, for example, price, you can try using the OpenAI module designed for scraping data.
A database? Please to store what exactly? Please clarify that point for me.
What I need is to get the updated info from the website….when it’s updated.
How do you plan to track when the website is updated?
Let's say you want to track when a price changes.
On Monday you scrape the price value and save it in your database.
On Tuesday you scrape the price again and compare Monday's value to Tuesday's value. If they are not equal, the information has been updated and you can store the current value in your database.
This would be repeated as often as you set your code to run. (daily, weekly, etc.)
If you have other ideas on how this could be done, I would like to hear them.