Puppeteer?
Hi, when I AI-generate a node for scraping web content, the buildship model often creates a script with Puppeteer (which I would really love to use as there is no known alternative for me (axios is insufficient)). Whenever I try to test a puppeteer node, the script just stalls.
Any ideas?
3 Replies
cc @Deepanshu, would you like to answer this?
@protoys can you share the generated script if possible?
prompt:
I need a node that scrapes dynamic web content, such as javascript, and remains undetected by the robot.txt of the scraped target.
code:
import puppeteer from 'puppeteer';
export default async function webScraper({
url
}) {
const browser = await puppeteer.launch({
headless: true
});
const page = await browser.newPage();
await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36');
await page.goto(url, {
waitUntil: 'networkidle0'
});
const content = await page.content();
await browser.close();
return content;
}
please note:
in this example I used a prompt that is intended to force a puppeteer-based solution proposition.