protoys
protoys12mo ago

Puppeteer?

Hi, when I AI-generate a node for scraping web content, the buildship model often creates a script with Puppeteer (which I would really love to use as there is no known alternative for me (axios is insufficient)). Whenever I try to test a puppeteer node, the script just stalls. Any ideas?
3 Replies
Gaurav Chadha
Gaurav Chadha12mo ago
cc @Deepanshu, would you like to answer this?
Deepanshu
Deepanshu12mo ago
@protoys can you share the generated script if possible?
protoys
protoysOP12mo ago
prompt: I need a node that scrapes dynamic web content, such as javascript, and remains undetected by the robot.txt of the scraped target. code: import puppeteer from 'puppeteer'; export default async function webScraper({ url }) { const browser = await puppeteer.launch({ headless: true }); const page = await browser.newPage(); await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36'); await page.goto(url, { waitUntil: 'networkidle0' }); const content = await page.content(); await browser.close(); return content; } please note: in this example I used a prompt that is intended to force a puppeteer-based solution proposition.