Puppeteer
For your future low code scraping users, it'd be great to have an option to put a list into the crawler.
Had to change url to urls and input an array instead of string.
I used GPT to edit the node logic & it was simply adding this;
for (const url of urls) {
const response = await fetch("https://puppeteer.buildship.run/v1/crawl", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
url, selector, maxRequestsPerCrawl, maxConcurrency, proxyUrls
})
});
Had to change url to urls and input an array instead of string.
I used GPT to edit the node logic & it was simply adding this;
for (const url of urls) {
const response = await fetch("https://puppeteer.buildship.run/v1/crawl", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
url, selector, maxRequestsPerCrawl, maxConcurrency, proxyUrls
})
});
