tom
tomβ€’3w ago

tom - Encoding buffer to webP: "input too large...

Encoding buffer to webP: "input too large" after Upload File Trigger
Solution:
hey @tom, the webp-converter package you were using in your code has been deprecated and seems to be broken. But luckily we can just use ffmpeg to do the conversion for us and BuildShip has this installed by default. I'm sharing with you a workflow that takes a file buffer and converts it to webp using ffmpeg and return the converted buffer. It then upload this buffer to BuildShip storage and returns the download URL. You can of course update it to return the buffer directly. https://buildship.app/remix/62601e1e-dcb5-41ec-9c94-ceb41aa09a2c...
BuildShip - Visual backend workflow builder
Visually build workflows and powerful backend logic for your apps. Powered by AI, connect to anything with or without code.
Jump to solution
9 Replies
tom
tomOPβ€’3w ago
I might be missing some principle aspect of Buildship. I'm using the Upload File trigger, after that a node which is supposed to encode it in webP:
import webp from 'webp-converter';
import fs from 'fs';
import { v4 as uuidv4 } from 'uuid';

export default async function convertToWebp({ imageBuffer }) {
let isSuccess = true;
try {
const maxBufferSize = 10485760; // 10MB

if (Buffer.byteLength(imageBuffer, 'base64') > maxBufferSize) {
throw new Error('Input buffer size exceeds the limit of 10MB.');
}

const tempFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.png`;
const outputFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.webp`;

// Write the input buffer to a temporary file
fs.writeFileSync(tempFileName, Buffer.from(imageBuffer, 'base64'));

// Convert the temporary file to WebP format
await webp.cwebp(tempFileName, outputFileName, "-q 80");

// Read the converted WebP file
const webpBuffer = fs.readFileSync(outputFileName);

// Clean up temporary files
fs.unlinkSync(tempFileName);
fs.unlinkSync(outputFileName);

return { webpBuffer: webpBuffer.toString('base64'), isSuccess };
} catch (error) {
isSuccess = false;
console.error(`WebP conversion error: ${error.message}`);
return { webpBuffer: null, isSuccess };
}
}
import webp from 'webp-converter';
import fs from 'fs';
import { v4 as uuidv4 } from 'uuid';

export default async function convertToWebp({ imageBuffer }) {
let isSuccess = true;
try {
const maxBufferSize = 10485760; // 10MB

if (Buffer.byteLength(imageBuffer, 'base64') > maxBufferSize) {
throw new Error('Input buffer size exceeds the limit of 10MB.');
}

const tempFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.png`;
const outputFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.webp`;

// Write the input buffer to a temporary file
fs.writeFileSync(tempFileName, Buffer.from(imageBuffer, 'base64'));

// Convert the temporary file to WebP format
await webp.cwebp(tempFileName, outputFileName, "-q 80");

// Read the converted WebP file
const webpBuffer = fs.readFileSync(outputFileName);

// Clean up temporary files
fs.unlinkSync(tempFileName);
fs.unlinkSync(outputFileName);

return { webpBuffer: webpBuffer.toString('base64'), isSuccess };
} catch (error) {
isSuccess = false;
console.error(`WebP conversion error: ${error.message}`);
return { webpBuffer: null, isSuccess };
}
}
Even with a filesize of 90KB, I'm getting an error of Input too large
Gaurav Chadha
Gaurav Chadhaβ€’3w ago
@tom The response from the logs - input&output is too large and is not an error or issue, we are excluding very large JSON responses in the logs system currently and displaying the above message, it should work correctly if you connect it with your client (frontend) and send the response. We'll update the limit and will improve the UX around this. You'll still be able to use the output in the next nodes.
tom
tomOPβ€’3w ago
@Gaurav Chadha thank you for a quick reply. I was hoping I can blame BuildShip for my incompetence πŸ˜„ Then I'm failing for some other reason. Do you have any recommendations how to convert the images to webP? The AI suggest usings Jimp package, but it seems that there's some issue with it currently, also reported here: https://github.com/jimp-dev/jimp/issues/1350
GitHub
TypeError: Jimp.read is not a function Β· Issue #1350 Β· jimp-dev/jimp
I expect to be able to use the Jimp.read() function to load and manipulate images without encountering an error. The Jimp.read() function is throwing a Jimp.read is not a function error, and I'...
tom
tomOPβ€’3w ago
This is what I ended up using: https://pastecode.io/s/57oruy2o (can't paste here due to message size limit), but I'm running into issues creating the temp paths. WebP conversion promise rejected: ENOENT: no such file or directory, open '/usr/src/app/bucket/builtNodes/temp/64baa5c7-7adb-451d-b2c2-f3476c4a0647.webp' Is there some docs about how to create the temp paths and I'm just missing something obvious?
Gaurav Chadha
Gaurav Chadhaβ€’3w ago
cc @Luis can you help check this?
Luis
Luisβ€’3w ago
Sure thing
Solution
Luis
Luisβ€’3w ago
hey @tom, the webp-converter package you were using in your code has been deprecated and seems to be broken. But luckily we can just use ffmpeg to do the conversion for us and BuildShip has this installed by default. I'm sharing with you a workflow that takes a file buffer and converts it to webp using ffmpeg and return the converted buffer. It then upload this buffer to BuildShip storage and returns the download URL. You can of course update it to return the buffer directly. https://buildship.app/remix/62601e1e-dcb5-41ec-9c94-ceb41aa09a2c
BuildShip - Visual backend workflow builder
Visually build workflows and powerful backend logic for your apps. Powered by AI, connect to anything with or without code.
tom
tomOPβ€’3w ago
@Gaurav Chadha , @Luis you my dear sirs are my heroes! Works perfectly and also gives me the chance to learn Buildship and JS/TS more learning the code you provided! Thank you so much! Buildship earned a forever 10 on the NPS from me 🀝
Luis
Luisβ€’3w ago
Glad to hear that @tom πŸ™‚. We'll be around if you ever need more help in the future.