tom•5d ago

tom - Encoding buffer to webP: "input too large...

Encoding buffer to webP: "input too large" after Upload File Trigger

Solution:

hey @tom, the webp-converter package you were using in your code has been deprecated and seems to be broken. But luckily we can just use ffmpeg to do the conversion for us and BuildShip has this installed by default. I'm sharing with you a workflow that takes a file buffer and converts it to webp using ffmpeg and return the converted buffer. It then upload this buffer to BuildShip storage and returns the download URL. You can of course update it to return the buffer directly. https://buildship.app/remix/62601e1e-dcb5-41ec-9c94-ceb41aa09a2c...

BuildShip - Visual backend workflow builder

Visually build workflows and powerful backend logic for your apps. Powered by AI, connect to anything with or without code.

Jump to solution

9 Replies

tom•5d ago

I might be missing some principle aspect of Buildship. I'm using the Upload File trigger, after that a node which is supposed to encode it in webP:

 import webp from 'webp-converter';
import fs from 'fs';
import { v4 as uuidv4 } from 'uuid';

export default async function convertToWebp({ imageBuffer }) {
    let isSuccess = true;
    try {
        const maxBufferSize = 10485760; // 10MB

        if (Buffer.byteLength(imageBuffer, 'base64') > maxBufferSize) {
            throw new Error('Input buffer size exceeds the limit of 10MB.');
        }

        const tempFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.png`;
        const outputFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.webp`;

        // Write the input buffer to a temporary file
        fs.writeFileSync(tempFileName, Buffer.from(imageBuffer, 'base64'));

        // Convert the temporary file to WebP format
        await webp.cwebp(tempFileName, outputFileName, "-q 80");

        // Read the converted WebP file
        const webpBuffer = fs.readFileSync(outputFileName);

        // Clean up temporary files
        fs.unlinkSync(tempFileName);
        fs.unlinkSync(outputFileName);

        return { webpBuffer: webpBuffer.toString('base64'), isSuccess };
    } catch (error) {
        isSuccess = false;
        console.error(`WebP conversion error: ${error.message}`);
        return { webpBuffer: null, isSuccess };
    }
}

 import webp from 'webp-converter';
import fs from 'fs';
import { v4 as uuidv4 } from 'uuid';

export default async function convertToWebp({ imageBuffer }) {
    let isSuccess = true;
    try {
        const maxBufferSize = 10485760; // 10MB

        if (Buffer.byteLength(imageBuffer, 'base64') > maxBufferSize) {
            throw new Error('Input buffer size exceeds the limit of 10MB.');
        }

        const tempFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.png`;
        const outputFileName = `${process.env.BUCKET_FOLDER_PATH}/temp/${uuidv4()}.webp`;

        // Write the input buffer to a temporary file
        fs.writeFileSync(tempFileName, Buffer.from(imageBuffer, 'base64'));

        // Convert the temporary file to WebP format
        await webp.cwebp(tempFileName, outputFileName, "-q 80");

        // Read the converted WebP file
        const webpBuffer = fs.readFileSync(outputFileName);

        // Clean up temporary files
        fs.unlinkSync(tempFileName);
        fs.unlinkSync(outputFileName);

        return { webpBuffer: webpBuffer.toString('base64'), isSuccess };
    } catch (error) {
        isSuccess = false;
        console.error(`WebP conversion error: ${error.message}`);
        return { webpBuffer: null, isSuccess };
    }
}

Even with a filesize of 90KB, I'm getting an error of Input too large

Gaurav Chadha•5d ago

@tom The response from the logs - input&output is too large and is not an error or issue, we are excluding very large JSON responses in the logs system currently and displaying the above message, it should work correctly if you connect it with your client (frontend) and send the response. We'll update the limit and will improve the UX around this. You'll still be able to use the output in the next nodes.

tom•5d ago

@Gaurav Chadha thank you for a quick reply. I was hoping I can blame BuildShip for my incompetence 😄 Then I'm failing for some other reason. Do you have any recommendations how to convert the images to webP? The AI suggest usings Jimp package, but it seems that there's some issue with it currently, also reported here: https://github.com/jimp-dev/jimp/issues/1350

GitHub

TypeError: Jimp.read is not a function · Issue #1350 · jimp-dev/jimp

I expect to be able to use the Jimp.read() function to load and manipulate images without encountering an error. The Jimp.read() function is throwing a Jimp.read is not a function error, and I'...

tom•5d ago

This is what I ended up using: https://pastecode.io/s/57oruy2o (can't paste here due to message size limit), but I'm running into issues creating the temp paths.

WebP conversion promise rejected: ENOENT: no such file or directory, open '/usr/src/app/bucket/builtNodes/temp/64baa5c7-7adb-451d-b2c2-f3476c4a0647.webp'

Is there some docs about how to create the temp paths and I'm just missing something obvious?

Gaurav Chadha•5d ago

cc @Luis can you help check this?

Luis•5d ago

Sure thing

Solution

Luis•5d ago

BuildShip - Visual backend workflow builder

Visually build workflows and powerful backend logic for your apps. Powered by AI, connect to anything with or without code.

tom•4d ago

@Gaurav Chadha , @Luis you my dear sirs are my heroes! Works perfectly and also gives me the chance to learn Buildship and JS/TS more learning the code you provided! Thank you so much! Buildship earned a forever 10 on the NPS from me 🤝

Luis•4d ago

Glad to hear that @tom 🙂. We'll be around if you ever need more help in the future.