How do I image upload capabilities on OpenAI assistant API on Buildship?

I followed this documentation to set up openai assistant node https://docs.buildship.com/ai-assistant/openai-assistant. It has thread ID and prompt in the body, but how do I add image upload to my assistant?

Solution

Passing an image_url into the API call body.
Creating a conditional to check if the image_url is available.
If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node.
If false, you will take the prompt and pass it to the OpenAI Assistant node.

Jump to solution

Henry Moses•6/3/24, 1:16 PM

Hey @Ada, what frontend are you using?

AdaOP•6/3/24, 1:32 PM

@Henry Moses I’m using flutterflow

Henry Moses•6/3/24, 1:35 PM

You want to

Pass in image_url in the body
Then pass that through the assistant node

This will allow a user to upload a image on the frontend and you can pass that image through the assistant node

AdaOP•6/3/24, 3:21 PM

@Henry Moses I passed image_url in the body, but I don’t know how to pass it to the assistant node. I didn’t see any options to image_url key into it. Could you explain step by step? I would really appreciate it.

Henry Moses•6/3/24, 4:22 PM

Yes, you can use the "upload file from URL" node. The URL will be the image_url, and the destination will be {unique-image-name}.png.

Screenshot_2024-06-03_at_12.21.25_PM.png

AdaOP•6/3/24, 10:29 PM

@Henry Moses this is how my nodes look like now, do I need to add image_url in the 3rd node (OpenAI assistant node)? sorry I'm very new with buildship
Really appreciate your help.

Henry Moses•6/3/24, 10:39 PM

Hey!

Henry Moses•6/3/24, 10:39 PM

Looking at this now

AdaOP•6/4/24, 12:59 AM

@Henry Moses thx. is it correct?

Henry Moses•6/4/24, 1:01 AM

What response are you getting? And yes, everything is right, just curious if you're getting the response you need.

AdaOP•6/4/24, 1:30 AM

@Henry Moses when i test it with an image URL, it doesn't seem to see the image.

Henry Moses•6/4/24, 2:04 AM

You could use the GPT-4 Vision API before the Assistant Node and pass it in the prompt. Use that for now and I can find another option for the future!

AdaOP•6/4/24, 2:22 AM

@Henry Moses thanks for the suggestion.
if i use gpt-4 vision api, would it still reply as the assistant?

Henry Moses•6/4/24, 3:41 AM

So use the gpt-4 vision api to describe what’s in the image. Then pass the response to the AI assistant node.

AdaOP•6/4/24, 12:55 PM

@Henry Moses could you explain how to do that?

Henry Moses•6/4/24, 1:00 PM

Can you join this channel

https://discord.gg/dvWJEkYU

Henry Moses•6/4/24, 1:01 PM

I can walk you through it!

Henry Moses•6/4/24, 1:08 PM

Recording a video for you now!

AdaOP•6/4/24, 1:27 PM

Sorry I'm not available to join the channel at the moment.
A video would be great! Thanks again

Henry Moses•6/4/24, 2:03 PM

Hey @Ada, in this video I show you how to use GPT-4 Vision and the AI assistant. You will see how to take the response from GPT-4 Vision and pass it to the AI assistant. Please be mindful that you need to ensure your prompt is based on the value you want to provide.

Henry Moses•6/4/24, 2:13 PM

https://www.loom.com/share/22899604dfcf48ebb71745aca2a51ab7?sid=1a09dae7-b16d-4edf-95c0-cf2be19c9852

Loom

ForAda

AdaOP•6/4/24, 10:39 PM

thanks for the video i will try it now

AdaOP•6/4/24, 11:06 PM

it's working now!
the test expects an image input everytime I prompt now.
sometimes the user would just want to ask a question with no images.
is there a way to ignore empty image_url?

AdaOP•6/4/24, 11:07 PM

@Henry Moses

Henry Moses•6/4/24, 11:33 PM

You can create a conditional

Henry Moses•6/4/24, 11:34 PM

Let me show you!

Henry Moses•6/4/24, 11:34 PM

One sec

Henry Moses•6/4/24, 11:38 PM

Henry Moses•6/4/24, 11:39 PM

AdaOP•6/5/24, 12:29 AM

it's working now, thanks for all your help! @Henry Moses

Henry Moses•6/5/24, 12:58 AM

Any time

AdaOP•6/5/24, 1:13 AM

@Henry Moses i have another question, is it possible for gpt4 vision to read base64 instead of url?
Because I'm trying to implement image upload from camera or phone gallery to the app on flutterflow.
what would be the best way to go about this?

Henry Moses•6/5/24, 1:27 AM

You can use the upload base64 node!

But in this case you wouldn’t need it, in Flutterflow you can upload an image and pass it through the API. It will be saved in your Firebase storage and provide you with a URL. This URL is what you pass through the API call.

AdaOP•6/5/24, 1:31 AM

Ohhh that's perfect!

Henry Moses•6/5/24, 2:28 AM

Let me know if there’s anything else I can help with!

Solution

Henry Moses•6/5/24, 4:01 PM

Passing an image_url into the API call body.
Creating a conditional to check if the image_url is available.
If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node.
If false, you will take the prompt and pass it to the OpenAI Assistant node.

Henry Moses•6/5/24, 4:01 PM

The broken down solution ^^^

HHenry Moses 1. Passing an image_url into the API call body. 2. Creating a conditional to che...

Henry Moses•6/5/24, 4:01 PM

@Gaurav Chadha feel free to mark as the solution when you get back online! Thank you

HHenry Moses @Gaurav Chadha feel free to mark as the solution when you get back online! Thank...

Gaurav Chadha•6/5/24, 7:43 PM

Thanks @Henry Moses