How do I image upload capabilities on OpenAI assistant API on Buildship?
I followed this documentation to set up openai assistant node https://docs.buildship.com/ai-assistant/openai-assistant. It has thread ID and prompt in the body, but how do I add image upload to my assistant?
Solution:Jump to solution
1. Passing an image_url into the API call body.
2. Creating a conditional to check if the image_url is available.
3. If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node.
4. If false, you will take the prompt and pass it to the OpenAI Assistant node....
33 Replies
Hey @Ada, what frontend are you using?
@Henry Moses I’m using flutterflow
You want to
1. Pass in image_url in the body
2. Then pass that through the assistant node
This will allow a user to upload a image on the frontend and you can pass that image through the assistant node
@Henry Moses I passed image_url in the body, but I don’t know how to pass it to the assistant node. I didn’t see any options to image_url key into it. Could you explain step by step? I would really appreciate it.
Yes, you can use the "upload file from URL" node. The URL will be the image_url, and the destination will be {unique-image-name}.png.
@Henry Moses this is how my nodes look like now, do I need to add image_url in the 3rd node (OpenAI assistant node)? sorry I'm very new with buildship
Really appreciate your help.
Hey!
Looking at this now
@Henry Moses thx. is it correct?
What response are you getting? And yes, everything is right, just curious if you're getting the response you need.
@Henry Moses when i test it with an image URL, it doesn't seem to see the image.
You could use the GPT-4 Vision API before the Assistant Node and pass it in the prompt. Use that for now and I can find another option for the future!
@Henry Moses thanks for the suggestion.
if i use gpt-4 vision api, would it still reply as the assistant?
So use the gpt-4 vision api to describe what’s in the image. Then pass the response to the AI assistant node.
@Henry Moses could you explain how to do that?
Can you join this channel
https://discord.gg/dvWJEkYU
I can walk you through it!
Recording a video for you now!
Sorry I'm not available to join the channel at the moment.
A video would be great! Thanks again
Hey @Ada, in this video I show you how to use GPT-4 Vision and the AI assistant. You will see how to take the response from GPT-4 Vision and pass it to the AI assistant. Please be mindful that you need to ensure your prompt is based on the value you want to provide.
thanks for the video i will try it now
it's working now!
the test expects an image input everytime I prompt now.
sometimes the user would just want to ask a question with no images.
is there a way to ignore empty image_url?
@Henry Moses
You can create a conditional
Let me show you!
One sec
it's working now, thanks for all your help! @Henry Moses
Any time 🎉🎊
@Henry Moses i have another question, is it possible for gpt4 vision to read base64 instead of url?
Because I'm trying to implement image upload from camera or phone gallery to the app on flutterflow.
what would be the best way to go about this?
You can use the upload base64 node!
But in this case you wouldn’t need it, in Flutterflow you can upload an image and pass it through the API. It will be saved in your Firebase storage and provide you with a URL. This URL is what you pass through the API call.
Ohhh that's perfect!
Let me know if there’s anything else I can help with!
Solution
1. Passing an image_url into the API call body.
2. Creating a conditional to check if the image_url is available.
3. If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node.
4. If false, you will take the prompt and pass it to the OpenAI Assistant node.
The broken down solution ^^^
@Gaurav Chadha feel free to mark as the solution when you get back online! Thank you
Thanks @Henry Moses 🙌