Ada
Ada5mo ago

How do I image upload capabilities on OpenAI assistant API on Buildship?

I followed this documentation to set up openai assistant node https://docs.buildship.com/ai-assistant/openai-assistant. It has thread ID and prompt in the body, but how do I add image upload to my assistant?
Solution:
1. Passing an image_url into the API call body. 2. Creating a conditional to check if the image_url is available. 3. If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node. 4. If false, you will take the prompt and pass it to the OpenAI Assistant node....
Jump to solution
33 Replies
Henry Moses
Henry Moses5mo ago
Hey @Ada, what frontend are you using?
Ada
Ada5mo ago
@Henry Moses I’m using flutterflow
Henry Moses
Henry Moses5mo ago
You want to 1. Pass in image_url in the body 2. Then pass that through the assistant node This will allow a user to upload a image on the frontend and you can pass that image through the assistant node
Ada
Ada5mo ago
@Henry Moses I passed image_url in the body, but I don’t know how to pass it to the assistant node. I didn’t see any options to image_url key into it. Could you explain step by step? I would really appreciate it.
Henry Moses
Henry Moses5mo ago
Yes, you can use the "upload file from URL" node. The URL will be the image_url, and the destination will be {unique-image-name}.png.
No description
Ada
Ada5mo ago
@Henry Moses this is how my nodes look like now, do I need to add image_url in the 3rd node (OpenAI assistant node)? sorry I'm very new with buildship Really appreciate your help.
No description
No description
Henry Moses
Henry Moses5mo ago
Hey! Looking at this now
Ada
Ada5mo ago
@Henry Moses thx. is it correct?
Henry Moses
Henry Moses5mo ago
What response are you getting? And yes, everything is right, just curious if you're getting the response you need.
Ada
Ada5mo ago
@Henry Moses when i test it with an image URL, it doesn't seem to see the image.
No description
Henry Moses
Henry Moses5mo ago
You could use the GPT-4 Vision API before the Assistant Node and pass it in the prompt. Use that for now and I can find another option for the future!
Ada
Ada5mo ago
@Henry Moses thanks for the suggestion. if i use gpt-4 vision api, would it still reply as the assistant?
Henry Moses
Henry Moses5mo ago
So use the gpt-4 vision api to describe what’s in the image. Then pass the response to the AI assistant node.
Ada
Ada5mo ago
@Henry Moses could you explain how to do that?
Henry Moses
Henry Moses5mo ago
Can you join this channel https://discord.gg/dvWJEkYU I can walk you through it! Recording a video for you now!
Ada
Ada5mo ago
Sorry I'm not available to join the channel at the moment. A video would be great! Thanks again
Henry Moses
Henry Moses5mo ago
Hey @Ada, in this video I show you how to use GPT-4 Vision and the AI assistant. You will see how to take the response from GPT-4 Vision and pass it to the AI assistant. Please be mindful that you need to ensure your prompt is based on the value you want to provide.
Ada
Ada5mo ago
thanks for the video i will try it now
Ada
Ada5mo ago
it's working now! the test expects an image input everytime I prompt now. sometimes the user would just want to ask a question with no images. is there a way to ignore empty image_url?
No description
No description
No description
Ada
Ada5mo ago
@Henry Moses
Henry Moses
Henry Moses5mo ago
You can create a conditional Let me show you! One sec
Henry Moses
Henry Moses5mo ago
No description
Henry Moses
Henry Moses5mo ago
No description
Ada
Ada5mo ago
it's working now, thanks for all your help! @Henry Moses
Henry Moses
Henry Moses5mo ago
Any time 🎉🎊
Ada
Ada5mo ago
@Henry Moses i have another question, is it possible for gpt4 vision to read base64 instead of url? Because I'm trying to implement image upload from camera or phone gallery to the app on flutterflow. what would be the best way to go about this?
Henry Moses
Henry Moses5mo ago
You can use the upload base64 node! But in this case you wouldn’t need it, in Flutterflow you can upload an image and pass it through the API. It will be saved in your Firebase storage and provide you with a URL. This URL is what you pass through the API call.
Ada
Ada5mo ago
Ohhh that's perfect!
Henry Moses
Henry Moses5mo ago
Let me know if there’s anything else I can help with!
Solution
Henry Moses
Henry Moses5mo ago
1. Passing an image_url into the API call body. 2. Creating a conditional to check if the image_url is available. 3. If true, you will use the OpenAI Vision API and pass the response to the Assistant AI node. 4. If false, you will take the prompt and pass it to the OpenAI Assistant node.
Henry Moses
Henry Moses5mo ago
The broken down solution ^^^ @Gaurav Chadha feel free to mark as the solution when you get back online! Thank you
Gaurav Chadha
Gaurav Chadha5mo ago
Thanks @Henry Moses 🙌