Chat in OCI Generative AI

Use the provided large language chat models in OCI Generative AI to ask questions and get conversational responses through an AI chatbot.

• Console
• CLI
• API

• 1. In the navigation bar of the Console, select a region with Generative AI, for example, US Midwest (Chicago) or UK South (London). See which models are offered in your region.
  2. Open the navigation menu  and select Analytics & AI. Under AI Services, select Generative AI.
  3. Select a compartment that you have permission to work in. If you don't see the playground, ask an administrator to give you access to Generative AI resources and then return to the following steps.
  4. Click Playground.
  5. Click Chat.
  6. Select a model for the chat experience by performing one of the following actions:
     • In the Model list, select a pretrained model such as meta.llama-3.1-70b-instruct, cohere.command-r-08-2024, cohere.command-r-plus-08-2024, or a custom model. The custom models are displayed as model name (endpoint name).
     • Click View model details, select a model and then click Choose model.
     Note
     
     

     • The meta.llama-3.1-405b-instruct model isn't available for on-demand access in all regions. To access this model, perform one of the following options:

       • Set up dedicated access: Switch to a a region supported for dedicated clusters for the meta.llama-3.1-405b-instruct chat model. Then, create a a hosting cluster and an endpoint for this model.
       • Switch to an on-demand region: Switch to the US Midwest (Chicago) region that's supported for on-demand inferencing for the meta.llama-3.1-405b-instruct chat model.

     • The meta.llama-3.2-11b-vision-instruct model isn't available for on-demand access in any region. To access this model, perform the following option:

       • Set up dedicated access: Switch to a a region supported for dedicated clusters for the meta.llama-3.2-11b-vision-instruct chat model. Then, create a a hosting cluster and an endpoint for this model.

     Learn about costs and model retirements for on-demand and dedicated serving modes.

  7. Start a conversation by typing a prompt or selecting an example from the Example list to use as a base prompt or to learn from.
     For models that accept images such as meta.llama-3.2-90b-vision-instruct, you can add a .png or .jpg image with a size of 5 MB or less.
  8. (Optional) Set new values for the parameters. For parameter details, see Chat Model Parameters.
  9. Click Submit.
  10. Enter a new prompt or to continue your chat conversation, enter a follow-up prompt and click Submit.
      Important
      
      In the playground, for the vision models that accept images:

      • Submitting an image without a prompt doesn't work. When you submit an image, you must submit a prompt about that image in the same request. You can then submit follow-up prompts and the model keeps the context of the conversation.
      • To add the next image and text, you must clear the chat which results in losing context of the previous conversation because of clearing the chat.
  11. (Optional) To change the responses, click Clear chat, update the prompts and parameters, and click Submit. Repeat this step until you're happy with the output.
  12. (Optional) To copy the code that generated the output, click View code, select a programming language or a framework, click Copy code, paste the code into a file and save the file. Ensure that the file maintains the format of the pasted code.
      Tip
      
      

      If you're using the code in an application:

      • Ensure that you authenticate your code.
      • Review LlamaIndex Integration and LangChain Integration.
  13. (Optional) To start a new conversation, click Clear chat.
      Note
      
      

      • When you click Clear chat, the chat conversation is erased, but the model parameters remain unchanged, allowing you to continue using the last settings you applied.

        If you switch to a different feature, such as Generation, and then return to the Chat playground, both the chat conversation and model parameters will reset to their default values.

      Learn about Cohere chat parameters.

• To chat, use the chat-result operation in Generative AI Inference CLI.

  Enter the following command for a list of options to use with the chat-result operation.

  Command

  CopyTry It

  oci generative-ai-inference chat-result -h

  For a complete list of parameters and values for the OCIGenerative AI CLI commands, see Generative AI Inference CLI and Generative AI Management CLI.

• Run the Chat operation to chat using the large language models.

  For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.