Create a dedicated AI cluster resource in OCI
Generative AI to host endpoints for pretrained base models and custom models.
Important
- Not Available on-demand: All OCI
Generative AI foundational pretrained models supported for the on-demand serving mode that use the text generation and summarization APIs (including the playground) are now retired. We recommend that you use the chat models instead.
- Can be hosted on clusters: If you host a summarization or a generation model such as
cohere.command on a dedicated AI cluster, (dedicated serving mode), you can continue to use that model until it's retired. These models, when hosted on a dedicated AI cluster are only available in US Midwest (Chicago). See Retiring the Models for retirement dates and definitions.
-
In the navigation bar of the Console, select a region with Generative AI, for example, US Midwest (Chicago) or UK South (London). See which models are offered in your region.
-
Open the navigation menu and click Analytics & AI. Under AI Services, click Generative AI.
-
Select a compartment in which you want to host the models.
Ensure that you have
permission to use or manage
generative-ai-family and
object-family resources in this compartment.
-
In the left navigation, select a compartment that you have permission to work in.
-
Click Dedicated AI clusters.
-
Click Create dedicated AI cluster.
-
Select a compartment to create the dedicated AI cluster in. The default
compartment is the one you selected in step 3, but you can select any
compartment that you have permission to work in.
- (Optional)
Enter a name and description for the cluster. If you don't enter a name, the
system generates one that you can change later.
The generated name has the format
generativeaidedicatedaicluster<timestamp>.
For example:
generativeaidedicatedaicluster20240601202357
-
For Cluster type, click Hosting.
-
For Base model, select the base model for the models
that you want to host on this cluster.
meta.llama-3.2-11b-vision-instruct - Provisions 1 Small Generic V2 unit.meta.llama-3.2-90b-vision-instruct - Provisions 1 Large Generic V2 unit.meta.llama-3.1-70b-instruct - Provisions 1 Large Generic unit.meta.llama-3.1-405b-instruct - Provisions 1 Large Generic 4 unit.meta.llama-3-70b-instruct - Provisions 1 Large Generic unit. (This model is deprecated.)cohere.command-r-16k - Provisions 1 Small Cohere V2 unit.cohere.command-r-08-2024 - Provisions 1 Small Cohere V2 unit.cohere.command-r-plus - Provisions 1 Large Cohere V2_2 unit.cohere.command-r-plus-08-2024 - Provisions 1 Large Cohere V2_2 unit.
cohere.embed.english-light-v3.0 - Provisions 1 Embed Cohere unit.cohere.embed.english-v3.0 - Provisions 1 Embed Cohere unit.cohere.embed.multilingual-light-v3.0 - Provisions 1 Embed Cohere unit.cohere.embed.multilingual-v3.0 - Provisions 1 Embed Cohere unit.
Note
The model list only includes the supported version of the base models.
- (Optional)
Increase the number of instances in the Model replica field.
Important
When you create a cluster for hosting models for inference, by default one unit is created for the base model that you select. To increase the throughput, you can increase the number of instances in the Model replica field now, or later when you edit the cluster. For example, creating two model replicas on this cluster, requires two units.
-
Read the commitment unit hours for the hosting cluster and select the
checkbox to agree to the commitment.
- (Optional)
Click Show advanced options and assign tags to this
cluster.
-
Click Create.
Note
Clusters take a few minutes to create. After the cluster is in an active
state, you can select that cluster to host a model, when you create an
endpoint for that model.