Coder Legion - Recent posts tagged oci-genai

OCI Generative AI Agents: Building Enterprise RAG Applications Without Code

Fri, 23 Jan 2026 07:51:18 +0000

Oracle Cloud Infrastructure (OCI) Generative AI Agents represents a paradigm shift in building AI applications—enabling enterprises to deploy production-grade RAG (Retrieval-Augmented Generation) systems without writing a single line of code. This comprehensive guide explores the architecture, capabilities, and practical implementation of OCI Gen AI Agents for enterprise use cases.

What are OCI Generative AI Agents?

OCI Generative AI Agents is a fully managed service that combines the power of large language models (LLMs) with an intelligent retrieval system to create contextually relevant answers by searching your knowledge base, making your AI applications smart and efficient.

AI Agents are applications of LLMs packaged and validated, ready to use out of the box for enterprise deployments.

Core Capabilities

An AI agent is an LLM-based autonomous system that understands and generates human-like text with high answerability and groundedness. Specifically, an AI agent can:

Perform complex tasks autonomously: Execute multi-step workflows without constant human intervention
Mimic human chain-of-thought processing: Reason through problems systematically
Be an effective tool for automating processes: Handle repetitive tasks at scale
Utilize your knowledge: Access and reason over enterprise-specific information

How Agents Work

An AI agent understands and interprets user queries, determines necessary actions, retrieves data if necessary, and executes actions to deliver accurate, contextually relevant responses.

The Complete Workflow:

User Input: Natural language query submitted to the agent
Query Understanding: LLM interprets the intent and context
Knowledge Retrieval: Agent searches knowledge base for relevant information
Response Formulation: LLM generates answer based on retrieved context
Output: User receives response with source citations

Key Agent Components

Knowledge Base

A knowledge base is the base for all the data sources that an agent can use to retrieve information for its chat answers. An agent connects to a knowledge base, which is a vector-based storage that can connect to and ingest data from a data source.

Knowledge bases support multiple data store types:

OCI Object Storage: Simple file-based storage (PDF, TXT, JSON, HTML, Markdown)
OCI OpenSearch: Pre-indexed documents with custom chunking
Oracle Database 23ai: AI Vector Search with structured and unstructured data
MySQL HeatWave: In-database vector capabilities

Data Sources

Data sources provide connection information to the data stores that an agent uses to generate responses. Each knowledge base can have multiple data sources, though only one bucket is allowed per Object Storage data source.

Data Ingestion

Data ingestion is a process that extracts data from data source documents, converts it into a structured format suitable for analysis, and then stores it in a knowledge database.

The ingestion pipeline handles:

Document parsing and text extraction
Chunking into smaller segments
Embedding generation
Vector indexing
Metadata extraction

Agent Concepts and Terminology

Session

A session is a series of exchanges where the user sends queries or prompts and the agent responds with relevant information. Sessions maintain conversation context, enabling multi-turn dialogues where follow-up questions reference previous exchanges.

Agent Endpoint

An agent endpoint is a specific point of access in a network or system that agents use to interact with other systems or devices. Endpoints provide:

REST API access for programmatic integration
Chat interface for interactive testing
Authentication and authorization controls
Rate limiting and monitoring

Trace

A trace is a feature to track and display conversation history during a chat conversation. Traces enable:

Debugging agent responses
Understanding retrieval decisions
Monitoring performance
Auditing user interactions

Citation

A citation is the source of information for an agent's response. OCI Generative AI Agents provides source attribution for all answers, displaying:

Document name
Page number (for PDFs)
Hyperlinks to original sources
Relevance scores

This transparency enables users to verify information and builds trust in agent responses.

Content Moderation

Content moderation is a feature designed to help detect or filter out certain toxic, violent, abusive, hateful, threatening, insulting, and harassing phrases from generated responses or user prompts in LLMs.

Content moderation protects both users and organizations by:

Filtering inappropriate user inputs
Screening generated outputs
Blocking harmful content
Maintaining brand safety

Object Storage Guidelines for GenAI Agents

When using Object Storage as your data source, follow these critical guidelines to ensure successful ingestion.

File Requirements

Data for GenAI Agents must be uploaded as files to an Object Storage bucket.

Supported File Types:

As of 2025, OCI Generative AI Agents supports:

PDF files
TXT files
JSON files
HTML files
Markdown (MD) files

File Size Limits:

Each file must be no larger than 100 MB
Files exceeding this limit are ignored during ingestion
PDF images, charts, and reference tables must not exceed 8 MB within each PDF

Bucket Restrictions:

Only one bucket is allowed per data source
However, you can use multiple data sources per knowledge base
You can ingest up to 1,000 files from an OCI Object Storage bucket into a knowledge base

Chart and Table Guidelines

Charts:

No special preparation is needed for charts as long as they're two-dimensional with labeled axes. The model can answer questions about charts without explicit explanations.

Requirements for optimal chart understanding:

Two-dimensional visualization
Clearly labeled X and Y axes
Legend (if multiple data series)
Title or caption
Size within 8 MB limit

Tables:

Use reference tables with several rows and columns. For enhanced table understanding, ensure:

All cells separated with visible lines or object boundaries
All columns including the first column have header names
Each table has more than one column and more than one row (excluding headers)
Tables use consistent formatting

Example ingestion message:

Count of tables that support enhanced table understanding in following PDFs:
- 2025_Report1.pdf has 4 tables processed successfully
- 2025_Report2.pdf has 3 tables processed successfully

Hyperlinks and URLs

All hyperlinks present in PDF documents are extracted and displayed as hyperlinks in chat responses. This enables users to navigate directly to referenced resources without manual searching.

Planning for Future Data

If data isn't yet available, create an empty folder for the data source and populate it later. This allows you to:

Set up knowledge base infrastructure in advance
Define data source connections
Run ingestion jobs when content becomes available
Maintain consistent agent configuration

Advanced Features (March 2025 Release)

Oracle released significant enhancements to the RAG tool in March 2025 based on customer feedback.

Enhanced Response Quality

Improved accuracy by refining contextual understanding and relevance of generated answers through:

Better semantic understanding
More precise context extraction
Improved answer synthesis

Hybrid Search

Combines keyword and vector search for highly precise, efficient retrieval. Hybrid search uses:

Keyword (BM25) search: Exact term matching for specific terminology
Vector search: Semantic similarity for conceptual matches
Fusion ranking: Combining scores from both approaches

This dual approach ensures agents find documents matching both semantic meaning and specific keywords.

Extract insights from rich content formats such as images and charts. Enhanced capabilities include:

Image understanding within PDFs
Chart data extraction
Table structure recognition
Diagram interpretation

Custom Instructions

Define preferences to fine-tune behavior and response styles of the RAG tool. Custom instructions enable:

Setting agent personality and tone
Defining response formats
Establishing domain-specific terminology
Configuring citation styles

Multi-Lingual Support

Seamlessly interact across several languages, including:

French
Spanish
Portuguese
Arabic
German
Italian
Japanese
English

This enables global deployment with localized experiences.

Multiple Knowledge Bases

To help improve coverage and depth of knowledge, set up agents to retrieve information from several knowledge bases. Benefits include:

Specialized knowledge bases per domain
Improved organization and maintenance
Better access control
Reduced search latency

Metadata Ingestion & Filtering

Use metadata to refine searches and categorize content more effectively. Metadata filtering enables:

Date-based filtering (e.g., "documents from 2024")
Department/category filtering
Author-based retrieval
Custom taxonomy support

Users can apply metadata filters during chat conversations to narrow results.

Building Your First Agent: Step-by-Step

Prerequisites

OCI Account: With appropriate IAM permissions
Identity Domain: Required for agent creation
Region Subscription: Chicago region (primary availability)
Object Storage Bucket: With source documents uploaded

Step 1: Create Knowledge Base

Navigate to Analytics & AI → Generative AI Agents → Knowledge Bases

Click Create knowledge base
Enter name (1-255 characters, start with letter or underscore)
Select compartment
Choose data store type (Object Storage, OpenSearch, or Database)
Configure data source:
- Select Object Storage bucket
- Choose files or use prefixes
- Select "Automatically start ingestion job"

Step 2: Monitor Ingestion

The ingestion job automatically starts and processes:

Document parsing
Text extraction
Chunking (512 tokens for OpenSearch)
Embedding generation
Vector indexing

Check ingestion logs to confirm success and view detailed status including number of ingested, ignored, and failed files.

Step 3: Create Agent

Navigate to Agents page:

Click Create agent
Enter agent name and description
Select LLM (Command R+, Llama 4, etc.)
Add RAG tool configuration:
- Select knowledge base(s)
- Configure custom instructions
- Set hybrid search options
Enable content moderation (optional)
Choose to create endpoint now or later

Step 4: Create Agent Endpoint

If not created during agent setup:

Navigate to agent details
Click Create endpoint
Enter endpoint name
Select region
Configure authentication

Copy the endpoint OCID for API access.

Step 5: Test in Playground

Launch chat from agent or endpoint details page
Select agent and endpoint
Submit test queries
Review responses and citations
Verify knowledge base retrieval

Step 6: Integrate via API

Use the REST API to integrate with applications:

import requests

endpoint_url = "https://agent.generativeai.us-chicago-1.oci.oraclecloud.com/20240531/agentEndpoints/{endpoint_id}/actions/chat"

payload = {
    "message": "What is our company policy on remote work?",
    "sessionId": "session-123",
    "shouldStream": False
}

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer <your_token>"
}

response = requests.post(endpoint_url, json=payload, headers=headers)
agent_response = response.json()

print(agent_response['message'])
print(agent_response['citations'])

Use Cases for OCI Generative AI Agents

1. Customer Support

In the customer service industry, RAG agents retrieve information from a company's knowledge base to provide correct and contextually relevant answers to customer inquiries, reducing response times and improving customer satisfaction.

Benefits:

24/7 availability
Instant access to knowledge base
Consistent responses
Escalation to human agents when needed

2. Legal Research

Legal professionals can use RAG agents to quickly find precedents and relevant case law from vast legal databases, streamlining the research process and ensuring thorough consideration of relevant legal texts.

3. Healthcare and Medical Guidance

In healthcare, RAG agents help doctors and medical staff by providing diagnostic support, retrieving medical literature, treatment protocols, and patient history to suggest potential diagnoses and treatments.

4. Financial Analysis

In finance, RAG agents analyze large volumes of financial data, reports, and news to provide analysis and recommendations for traders and analysts, helping them make informed decisions.

5. Travel Planning

In the travel industry, RAG agents serve as interactive travel guides, pulling information on destinations, weather, local attractions, and regulations to provide personalized travel advice and itineraries.

6. HR and Internal Knowledge

Deploy agents for employee self-service:

Benefits information
Policy questions
Onboarding guidance
Training materials

Best Practices for Production Agents

1. Organize Knowledge Bases Strategically

Create separate knowledge bases for different domains
Use multiple data sources for comprehensive coverage
Maintain clear naming conventions
Document knowledge base purposes

2. Optimize Document Preparation

Keep files under 100 MB
Ensure charts have clear labels and legends
Format tables with proper headers
Include descriptive filenames
Add custom URLs for citation tracking

3. Leverage Metadata Effectively

Tag documents with:

Department/category
Publication date
Author
Document type
Sensitivity level

4. Implement Content Moderation

Enable input and output filtering
Define custom moderation rules
Monitor flagged content
Establish escalation procedures

5. Monitor and Iterate

Track key metrics:

Response accuracy
Citation quality
User satisfaction
Query volume
Latency

6. Plan for Scale

Request limit increases proactively (agents, knowledge bases, files)
Implement caching for common queries
Design for multi-region deployment
Establish backup and recovery procedures

Limitations and Considerations

Regional Availability

As of early 2025, OCI Generative AI Agents is primarily available in:

US Midwest (Chicago) - Primary region

Additional regions are planned for future release.

Resource Limits

Default limits per tenancy:

2 agents
2 knowledge bases per agent
1,000 files per Object Storage data source

Limits can be increased through service request.

File Type Restrictions

Supported formats are limited to PDF, TXT, JSON, HTML, and Markdown. Other formats (DOCX, XLSX, PPTX) require conversion before upload.

Security and Compliance

Data Privacy

Data remains within your tenancy
No training on customer data
Encryption at rest and in transit
Access controlled via IAM policies

Enterprise Features

RBAC: Fine-grained access control
Audit Logging: Complete activity trails
Content Moderation: Built-in safety controls
Citation Tracking: Source attribution for all responses

Conclusion

OCI Generative AI Agents democratizes enterprise AI by providing a fully managed, no-code platform for building production-grade RAG applications. Key advantages include:

Ease of Use:

No coding required for basic setup
Visual interface for configuration
Automated ingestion and indexing
Built-in testing playground

Enterprise Features:

Multiple knowledge base support
Hybrid search capabilities
Multi-lingual support
Content moderation
Source attribution

Flexibility:

Support for various data sources
Custom instructions
Metadata filtering
API and chat interfaces

Scalability:

Fully managed service
Enterprise-grade security
Compliance ready
Oracle infrastructure reliability

By following the guidelines for Object Storage (100 MB file limit, supported formats, chart/table preparation), leveraging advanced features (hybrid search, multi-modal parsing, metadata filtering), and implementing best practices, organizations can deploy sophisticated AI agents that deliver accurate, grounded, and trustworthy responses to users.

Whether you're building customer support chatbots, internal knowledge assistants, or specialized research tools, OCI Generative AI Agents provides the foundation for enterprise-grade AI applications—without the complexity of traditional RAG implementations.

OCI Generative AI and LangChain: Building Enterprise AI Applications with Oracle

Mon, 19 Jan 2026 17:14:03 +0000

Integrating Oracle Cloud Infrastructure (OCI) Generative AI with LangChain unlocks powerful capabilities for building enterprise-grade AI applications. This comprehensive guide explores the official integration, demonstrates practical implementations, and shows how Oracle Database 23ai complements LangChain workflows for complete end-to-end AI solutions.

What is LangChain?

LangChain is a powerful, open-source framework for developing applications powered by large language models (LLMs). It provides essential tools for managing workflows, maintaining context, and integrating with external systems, streamlining the development of AI-powered applications.

Key Capabilities:

LangChain acts as a bridge, connecting LLMs with other data sources and tools, dramatically simplifying application development. The framework enables developers to:

Build chatbots and conversational AI
Create autonomous agents
Implement Retrieval-Augmented Generation (RAG) workflows
Manage prompt templates and chains
Maintain conversation memory
Integrate with vector databases and external APIs

Language Support:

LangChain primarily supports Python but also has libraries for JavaScript, TypeScript, and Java (through LangChain4j).

OCI Generative AI Integration with LangChain

Oracle Cloud Infrastructure Generative AI has an official integration with LangChain, providing native support through wrapper classes that make OCI's LLMs and embedding models easily accessible within LangChain applications.

The Official langchain-oci Package

As of 2025, Oracle maintains the official langchain-oci package that replaces the earlier community integrations.

Important Migration Note: The OCI integrations available in langchain-community are now deprecated. All future development, bug fixes, and feature enhancements are hosted in the new langchain-oci package.

Installation:

pip install langchain-oci

This package provides:

ChatOCIGenAI: Chat models from OCI Generative AI
OCIGenAI: Text generation from OCI Generative AI
OCIGenAIEmbeddings: Embedding models from OCI Generative AI
ChatOCIModelDeployment: Integration with OCI Data Science model endpoints
Oracle AI Vector Search: Native vector store integration

Available Model Classes

1. ChatOCIGenAI - Conversational AI

The primary class for building chat applications with OCI Generative AI models.

from langchain_oci import ChatOCIGenAI

llm = ChatOCIGenAI(
    model_id="meta.llama-4-maverick-17b-128e-instruct-fp8",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="MY_COMPARTMENT_ID",
    model_kwargs={"temperature": 0, "max_tokens": 500},
    auth_profile="MY_AUTH_PROFILE",
    is_stream=True
)

response = llm.invoke("Sing a ballad of LangChain.")
print(response.content)

Supported Features:

Multi-turn conversations with context retention
Streaming responses for real-time output
Structured output generation
Tool calling and function execution
Parallel tool calling (Llama 4+ models only)

2. OCIGenAI - Text Generation

For simpler text generation tasks without conversational context.

from langchain_oci import OCIGenAI

llm = OCIGenAI()
response = llm.invoke("The meaning of life is")
print(response)

3. OCIGenAIEmbeddings - Vector Representations

Convert text into semantic embeddings for similarity search and RAG applications.

from langchain_oci import OCIGenAIEmbeddings

embeddings = OCIGenAIEmbeddings(
    model_id="cohere.embed-v4.0",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="compartment_id"
)

# Single query embedding
query = "This is a query in English."
response = embeddings.embed_query(query)

# Multiple document embeddings
documents = ["This is a sample document", "and here is another one"]
response = embeddings.embed_documents(documents)

Authentication Methods:

LangChain OCI integration supports standard SDK authentication:

API Key (default)
Security Token
Instance Principal
Resource Principal

Advanced Features

Structured Output:

ChatOCIGenAI supports structured output generation using function calling, JSON schema, or JSON mode.

from langchain_oci import ChatOCIGenAI

llm = ChatOCIGenAI(
    model_id="meta.llama-4-maverick-17b-128e-instruct-fp8",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="MY_COMPARTMENT_ID"
)

# Define your structured output schema
structured_llm = llm.with_structured_output(YourSchema)

Parallel Tool Calling:

Llama 4+ models support parallel tool calling, allowing multiple tools to execute simultaneously.

llm_with_tools = llm.bind_tools(
    [get_weather, calculate_tip, get_population],
    parallel_tool_calls=True  # Tools can execute simultaneously
)

Note: Llama 3.x (including 3.3) and Cohere models will raise an error if parallel tool calling is used.

LangChain Prompt Templates

LangChain prompts can be created using two types of prompt templates, each optimized for different use cases.

1. String Prompt Template

Outputs a simple string, ideal for basic text generation tasks.

from langchain_core.prompts import PromptTemplate

template = """
You are a helpful assistant. Translate the following text to French:
{text}
"""

prompt = PromptTemplate.from_template(template)

2. Chat Prompt Template

Supports a list of messages with different roles (system, user, assistant), perfect for conversational AI.

from langchain_core.prompts import ChatPromptTemplate

template = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant specialized in {domain}."),
    ("user", "{question}")
])

prompt = template.invoke({
    "domain": "Italian cuisine",
    "question": "What are the best ingredients for carbonara?"
})

Message Types:

System messages: Set behavior and context
User messages: User queries and inputs
Assistant messages: Previous AI responses (for context)
Function messages: Tool/function call results

Creating Chains with LCEL

LangChain Expression Language (LCEL) enables declarative chain creation, offering a more concise and composable approach than traditional Python classes.

Declarative Chains with LCEL

LCEL uses the pipe operator (|) to chain components together.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_oci import ChatOCIGenAI

# Define components
prompt = ChatPromptTemplate.from_template(
    "If the text is a question, first translate the question and then "
    "answer the question in French.: {text}"
)

llm = ChatOCIGenAI(
    model_id="meta.llama-4-scout-17b-16e-instruct",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="<sandbox-compartment>",
    model_kwargs={"temperature": 0, "max_tokens": 500}
)

output_parser = StrOutputParser()

# Create chain using LCEL
chain = prompt | llm | output_parser

# Execute chain
result = chain.invoke({"text": "What are the four seasons in a year?"})
print(result)

Output:

La traduction de la question est : Quelles sont les quatre saisons dans une année?
Les quatre saisons dans une année sont : le printemps, l'été, l'automne et l'hiver.

Traditional Python Classes

Alternatively, use Python classes like LLMChain for explicit chain construction.

from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate
from langchain_oci import ChatOCIGenAI

llm = ChatOCIGenAI(model_id="meta.llama-4-scout-17b-16e-instruct", ...)
prompt = PromptTemplate.from_template("Translate to French: {text}")

chain = LLMChain(llm=llm, prompt=prompt)
result = chain.run(text="Hello, world!")

LCEL vs Python Classes:

LCEL: More concise, composable, modern approach (recommended)
Python Classes: More explicit, traditional, verbose

LangChain Memory: Maintaining Conversation Context

LangChain memory provides the ability to store information about past interactions, crucial for building conversational applications.

Memory Types

Conversation Buffer Memory:
Stores the entire conversation history.

from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory()
memory.save_context({"input": "Hi"}, {"output": "Hello! How can I help?"})
memory.save_context({"input": "What's the weather?"}, {"output": "It's sunny!"})

print(memory.load_memory_variables({}))

Conversation Buffer Window Memory:
Maintains only the last K interactions to manage token limits.

from langchain.memory import ConversationBufferWindowMemory

memory = ConversationBufferWindowMemory(k=2)  # Keep last 2 exchanges

Conversation Summary Memory:
Uses an LLM to summarize conversation history, maintaining context while reducing tokens.

from langchain.memory import ConversationSummaryMemory

memory = ConversationSummaryMemory(llm=llm)

Oracle Database 23ai as a Vector Store

Oracle Database 23ai can be used as a powerful vector store to generate and manage embeddings, seamlessly integrating with LangChain for RAG applications.

Why Oracle Database 23ai for Vector Storage?

Oracle Database 23ai supports native AI Vector Search, enabling:

Semantic similarity search on text, documents, images, and unstructured data
Combined relational and vector queries in a single system
Pre-filtering, in-filtering, and post-filtering for enhanced search capabilities
Multiple distance metrics: Cosine, Euclidean, dot product
HNSW indexing for high-performance similarity search
Integration with existing enterprise data without data duplication

LangChain Oracle Vector Store Integration

Oracle AI Vector Search integrates with LangChain at various levels, providing document loaders, text splitters, embeddings, and vector stores.

Installation:

pip install langchain-oracle

Basic Vector Store Usage:

from langchain_community.vectorstores import OracleVS
from langchain_oracle.embeddings import OCIGenAIEmbeddings
from langchain.schema import Document
import oracledb

# Connect to Oracle Database 23ai
conn = oracledb.connect(
    user="username",
    password="password",
    dsn="localhost/FREEPDB1"
)

# Create embedding model
embeddings = OCIGenAIEmbeddings(
    model_id="cohere.embed-v4.0",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="compartment_id"
)

# Prepare documents
documents = [
    Document(
        page_content="LangChain is a framework for LLM applications",
        metadata={"id": "doc1", "source": "documentation"}
    ),
    Document(
        page_content="Oracle Database 23ai supports vector search natively",
        metadata={"id": "doc2", "source": "announcement"}
    )
]

# Create vector store
vector_store = OracleVS.from_documents(
    documents,
    embeddings,
    client=conn,
    table_name="LANGCHAIN_VECTORS",
    distance_strategy=DistanceStrategy.COSINE
)

Creating HNSW Index for Performance

import oraclevs

oraclevs.create_index(
    conn,
    vector_store,
    params={
        "idx_name": "hnsw_idx1",
        "idx_type": "HNSW",
        "accuracy": 97,  # Target accuracy percentage
        "parallel": 8     # Number of parallel workers
    }
)

Similarity Search

# Perform similarity search
query = "What is LangChain?"
results = vector_store.similarity_search(query, k=3)

for doc in results:
    print(f"Content: {doc.page_content}")
    print(f"Metadata: {doc.metadata}\n")

Accessing OCI Generative AI from Oracle Database

Oracle Database 23ai can directly access OCI Generative AI services through built-in APIs and Select AI capabilities.

Using DBMS_CLOUD for REST API Access

You can access OCI Generative AI service using the DBMS_CLOUD REST API directly from SQL.

-- Create credential for OCI authentication
BEGIN
  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'OCI_CRED',
    username => 'ocid1.user.oc1...',
    password => '<your-auth-token>'
  );
END;
/

-- Call OCI Generative AI endpoint
DECLARE
  l_response CLOB;
BEGIN
  l_response := DBMS_CLOUD.SEND_REQUEST(
    credential_name => 'OCI_CRED',
    uri => 'https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/...',
    method => 'POST',
    body => '{
      "model": "cohere.command-r-plus",
      "prompt": "Explain quantum computing",
      "max_tokens": 500
    }'
  );
  
  DBMS_OUTPUT.PUT_LINE(l_response);
END;
/

Select AI: Natural Language to SQL

Select AI enables SQL access to generative AI using Large Language Models (LLMs) and embedding models, including support for natural language to SQL query generation and retrieval-augmented generation.

How Select AI Works

Select AI translates natural language questions into SQL queries that the database understands, then uses LLMs trained to infer user intent to interpret queries and return desired answers.

Key Capabilities:

Select AI automatically builds and maintains a vector index using metadata for applicable objects in your schema, using semantic similarity search to choose the most relevant objects based on user prompts.

Natural language to SQL (NL2SQL / text-to-SQL) query generation
Automatic metadata augmentation with table/column comments, constraints, annotations
Semantic similarity search to select relevant schema objects
RAG support for context-augmented responses
Multi-dialect support including Oracle SQL and SQLite

Setting Up Select AI

-- Create credential for LLM provider
BEGIN
  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'COHERE_CRED',
    username => 'COHERE',
    password => '<your-api-key>'
  );
END;
/

-- Create AI profile
BEGIN
  DBMS_CLOUD_AI.CREATE_PROFILE(
    profile_name => 'COHERE',
    attributes => '{
      "provider": "cohere",
      "credential_name": "COHERE_CRED",
      "model": "cohere.command-a-03-2025",
      "object_list": [{"owner": "ADB_USER"}],
      "max_tokens": 512,
      "temperature": 0.5,
      "comments": true,
      "constraints": true
    }'
  );
END;
/

Using Select AI

Run SQL from Natural Language:

SELECT AI What are the top 5 products by revenue this quarter?;

Show Generated SQL:

SELECT AI showsql What are the top 5 products by revenue this quarter?;

Explain SQL in Natural Language:

SELECT AI explainsql SELECT * FROM sales WHERE region = 'West';

Chat Mode:

SELECT AI chat Who won the 2024 World Series?;

Select AI with RAG

Select AI supports RAG using Oracle AI Vector Search, integrating vector database content to deliver more accurate results from LLMs through semantic similarity search.

Select AI RAG automates the orchestration steps of transforming prompts into vectors and performing semantic similarity search on vector database content, making it easy to use LLMs with your private data using natural language prompts.

Complete RAG Application Example

Here's a complete example combining OCI Generative AI, LangChain, and Oracle Database 23ai:

from langchain_oci import ChatOCIGenAI, OCIGenAIEmbeddings
from langchain_community.vectorstores import OracleVS
from langchain_core.prompts import ChatPromptTemplate
from langchain.schema import Document
from langchain_core.output_parsers import StrOutputParser
import oracledb

# 1. Connect to Oracle Database 23ai
conn = oracledb.connect(
    user="username",
    password="password",
    dsn="localhost/FREEPDB1"
)

# 2. Initialize embedding model
embeddings = OCIGenAIEmbeddings(
    model_id="cohere.embed-v4.0",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="compartment_id"
)

# 3. Load documents into vector store
documents = [
    Document(page_content="Your document content here", metadata={"source": "doc1"}),
    # ... more documents
]

vector_store = OracleVS.from_documents(
    documents,
    embeddings,
    client=conn,
    table_name="KNOWLEDGE_BASE"
)

# 4. Initialize LLM
llm = ChatOCIGenAI(
    model_id="meta.llama-4-scout-17b-16e-instruct",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="compartment_id",
    model_kwargs={"temperature": 0.3, "max_tokens": 1000}
)

# 5. Create RAG chain
template = """
Answer the question based on the following context:

Context: {context}

Question: {question}

Answer:
"""

prompt = ChatPromptTemplate.from_template(template)

# 6. Query function
def ask_question(question: str):
    # Retrieve relevant documents
    docs = vector_store.similarity_search(question, k=3)
    context = "\n\n".join([doc.page_content for doc in docs])
    
    # Create and execute chain
    chain = prompt | llm | StrOutputParser()
    response = chain.invoke({"context": context, "question": question})
    
    return response

# 7. Use the RAG system
answer = ask_question("What are the benefits of Oracle Database 23ai?")
print(answer)

Best Practices for OCI + LangChain Applications

1. Choose the Right Components

ChatOCIGenAI for conversational applications
OCIGenAI for simple text generation
OCIGenAIEmbeddings for semantic search and RAG
Oracle Database 23ai as vector store for enterprise data

2. Optimize Embedding Storage

Use appropriate distance metrics (cosine for text similarity)
Create HNSW indexes for large datasets
Leverage metadata filtering for refined searches
Batch document processing for efficiency

3. Manage Memory Effectively

Use ConversationBufferWindowMemory for long conversations
Implement ConversationSummaryMemory for token efficiency
Clear memory when starting new conversation threads

4. Leverage LCEL for Composability

Build modular, reusable chains
Combine multiple chains for complex workflows
Use parallel execution where appropriate

5. Implement Proper Error Handling

try:
    response = chain.invoke(input_data)
except Exception as e:
    logging.error(f"Chain execution failed: {e}")
    # Implement fallback behavior

6. Monitor and Optimize Costs

Cache frequently accessed embeddings
Use appropriate model sizes (Scout vs Maverick)
Implement rate limiting for user requests
Monitor token usage across applications

The integration of OCI Generative AI with LangChain provides a powerful, enterprise-ready platform for building AI applications. Key takeaways:

Official Integration:

langchain-oci package provides native support for OCI Generative AI
Supports chat models, text generation, and embeddings
Compatible with OCI Data Science model deployments

Prompt Engineering:

String and Chat prompt templates for different use cases
LCEL for declarative chain creation
Memory management for conversational context

Oracle Database 23ai:

Native vector store with LangChain integration
Combines relational and vector search in one system
Select AI for natural language to SQL generation

Complete AI Stack:
By combining OCI Generative AI (LLMs + embeddings), LangChain (orchestration), and Oracle Database 23ai (vector store + Select AI), you can build sophisticated RAG applications, conversational agents, and enterprise AI solutions with security, scalability, and performance.

OCI Generative AI Security: Dedicated GPUs, RDMA Networks, and Enterprise-Grade Data Protection

Sat, 17 Jan 2026 18:12:41 +0000

Security and privacy of customer workloads is an essential design tenet for Oracle Cloud Infrastructure (OCI) Generative AI. In an era where data breaches and AI security vulnerabilities dominate headlines, OCI has architected a comprehensive security framework that ensures enterprise-grade protection for AI workloads. This deep dive explores the multi-layered security approach that makes OCI Generative AI one of the most secure platforms for enterprise AI deployments.

The Foundation: GPU Isolation and Dedicated Infrastructure

Dedicated GPU Allocation

GPUs allocated for a customer's generative AI tasks are pooled within a dedicated RDMA network, ensuring they are exclusively allocated to a single customer and not shared with others. This fundamental isolation guarantees that customer data remains secure and inaccessible to unauthorized parties.

Key Isolation Characteristics:

Physical Separation: Each customer's GPU cluster operates on dedicated hardware, preventing any cross-contamination or data leakage between tenants.

Network Isolation: RDMA cluster networks with less than 10 microsecond latency provide ultra-high-bandwidth connections (1.6 TB/sec internode bandwidth) while maintaining complete isolation from other customers' workloads.

Compute Isolation: OCI Supercluster can deploy up to 32,768 GPUs per cluster, all dedicated to a single customer with no sharing of computational resources.

RDMA Network Architecture

Remote Direct Memory Access (RDMA) is a critical component of OCI's AI infrastructure, providing both performance and security benefits.

RDMA Technology Overview:

RDMA allows for low-latency connections between nodes and access to GPU memory without involving the CPU. This technology enables:

Ultra-Low Latency: Less than 10 microseconds between nodes
High Bandwidth: 1.6 TB/sec internode bandwidth for massive data transfers
CPU Offload: Direct memory-to-memory transfers without CPU intervention
Security: Dedicated network fabric preventing cross-tenant access

Network Types:

OCI's cluster network uses RDMA over Converged Ethernet Version 2 (RoCE v2) on top of NVIDIA ConnectX-7 network interface cards (NICs) to support high-throughput and latency-sensitive workloads.

For advanced deployments, OCI utilizes NVIDIA Quantum InfiniBand or NVIDIA Spectrum-X Ethernet with RDMA over Converged Ethernet (RoCE) configuration.

Scale and Performance:

OCI enables customers to cluster up to 4,096 bare metal nodes, each with 8 GPUs, totaling up to 32,768 GPUs. The latest infrastructure scales even further—zettascale OCI Supercluster scales up to 131,072 GPUs, making it the largest hyperscale AI supercomputer in the cloud.

Hardware Security Modules (HSM)

Keys are stored on Hardware Security Modules (HSMs) that meet Federal Information Processing Standards (FIPS) 140-2 Security Level 3 security certification.

HSM Protection Modes:

Software Protection: Keys stored and processed on servers, recommended for most use cases with strong encryption.

HSM Protection: Keys stored on HSMs meeting FIPS 140-2 Security Level 3 certification, recommended for stringent compliance requirements like financial services and healthcare.

Model Endpoint Security

Single-Customer Model Endpoints

For strong data privacy and security, a dedicated GPU cluster only handles fine-tuned models for a single customer. This architectural decision ensures:

Data Isolation: No model from one customer processes data from another customer
Performance Isolation: Resources allocated exclusively for your workloads
Compliance: Easier to meet regulatory requirements with clear tenant boundaries
Security Boundaries: Physical and logical separation of processing environments

Endpoint Access Control

Model endpoints implement multiple layers of access control:

Authentication: API keys, OAuth tokens, or IAM credentials required for all requests
Authorization: Role-Based Access Control (RBAC) determines what actions authenticated users can perform
Network Security: Private endpoints within Virtual Cloud Networks (VCNs), with optional public access through controlled ingress
Rate Limiting: Configurable throttling to prevent abuse and ensure fair resource allocation

Customer Data and Model Isolation

Tenancy-Level Isolation

Customer data access is restricted within a customer's tenancy so that one customer's data cannot be seen by another customer. This fundamental principle manifests across multiple dimensions:

Compartment Isolation:

OCI uses compartments to organize and isolate resources. The architecture implements compartmentalization and private subnets to isolate different operational environments.

Best Practice: Create separate compartments for:

Development environments
Staging/testing
Production workloads
Different business units or projects

Data Separation:

Training data, fine-tuned model weights, embeddings, and inference logs are all stored within customer-controlled storage with strict access controls preventing cross-tenant access.

Workload Isolation:

A single NVIDIA GB200 NVL72 rack can be configured to launch multiple NVLink groupings for smaller workloads and provide strong isolation for efficient workload distribution.

Network-Level Isolation

Private Subnets:

Resources deployed in private subnets within VCNs, preventing direct internet access unless explicitly configured.

Security Lists and Network Security Groups (NSGs):

Stateful firewalls controlling inbound and outbound traffic at the subnet and network interface level.

Service Gateway and NAT Gateway:

Controlled access to Oracle services and internet resources without exposing instances to inbound internet traffic.

Integrated OCI Security Services

OCI Generative AI leverages Oracle's comprehensive security ecosystem to provide defense-in-depth.

OCI Identity and Access Management (IAM)

IAM provides the foundation for authentication and authorization across all OCI services.

Key IAM Capabilities:

User and Group Management: Create users, organize them into groups, and assign permissions based on job functions.

Policies: Stringent access controls using OCI IAM policies enforce least-privilege access.

Example IAM Policy for Generative AI:

Allow group GenAI-Engineers to manage generative-ai-family in compartment GenAI-Prod
Allow group GenAI-Engineers to use virtual-network-family in compartment GenAI-Networking
Allow group Data-Scientists to read generative-ai-endpoints in compartment GenAI-Prod

Identity Providers: Integration with corporate identity systems (SAML, SCIM) for single sign-on.

Multi-Factor Authentication (MFA): Additional security layer requiring multiple forms of verification.

OCI Key Management Service (Vault)

The Key Management Service is critical for protecting data at rest and managing encryption keys.

Vault Architecture:

Oracle Vault is a logical grouping of keys. There are two types of vaults: Private and Virtual, which have different levels of isolation, pricing, and computing.

Key Features:

Centralized Key Management: OCI Key Management provides centralized management of the encryption of your data.

Customer-Managed Keys (CMK): Customers create and control their own encryption keys, maintaining full ownership.

Oracle-Managed Keys: Default encryption with Oracle-managed keys that are automatically rotated.

Key Rotation: Each master encryption key (MEK) is automatically assigned a key version. When you rotate a key, the Vault service generates a new key version.

Integration with Generative AI:

OCI Generative AI uses Key Management Service to:

Store Base Model Keys Securely: Encryption keys for foundational models are managed through Vault
Encrypt Fine-Tuned Models: Custom model weights encrypted with customer-controlled keys
Protect Training Data: Datasets used for fine-tuning encrypted before storage
Secure Inference Data: Input and output data encrypted in transit and at rest

OCI Object Storage Security

Object Storage buckets store critical AI assets including training data, model weights, and embeddings.

Default Encryption:

OCI encrypts all objects by default with Oracle-managed keys, which are periodically rotated. This ensures data is protected even if customers don't explicitly configure encryption.

Customer-Managed Encryption:

OCI provides options for customers to use their own keys for more security. Using IAM policies that authenticate users performing tasks, a best practice is creating and periodically rotating encryption keys in the Vault service to protect resources in Object Storage.

Encryption at Multiple Layers:

Object-Level Encryption: Each object encrypted with a unique Data Encryption Key (DEK)
Bucket-Level Encryption: Entire buckets can be encrypted with a specific Master Encryption Key (MEK)
Client-Side Encryption: Customers can encrypt data before sending it to Oracle Object Storage for additional control

Required IAM Policies:

To use customer-managed keys with Object Storage, create a policy like:

allow service objectstorage-<region_name> to use keys in compartment <compartment_name>

Access Control:

Pre-Authenticated Requests (PARs): Time-bound access to objects without requiring IAM credentials
Bucket Visibility: Private by default, with explicit configuration required for public access
Versioning: Track and restore previous versions of objects

Security Best Practices:

Restrict BUCKET_UPDATE permission to a minimal set of IAM groups to minimize the possibility of existing buckets being made public inadvertently or maliciously.

Cloud Guard: Active Threat Detection

Comprehensive logging and audit trails maintained for all significant operations.

Cloud Guard continuously monitors your OCI environment for security weaknesses and threats.

Detector Rules for Object Storage:

Cloud Guard includes detector rules for Object Storage to identify public buckets, unencrypted data, and other security issues.

Capabilities:

Automated Detection: Identifies misconfigurations and security threats in real-time
Responder Actions: Automatically remediate issues or notify administrators
Problem Dashboard: Centralized view of all detected security problems
Integration: Works across all OCI services including Generative AI resources

Data Protection Throughout the AI Lifecycle

Training Data Security

Ingestion:

Data uploaded to Object Storage with automatic encryption
Customer-managed keys available for additional control
Access restricted via IAM policies

Processing:

Training occurs on dedicated GPUs isolated within customer's RDMA network
No data sharing between customers
Temporary data purged after training completion

Storage:

Fine-tuned model weights stored in Object Storage buckets
Encrypted by default and managed by Key Management Service
Version control for model iterations

Inference Data Security

Input Processing:

Requests authenticated via IAM or API keys
Data encrypted in transit using TLS 1.2+
Processed on dedicated endpoints with no cross-tenant access

Output Generation:

Responses encrypted in transit
Optional logging with encryption at rest
Retention policies configurable per use case

Monitoring:

Continuous monitoring of resource usage to proactively mitigate contention
Audit logs for all API calls
Integration with OCI Logging Analytics

Advanced Security Features

Security Zones

Security Zones provide compartment-associated Oracle-defined recipes of security policies based on best practices.

Enforced Policies:

No Public Access: Resources cannot be accessible from public internet
Customer-Managed Keys: Data encryption enforced using Customer-Managed Keys for block volumes, boot volumes, and Object Storage buckets
Deny Public Buckets: Public access to Object Storage buckets is denied to prevent accidental exposure

Use Case:

Deploy Generative AI workloads in Security Zones to automatically enforce security best practices and maintain compliance.

Compliance and Certifications

OCI maintains extensive compliance certifications relevant to AI workloads:

Regional Compliance:

GDPR: European data protection requirements
HIPAA: Healthcare data in the United States
FedRAMP: U.S. government cloud security
ISO 27001/27017/27018: International security standards

Industry-Specific:

PCI DSS: Payment card industry
SOC 1/2/3: Service organization controls
FIPS 140-2 Level 3: Cryptographic module security

Sovereign Cloud Options

Oracle's distributed cloud, AI infrastructure, and generative AI services enable governments and enterprises to deploy AI factories that run cloud services locally and within a country's secure premises with operational controls supporting sovereign goals.

Available Options:

OCI Dedicated Region: Entire cloud region operated exclusively for a single customer
Oracle Alloy: Cloud platform for partners to deliver sovereign cloud
Oracle EU Sovereign Cloud: European data residency and operational sovereignty
Oracle Government Cloud: FedRAMP-authorized regions for U.S. government

Best Practices for Secure Generative AI Deployments

1. Implement Least-Privilege Access

Principle: Grant only the minimum permissions necessary for each role.

Implementation:

# Development team - read-only on production
Allow group GenAI-Developers to read generative-ai-family in compartment GenAI-Prod

# Production team - full management
Allow group GenAI-Production to manage generative-ai-family in compartment GenAI-Prod

2. Use Customer-Managed Keys

Benefit: Full control over encryption key lifecycle and usage.

Steps:

Create Vault in Key Management Service
Generate or import master encryption key
Create IAM policy allowing services to use key
Configure buckets and resources to use customer key

3. Enable Comprehensive Logging

Audit Logs: Enable audit logging for all API calls to Generative AI services
Flow Logs: Monitor network traffic to/from GPU clusters
Application Logs: Capture inference requests and responses for analysis

4. Leverage Network Isolation

Private Endpoints: Deploy model endpoints in private subnets
Service Gateway: Access Oracle services without internet exposure
Bastion Hosts: Secure administrative access to resources

5. Implement Defense-in-Depth

Multiple Layers:

Network security (NSGs, Security Lists)
Identity security (IAM, MFA)
Data security (encryption at rest and in transit)
Application security (input validation, rate limiting)
Monitoring security (Cloud Guard, Logging Analytics)

6. Regular Security Reviews

Periodic Assessments:

Review IAM policies quarterly
Rotate encryption keys according to policy
Audit user access and remove unused accounts
Update network security rules based on traffic patterns

7. Secure Development Practices

The architecture includes a CI/CD pipeline for promoting models from the playground environment to production.

Separation of Environments:

Development: Unrestricted experimentation with synthetic data
Staging: Production-like with access controls
Production: Maximum security with customer data

Monitoring and Incident Response

Real-Time Monitoring

Metrics to Track:

API request rates and latency
Authentication failures
Unusual access patterns
Resource utilization anomalies

Alerting:
Configure Cloud Events to trigger notifications for:

IAM policy changes
Network security group modifications
Encryption key access
Dedicated cluster scaling events

Incident Response

Preparation:

Define incident response procedures
Establish communication channels
Designate response team members

Detection:

Cloud Guard active monitoring
Log analysis for anomalies
User-reported issues

Containment:

Isolate affected resources
Revoke compromised credentials
Block suspicious network traffic

Recovery:

Restore from backups if necessary
Rotate encryption keys
Update security configurations

Lessons Learned:

Document incident timeline
Identify root causes
Update procedures and controls

Future Security Enhancements

Oracle continues investing in security capabilities:

AI-Powered Security:
Machine learning models detecting anomalous behavior in API usage patterns and access patterns.

Zero Trust Architecture:
Moving toward continuous verification of all access requests regardless of source.

Confidential Computing:
NVIDIA BlueField-3 DPUs accelerate networking, storage, and security workloads, enabling hardware-based isolation for sensitive computations.

Enhanced Compliance:
Expanding certifications and attestations for emerging regulatory requirements.

OCI Generative AI security is built on a comprehensive, multi-layered approach that addresses the unique challenges of enterprise AI deployments:

Infrastructure Security:

Dedicated GPUs pooled within RDMA networks exclusively for single customers
Dedicated GPU clusters handling only customer's base and fine-tuned models
Ultra-low latency RDMA networking with complete tenant isolation

Data Security:

Customer data access restricted within tenancy
Object Storage encrypted by default with periodic key rotation
Customer-managed keys for maximum control

Identity and Access:

Comprehensive IAM with fine-grained policies
Integration with enterprise identity providers
Multi-factor authentication support

Compliance:

FIPS 140-2 Level 3 certified HSMs
Sovereign cloud options for data residency
Extensive regulatory certifications

Security and privacy of customer workloads is truly an essential design tenet at OCI. By combining dedicated infrastructure, comprehensive encryption, identity controls, and continuous monitoring, OCI Generative AI provides enterprise-grade security that enables organizations to confidently deploy AI at scale.

How does your organization approach AI security? What security features are most critical for your use cases? Share your experiences in the comments

OCI Generative AI: Dedicated Cluster Sizing, Pricing, and Fine-Tuning Deep Dive

Thu, 15 Jan 2026 07:33:12 +0000

Understanding how to properly size, price, and configure dedicated AI clusters is crucial for cost-effective deployment of custom LLMs on Oracle Cloud Infrastructure. This comprehensive guide walks you through cluster types, pricing calculations, fine-tuning configurations, and how to interpret your results.

Dedicated AI Cluster Unit Types

OCI Generative AI provides several dedicated cluster unit types, each optimized for specific models and workloads:

Available Unit Types

Small Cohere Dedicated (V2)

Unit identifier: SMALL_COHERE_V2
Suitable for: Cohere Command R models
Use case: Fine-tuning and hosting smaller Cohere models

Large Cohere Dedicated

Unit identifier: LARGE_COHERE
Suitable for: Cohere Command R+ models
Use case: Larger Cohere models requiring more compute

Embed Cohere Dedicated

Unit identifier: EMBED_COHERE
Suitable for: Cohere embedding models (English V3, Multilingual V3, Light variants)
Use case: Generating embeddings at scale

Large Meta Dedicated

Unit identifier: LARGE_META (or specific like Llama2-70-count)
Suitable for: Meta Llama models (70B, 405B parameters)
Use case: Hosting and fine-tuning large Meta Llama models

Understanding Unit Requirements

Different models require different numbers of units for fine-tuning and hosting. Let's use the Cohere Command R 08-2024 model as an example:

Fine-Tuning Requirements:
To fine-tune a Cohere Command R 08-2024 model, you need a preset value of 8 Small Cohere V2 units. This is non-negotiable—the fine-tuning cluster is automatically sized for optimal training performance.

Hosting Requirements:
To host the fine-tuned Command R 08-2024 model, you need a minimum of 1 Small Cohere V2 unit. You can add more units (up to 3 or more) to handle higher call volumes.

Total for Complete Workflow:
If you want to both fine-tune and host a Command R model, you'll need:

8 Small Cohere V2 units (fine-tuning cluster)
1+ Small Cohere V2 units (hosting cluster)
Total: Minimum 9 Small Cohere V2 units

Commitment Requirements and Billing

Understanding OCI's commitment model is essential for cost planning.

Minimum Commitments

Hosting Clusters: Require a minimum commitment of 744 unit-hours per cluster

744 hours = 31 days (one month)
This is your minimum billing period for hosting

Fine-Tuning Clusters: Require a minimum of 1 unit-hour

You pay only for the actual time used
Much more flexible than hosting commitments

Billing Mechanics

Character-Based Billing (On-Demand):
A transaction is one character. 10,000 transactions = 10,000 characters. Foundational models on-demand are charged per character for prompts and responses (except embeddings, where responses aren't counted).

Unit-Hour Billing (Dedicated Clusters):
If you're hosting foundational models or fine-tuning them on dedicated AI clusters, you're charged by the unit-hour rather than by transaction.

Real-World Pricing Example: Ryan's Fine-Tuning Journey

Let's walk through a complete pricing scenario based on Bob's example from Oracle documentation, adapted for our Ryan scenario:

Ryan's Requirements

Ryan wants to:

Fine-tune a Cohere Command R 08-2024 model
Host the fine-tuned model for production use
Create a new fine-tuned version weekly (4 times per month)

Step-by-Step Cost Calculation

Step 1: Fine-Tuning Cluster Costs

Ryan creates a fine-tuning dedicated AI cluster with the preset value of 8 Small Cohere V2 units. The fine-tuning job takes 5 hours to complete. Ryan creates a fine-tuning cluster every week.

Monthly Fine-Tuning Calculation:

Fine-tuning sessions per month: 4 (weekly)
Units per session: 8 Small Cohere V2
Hours per session: 5 hours
Total unit-hours: 4 sessions × 8 units × 5 hours = 160 unit-hours

Monthly fine-tuning cost = 160 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Step 2: Hosting Cluster Costs

For hosting, dedicated AI clusters require a minimum commitment of 744 unit-hours per cluster.

Let's assume Ryan uses 1 Small Cohere V2 unit for hosting:

Hosting units: 1 Small Cohere V2
Hours per month: 744 (minimum commitment)
Total unit-hours: 1 unit × 744 hours = 744 unit-hours

Monthly hosting cost = 744 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Step 3: Total Monthly Cost

Total monthly cost = Fine-tuning cost + Hosting cost
Total monthly cost = (160 + 744) unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>
Total monthly cost = 904 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Scaling Hosting for Higher Traffic

If Ryan decides to buy three units of Small Cohere V2 to handle higher call volume, the hosting calculation becomes:

Hosting units: 3 Small Cohere V2
Hours per month: 744 (minimum commitment applies per unit)
Total unit-hours: 3 units × 744 hours = 2,232 unit-hours

Monthly hosting cost = 2,232 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Fine-Tuning Configuration Parameters

Understanding fine-tuning hyperparameters is crucial for achieving optimal model performance.

Training Methods

OCI Generative AI supports two main parameter-efficient fine-tuning methods:

T-Few (Training Few):

Parameter-efficient method for Cohere models
Selectively updates only a fraction of model weights
Recommended for small datasets (<100,000 samples)
Ideal for changing instruction-following behavior

LoRA (Low-Rank Adaptation):

Parameter-efficient method for Llama models
Adds trainable adapter matrices while freezing base model
Balances efficiency with performance
Suitable for most fine-tuning scenarios

Hyperparameters Explained

OCI Generative AI fine-tunes each base model using hyperparameters based on the pretrained base model. Start training with default hyperparameter values.

Training Epochs Total:
An epoch is one complete pass through the entire training dataset. The number of epochs determines how many times the model sees all training examples.

Too few epochs: Model underfits, doesn't learn patterns well
Too many epochs: Model overfits, memorizes training data instead of generalizing
Optimal range: Typically 3-10 epochs depending on dataset size

Learning Rate:
The learning rate influences how much model parameters are adjusted in each iteration. A lower learning rate means smaller changes to parameter estimates, requiring longer training time but potentially achieving minimum loss.

Too high: Model may overshoot optimal parameters, fail to converge
Too low: Training takes excessively long, may get stuck in local minima
Typical range: 1e-5 to 1e-3 for fine-tuning

Techniques like learning rate schedules adjust the rate over time, or adaptive algorithms like Adam or RMSprop are commonly used.

Training Batch Size:
Number of training examples processed together before updating model parameters.

Larger batches: More stable gradient estimates, faster training, higher memory usage
Smaller batches: More gradient noise (can help escape local minima), lower memory usage
Typical range: 4-32 for LLM fine-tuning

Early Stopping Patience:
Number of epochs to wait for improvement before stopping training early. Prevents wasting compute on models that have stopped improving.

Example: If patience = 3, training stops if validation loss doesn't improve for 3 consecutive epochs

Early Stopping Threshold:
The minimum improvement required to reset the patience counter. Helps distinguish real improvements from noise.

Example: Threshold = 0.001 means validation loss must improve by at least 0.001 to count as improvement

Log Model Metrics Interval Steps:
Frequency of logging training metrics (loss, accuracy) during training.

Smaller intervals: More detailed monitoring, larger log files
Larger intervals: Less granular view, smaller storage requirements
Typical value: Every 10-100 steps

Hyperparameter Calculation

The model calculates totalTrainingSteps using: totalTrainingSteps = (totalTrainingEpochs × size(trainingDataset)) / trainingBatchSize

Example Calculation:

Training dataset: 1,000 examples
Training epochs: 5
Batch size: 10

Total training steps = (5 × 1,000) / 10 = 500 steps

Understanding Fine-Tuning Results

After training completes, evaluating model performance is critical. In the model's detail page, under Model Performance, check the values for accuracy and loss. If you're not happy with the results, create another model with either a larger dataset or different hyperparameters until performance improves.

Key Metrics Explained

Accuracy:
Accuracy measures the proportion of correct predictions among the total number of cases evaluated. In the context of LLM fine-tuning, accuracy measures when the generated tokens match the annotated tokens in your training data.

Formula:

Accuracy = (Number of correct token predictions) / (Total number of tokens)

Interpretation:

Accuracy = 0.90 (90%): The model predicts the correct token 90% of the time
Higher is better, but perfect accuracy (1.0) often indicates overfitting

Example:
If fine-tuning on a medical terminology task, accuracy of 85% means the model correctly generates medical terms 85% of the time.

Loss:
Loss (also called training loss) measures how wrong the generated outputs of the model are. The loss function quantifies the discrepancy between predicted outputs and actual targets.

Key Characteristics:

Loss decreases as the model improves
Lower loss indicates better fit to training data
Should decrease consistently across epochs (if not, check learning rate)

Common Loss Functions:

Cross-entropy loss: Standard for classification and language modeling
Mean squared error: Less common for LLMs

Interpreting Loss Curves:

Training loss started at step 25 at 2.86 and by step 200, total training and validation loss was insignificant at just 0.4.

Epoch 1: Loss = 2.5
Epoch 2: Loss = 1.8  ← Good! Decreasing
Epoch 3: Loss = 1.3  ← Still improving
Epoch 4: Loss = 1.1  ← Slowing down
Epoch 5: Loss = 1.05 ← Converging

Warning Signs:

Loss increases: Learning rate too high, or data quality issues
Loss plateaus early: Learning rate too low, or model capacity insufficient
Loss fluctuates wildly: Batch size too small, or learning rate too high

Training vs. Validation Loss

Training Loss:
Measured on the data the model is learning from. Always decreases with enough training.

Validation Loss:
Measured on held-out data the model hasn't seen. Indicates generalization ability.

Ideal Pattern:

Epoch  | Training Loss | Validation Loss
-------|---------------|----------------
1      | 2.5          | 2.6
2      | 1.8          | 1.9
3      | 1.3          | 1.4
4      | 1.1          | 1.2

Both decreasing = healthy training

Overfitting Pattern:

Epoch  | Training Loss | Validation Loss
-------|---------------|----------------
1      | 2.5          | 2.6
2      | 1.8          | 1.9
3      | 1.3          | 1.6  ← Warning!
4      | 1.0          | 1.8  ← Overfitting!

Training loss decreases but validation loss increases = model memorizing training data

Additional Evaluation Metrics

While accuracy and loss are primary, consider these secondary metrics:

Perplexity:
Exponential of the loss, more interpretable for language models. Lower is better.

Perplexity = e^(loss)
If loss = 1.5, perplexity = 4.48

F1-Score:
The F1-score is a combined measure of recall and precision, offering a balanced assessment of LLM performance in language tasks.

Particularly useful for classification tasks within fine-tuning (e.g., intent classification, sentiment analysis).

BLEU/ROUGE Scores:
For text generation quality, though traditional metrics like BLEU and ROUGE are considered insufficient for natural conversations.

Best Practices for Cost Optimization

1. Fine-Tuning Strategy

Minimize Training Sessions:

Prepare high-quality datasets before fine-tuning
Test hyperparameters on small subsets first
Batch multiple improvements into single training runs

Optimize Training Time:

Use early stopping to avoid unnecessary epochs
Monitor loss curves—stop if converging
Start with smaller epochs, increase if needed

2. Hosting Strategy

Right-Size Hosting Units:
Start with 1 unit and add more only as call volume demands. Remember the 744-hour minimum commitment.

Consider On-Demand for Low Volume:
If usage is sporadic or low, on-demand pricing might be more cost-effective than dedicated hosting.

Calculation for Break-Even:

On-demand cost per 10k characters: $X
Dedicated cost per unit-hour: $Y
Characters processed per hour: Z

Break-even: Z × $X = $Y

3. Development vs. Production

Development/Testing:

Use on-demand inferencing
Fine-tune infrequently with small datasets
No hosting commitment needed

Production:

Dedicated hosting for consistent performance
Batch fine-tuning updates (weekly/monthly)
Scale hosting units based on actual traffic

4. Monitoring and Optimization

Track Key Metrics:

Cost per fine-tuning session
Cost per inference (characters processed)
Utilization rate of hosting clusters
Model performance metrics (accuracy, loss)

Optimization Opportunities:

Reduce fine-tuning frequency if model performance stable
Adjust hosting capacity based on traffic patterns
Use cached responses for common queries

Common Pitfalls and Solutions

Pitfall 1: Over-Fine-Tuning

Problem: Creating new fine-tuned models too frequently
Cost Impact: Unnecessary compute charges
Solution: Fine-tune only when you have substantial new training data or performance degrades

Pitfall 2: Over-Provisioning Hosting

Problem: Allocating too many hosting units "just in case"
Cost Impact: Paying for unused capacity (744 hours/month minimum per unit)
Solution: Start with minimum units, monitor utilization, scale up as needed

Pitfall 3: Ignoring Model Metrics

Problem: Not evaluating accuracy and loss before deploying
Cost Impact: Poor model performance requiring additional fine-tuning sessions
Solution: Always check Model Performance metrics and iterate if needed

Pitfall 4: Poor Hyperparameter Choices

Problem: Using default hyperparameters without experimentation
Cost Impact: Suboptimal models requiring more training sessions
Solution: Test hyperparameters systematically, document what works

Practical Workflow Example

Here's a complete workflow with cost awareness:

Phase 1: Initial Fine-Tuning (Week 1)

1. Prepare 5,000 training examples
2. Create fine-tuning cluster (8 Small Cohere V2 units)
3. Fine-tune with default hyperparameters (5 hours)
4. Cost: 40 unit-hours × $price
5. Check accuracy: 82% ← Needs improvement

Phase 2: Iteration (Week 2)

1. Increase dataset to 10,000 examples
2. Adjust learning rate from 1e-4 to 5e-5
3. Fine-tune again (6 hours)
4. Cost: 48 unit-hours × $price
5. Check accuracy: 91% ← Acceptable!

Phase 3: Deployment (Week 3-4)

1. Create hosting cluster (1 Small Cohere V2 unit)
2. Deploy fine-tuned model
3. Cost: 744 unit-hours × $price (monthly minimum)
4. Monitor usage and performance

Total Month 1 Cost:

Fine-tuning: (40 + 48) = 88 unit-hours
Hosting: 744 unit-hours (minimum commitment)
Total: 832 unit-hours × $<price per unit-hour>

Conclusion

Effective use of OCI Generative AI dedicated clusters requires understanding three key areas:

1. Sizing: Match cluster unit types and counts to your specific models

Fine-tuning: 8 Small Cohere V2 units for Command R
Hosting: Start with 1 unit, scale based on traffic

2. Pricing: Plan for commitments and optimize costs

Fine-tuning: Pay for actual hours used (minimum 1 hour)
Hosting: 744-hour monthly minimum per cluster
Calculate break-even vs. on-demand based on usage

3. Configuration: Tune hyperparameters and monitor metrics

Start with defaults, iterate systematically
Watch accuracy (higher is better) and loss (lower is better)
Use early stopping to save compute time

By following these guidelines, you can deploy custom fine-tuned models on OCI cost-effectively while maintaining high performance.

Chat and Embedding Models in OCI Generative AI

Tue, 16 Dec 2025 08:45:24 +0000

Oracle Cloud Infrastructure (OCI) Generative AI provides two powerful categories of models: chat models for conversational AI and text generation, and embedding models for semantic understanding and search. Understanding how to configure and optimize these models is essential for building effective AI applications.

Understanding Tokens

Before diving into model parameters, it's crucial to understand that LLMs understand tokens rather than characters. A token can be a word, part of a word, or even punctuation. As a general rule, estimate approximately 4 characters per token—meaning 100 tokens is roughly 60-80 words.

This matters because all LLM limits, costs, and parameters are expressed in tokens, not words or characters.

Chat Models: Configuration Parameters

Chat models in OCI Generative AI offer extensive configuration options to control their behavior and output characteristics.

Max Output Tokens

The maximum number of tokens that you want the model to generate per response. Each model has different maximum output token limits specified in their key features.

Important: Because you're prompting a chat model, the response depends on the prompt and each response doesn't necessarily use up the maximum allocated tokens. Setting this parameter essentially creates a ceiling—the model will stop generating once it hits this limit.

Preamble Override

An initial context or guiding message for a chat model that changes the model's overall chat behavior and conversation style. When you don't provide a preamble, the model uses its default.

Default Preamble for Cohere Command R Family:

You are Command. You are an extremely capable large language model built by Cohere. 
You are given instructions programmatically via an API that you follow to the best 
of your ability.

Custom Preamble Example:

You are a helpful assistant specialized in Italian cuisine. 
Respond with enthusiasm and include regional context when discussing dishes.

With this preamble override, when asked about cannolis, the model responds with regional details about Sicily and specific bakeries like Caffe Sierra and Pasticceria Cappello.

Alternative Approach: You can also include preamble-like instructions directly in your conversation. For example: "Answer the following question in a marketing tone. Where's the best place to go sailing?"

Temperature

Temperature controls the randomness of the output. It's one of the most important parameters for shaping model behavior.

How Temperature Works:

Temperature modulates the probability distribution over words in the vocabulary
Temperature = 0: Makes the model deterministic, always selecting the word with the highest probability (essentially greedy decoding)
Low temperature (< 1.0): Distribution is more peaked around the most likely tokens, producing focused, predictable outputs
Temperature = 1.0: Standard probability distribution with no modification
High temperature (> 1.0): Distribution flattens across all words, increasing randomness and creativity

Best Practices:

Start with temperature set to 0 or less than one, and increase it as you regenerate prompts for more creative output. Be cautious: high temperatures can introduce hallucinations and factually incorrect information.

Use Cases:

Low temperature (0.0-0.3): Code generation, factual Q&A, technical documentation
Medium temperature (0.5-0.7): General chatbots, customer service, balanced responses
High temperature (0.8-1.0+): Creative writing, brainstorming, diverse idea generation

Top-K Sampling

Top-K tells the model to pick the next token from the top K tokens in its list sorted by probability.

A higher value for K generates more random output, making the text sound more natural. The parameter essentially limits the selection pool to the K most likely candidates at each step.

Range:

Minimum: 1 (considers only the single most probable token, essentially deterministic)
Maximum: Vocabulary size (considers all possible tokens)
OCI Default: 0 for Command models, -1 for Llama models (meaning the model considers all tokens and doesn't use this method)

Example: If K=3 and the top three tokens are:

"favorite" (40% probability)
"best" (30% probability)
"preferred" (15% probability)

The model randomly selects from only these three options, weighted by their probabilities.

Top-P (Nucleus Sampling)

Top-P is similar to Top-K but picks from top tokens based on the sum of their probabilities. It's more dynamic than Top-K because the number of tokens considered varies.

Top-P selects tokens whose cumulative probability exceeds a threshold (p), limiting the pool to the smallest set meeting this sum.

How It Works:

Top-P picks the next token based on cumulative probability, ensuring the sum of probabilities is below a certain threshold.

Example: If Top-P = 0.5 and tokens have probabilities:

Token A: 0.3 (30%)
Token B: 0.2 (20%)
Token C: 0.1 (10%)

The model only considers A and B (cumulative 50%), excluding C.

Range:

Minimum: 0.0 (considers only the single most probable token)
Maximum: 1.0 (considers all tokens)

A high top-p value means the model looks at more possible words, even less likely ones, making generated text more diverse.

Practical Guidelines:

Top-P = 0.5: Only considers words adding up to 50% probability—focused responses
Top-P = 0.9: Includes many more words—varied and original responses

The Interaction: Temperature, Top-K, and Top-P

The selection process works in stages: Top-K tokens with highest probabilities are sampled first, then tokens are further filtered based on Top-P, with the final token being selected using temperature.

Decision Framework:

For tasks requiring accuracy and focus (like code completion), use lower temperature, higher Top-P/Top-K, and moderate max tokens
For creative writing, experiment with higher temperature, lower Top-P/Top-K, and larger max token limits

Frequency and Presence Penalties

These parameters are useful if you want to get rid of repetition in your outputs.

Frequency Penalty:
A penalty assigned to a token when that token appears frequently. High penalties encourage fewer repeated tokens and produce more random output.

Presence Penalty:
A penalty assigned to each token when it appears in the output to encourage generating outputs with tokens that haven't been used.

Use Case: If your model keeps repeating the same phrases or concepts, increase these penalties to encourage more diverse vocabulary and concepts.

Seed Parameter (Reproducibility)

Assigning a number for the seed parameter is similar to tagging the request with a number. The LLM aims to generate the same set of tokens for the same integer in consecutive requests.

Important Notes:

Allowed values are integers; assigning large or small seed values doesn't affect the result
Especially useful for debugging and testing
Warning: The seed parameter might not produce the same result long-term because model updates might invalidate the seed

OCI Limits:

API: No maximum value
Console: Maximum value is 9999
Leave blank (Console) or null (API) to disable

Stop Sequences

A sequence of characters—such as a word, phrase, newline (\n), or period—that tells the model when to stop the generated output.

Example: If the stop sequence is a period (.), the model stops generating text once it reaches the end of the first sentence, even if the token limit is much higher.

Use Case: Useful for controlling output length and format, especially when you want responses limited to single sentences or specific sections.

Likelihood Display

Shows how likely a token is to follow the current generated token. Likelihood is defined by a number between -15 and 0, where more negative numbers mean less likely tokens.

Example: The word "favorite" is more likely to be followed by "food" or "book" rather than "zebra."

Important: This parameter doesn't influence the generation process itself but serves as a diagnostic tool to help understand the model's behavior.

Embedding Models: Converting Text to Vectors

Embedding models transform text into numerical representations that capture semantic meaning, enabling AI systems to understand relationships between words, sentences, and documents.

What Are Embeddings?

Embeddings are numerical representations of text converted to number sequences. A piece of text can be a phrase, sentence, or one or more paragraphs.

A vector embedding is a mapping from input (like a word, list of words, or image) into a list of floating-point numbers. That list represents the input in the multidimensional embedding space of the model.

OCI Embedding Model Dimensions

OCI Generative AI embedding models transform each phrase, sentence, or paragraph into an array with 384 (light models) or 1024 numbers, depending on the embedding model selected.

Available Models:

Cohere Embed English V3: 1024 dimensions
Cohere Embed English Light V3: 384 dimensions
Cohere Embed Multilingual V3: 1024 dimensions
Cohere Embed Multilingual Light V3: 384 dimensions
Cohere Embed 4: Latest version supporting text and images

Trade-offs:

Light models (384d): Faster processing, lower memory usage, suitable for most applications
Standard models (1024d): Richer semantic representation, better for complex similarity tasks

Types of Embeddings

Word Embeddings:
Capture properties and relationships of individual words. Word2vec was the most well-known embedding model for a long time, accepting only single words but very good at representing semantic meaning, typically outputting 300-dimensional vectors.

Sentence Embeddings:
Associate every sentence with a vector of numbers, capturing meaning at the sentence level rather than individual words.

Document Embeddings:
Represent entire paragraphs or documents as single vectors, useful for document similarity and classification tasks.

Semantic Similarity: The Core Principle

Embeddings that are numerically similar are also semantically similar. This is the fundamental principle that makes embeddings powerful.

Similar words, documents, or images have vectors that are also similar. For example, "basketball" and "baseball" have embedding vectors much closer to each other than "rainforest".

Measuring Similarity:

The most popular way to compare vectors is cosine similarity, which measures the cosine of the angle between two vectors in multi-dimensional space. The closer the vectors, the smaller the angle.

Other Distance Metrics:

Euclidean distance: Straight-line distance between vectors
Dot product (inner product): Especially effective for unit vectors
Manhattan distance: Sum of absolute differences

Cosine similarity is popular for text embeddings, measuring the angle between vectors. Dot product can also work, especially for unit vectors, offering performance benefits in vector databases.

Embedding Model Evolution

Word2vec was the pioneer, focusing on single words with 300 dimensions—lightweight and great for semantic meaning. Then came OpenAI's text-embedding-ada-002 in 2022, supporting up to 8192 tokens and outputting 1536 dimensions.

2025 State of the Art:

Advances in 2025, driven by LLMs and benchmarks, show transformer-based and instruction-tuned embeddings achieving top performance. Key trends include multilingual models (1000+ languages), domain-specific models (medicine, code), and multimodal models (text-image-audio).

Embedding Use Cases

1. Vector Databases and Semantic Search

The most common use case: Embeddings are mostly used for semantic searches where the search function focuses on the meaning of the text rather than finding results based on keywords.

Workflow:

User query is converted to a vector representation using an embedding model
Query vector is stored/compared in a vector database
Similar vectors are retrieved from private content
Retrieved content is sent to an LLM for response generation
LLM sends response back to the user

We use the embedding model to create vector embeddings for content we want to index. The vector embedding is inserted into the vector database with some reference to the original content.

2. Recommendation Systems

By embedding items (products, articles, movies) and user preferences, systems can recommend semantically similar items even if they don't share obvious keywords.

3. Text Classification and Clustering

Embeddings enable text classification and text clustering by grouping semantically similar items together.

Applications:

Classifying support tickets by department
Categorizing documents by topic
Detecting duplicate or similar content
Sentiment analysis grouping

4. Retrieval-Augmented Generation (RAG)

While the most common scenario for vector databases is retrieval-augmented generation (RAG), other possible use cases include recommendations, anomaly detection, and more.

5. Question Answering Systems

Create chatbots that respond to questions about your own data—for instance, a chatbot responding to employee questions on healthcare coverage. Hundreds of pages of documentation can be split into chunks, converted into embeddings, and searched based on vector similarity.

6. Semantic Caching

Reduce cost and latency of LLMs by caching LLM completions. LLM queries are compared using vector similarity—if a new query is similar enough to a previously cached query, the cached query is returned.

7. LLM Conversation Memory

Persist conversation history with an LLM as embeddings in a vector database. Applications can use vector search to pull relevant history or "memories" into LLM responses.

8. Cross-Lingual and Multimodal Search

We can use vector search across languages since embedding models are frequently trained on more than just English data, and we can use vector search with images if we use multimodal embedding models trained on both text and images.

Vector Databases: Storage and Retrieval

A vector database is a database that can store, manage, retrieve, and compare vectors.

Key Capabilities:

Vector databases are purpose-built to manage vector embeddings, offering data management (insert, delete, update), metadata storage and filtering, scalability with distributed processing, and real-time query performance.

Popular Vector Databases:

Pinecone
Weaviate
Qdrant
Milvus
Chroma
Oracle Database 23ai (with AI Vector Search)
Azure Redis
MyScale

Advanced Features:

Some vector databases can perform hybrid searches by first narrowing results based on characteristics or metadata before conducting vector search—making searches more effective and customizable.

OCI Embedding Model Input Requirements

Input data for text embeddings must meet the following requirements: maximum of 96 inputs allowed for each run, with each input less than 512 tokens for text-only models.

Handling Long Inputs:

If an input exceeds the 512 token limit, you can set the Truncate parameter to Start or End to cut off text to fit within the token limit. Setting it to None will produce an error if inputs exceed limits.

Multimodal Support:

For text and image embed models like Cohere Embed English Image V3, you can add either text or one image only.

Best Practices for OCI Generative AI

Chat Model Optimization

Start Simple: Begin with default parameters, then adjust based on results
Temperature First: Adjust temperature before other sampling parameters
Use Preambles Wisely: Set clear behavioral expectations with custom preambles
Implement Stop Sequences: Control output format and length precisely
Test Reproducibility: Use seed parameter during development for consistent testing
Monitor Penalties: Add frequency/presence penalties if seeing repetition

Embedding Model Selection

Choose Appropriate Dimensions: Use light models (384d) for most applications; standard models (1024d) for complex similarity tasks
Select Right Model: English vs. Multilingual based on your data
Batch Process: Take advantage of the 96-input limit for efficient processing
Index Strategically: Store embeddings in vector databases with appropriate indexes
Measure Appropriately: Use cosine similarity for most text applications

Integration Patterns

OCI Generative AI integrates with LangChain for building context-augmented applications and RAG solutions. It also works with LlamaIndex for accessing pretrained models or creating custom models on dedicated AI clusters.

Understanding chat and embedding models in OCI Generative AI enables you to build sophisticated AI applications. Chat models with properly tuned parameters (temperature, top-k, top-p, penalties) produce high-quality, contextually appropriate responses. Embedding models convert text into semantic vectors that power search, recommendations, and retrieval systems.

Key takeaways:

Tokens are the fundamental unit—everything is measured in tokens, not characters
Temperature controls creativity—low for precision, high for creativity
Sampling parameters work together—Top-K and Top-P filter candidates before temperature applies
Embeddings enable semantic understanding—similar meanings produce similar vectors
Vector databases make it practical—efficient storage and retrieval of embeddings at scale

Whether building chatbots, implementing semantic search, or creating RAG applications, mastering these parameters and concepts is essential for production-grade AI systems on OCI.

OCI Generative AI Service: Enterprise-Grade LLMs on Oracle Cloud

Fri, 12 Dec 2025 09:55:53 +0000

Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service providing enterprises with access to state-of-the-art, customizable large language models through a comprehensive API. OCI positions itself as a neutral, enterprise-focused platform offering unprecedented choice and flexibility in the generative AI space.

What is OCI Generative AI?

OCI Generative AI helps enterprises seamlessly integrate advanced language comprehension capabilities into applications, providing a complete end-to-end platform for building, customizing, and deploying LLM-powered applications at scale.

Key Capabilities:

Access to pretrained foundational models from multiple leading AI providers
Flexible fine-tuning with custom datasets on dedicated infrastructure
Enterprise-grade security, compliance, and data sovereignty
Integration with Oracle's broader AI ecosystem
Support for both on-demand usage and dedicated hosting

Available Models (2025)

Cohere Models:

Command A (03-2025): 256K token context, excels at tool use, agents, RAG, and multilingual tasks
Command R/R+: 128K context for RAG and enterprise use cases
Embed models: English v3.0, Multilingual v3.0, Embed 4

Meta Llama Models:

Llama 4 Maverick: 17B active from ~400B total (mixture-of-experts)
Llama 4 Scout: 17B active from ~109B total
Llama 3.3 (70B), Llama 3.2 Vision (90B, 11B), Llama 3.1 (405B, 70B)

Google Gemini Models (Coming Soon):
Oracle will be the only hyperscaler aside from Google Cloud to offer Gemini as a managed service.

Gemini 2.5 Pro/Flash/Flash-Lite

xAI Grok Models:

Grok 4/Grok 4 Fast, Grok 3 series, Grok Code Fast 1

OpenAI Models:

gpt-oss-120b and gpt-oss-20b

Core Features

1. Pretrained Models

Chat Models: Multi-turn conversational AI
Text Generation: Content creation, code generation, documentation
Embedding Models: Vector representations for semantic search (384-1024 dimensions)
Rerank Models: Relevance scoring for search results

2. Fine-Tuning

Strategies:

T-Few & Vanilla for Cohere models (with layer-specific optimization)
LoRA for Llama 3 models (efficient parameter adaptation)

Customization:

Training epochs, learning rate, batch size
Early stopping controls
JSONL format with prompt/completion pairs

3. Dedicated AI Clusters

GPU-based compute resources exclusive to your tenancy:

Isolated infrastructure for fine-tuning
Private GPUs for hosting with zero-downtime scaling
Multiple cluster sizes (Small/Large/Large Generic 2/4 for 405B models)

4. Deployment Options

On-Demand: Pay-per-character, ideal for experimentation
Dedicated Clusters: Predictable costs for production workloads

5. OCI Generative AI Agents (2024-2025)

Fully managed RAG service combining LLMs with enterprise search.

Agent Hub Features (March 2025):

SQL Tool: Self-correction, multi-dialect support (Oracle SQL, SQLite)
Enhanced RAG Tool: Hybrid search, multi-modal parsing, multi-lingual support (7+ languages)
Integration with OCI Object Storage, OpenSearch, Oracle Database 23ai

Use Cases

Text Generation: Blog posts, marketing copy, documentation
Semantic Search: Intent-based search, recommendations, document retrieval
Document Summarization: Executive summaries, support tickets, research papers
Classification: Support ticket routing, sentiment analysis, intent detection
Question Answering: Intelligent responses from documents and knowledge bases
Enterprise Knowledge Management: RAG-powered customer support

Developer Experience

Access Methods:

OCI Console Playground
REST API
OCI CLI
SDKs (Python, Java, TypeScript, Node.js)

Framework Integration:

LangChain: Prompt templating, memory, chain-of-thought
LlamaIndex: Context-augmented applications, RAG solutions

Tool Use: Chat models can integrate with external tools and APIs for complex queries requiring external data.

Security & Compliance

Key Features:

Data isolation within your tenancy
Encryption at rest and in transit
Fine-grained IAM policies and RBAC
Comprehensive audit trails
Private endpoints and VPN connectivity

Regional Availability:

US (Chicago, Phoenix, Ashburn)
Europe (Frankfurt, London, Amsterdam)
Asia Pacific (Tokyo, Mumbai, Seoul)
Middle East (Dubai, Jeddah)
Latin America (São Paulo)
Sovereign Cloud: Oracle EU Sovereign Cloud for data residency

Advanced Features

Configurable Parameters:

Temperature (0.0-1.0)
Top-k and top-p sampling
Frequency/presence penalties
Seed parameter for reproducibility

Performance Benchmarks:
RAG scenarios with 2,000-token prompts and 200-token responses benchmarked across cluster types.

Pricing

Free Tier: $300 credits for trial
On-Demand: Pay-per-character (input/output)
Dedicated Clusters: Predictable monthly costs

Oracle's AI Ecosystem

Oracle Database 23ai:

Native AI Vector Search
In-database LLM integration
Support for RAG workflows

MySQL HeatWave:

In-database LLMs (HeatWave GenAI)
Automated vector store

Oracle Fusion Applications:
Generative AI embedded across ERP, HCM, SCM, and CX applications.

OCI Data Science:
No-code access to open-source LLMs (Meta, Mistral AI) with Hugging Face Transformers and PyTorch.

Certification

OCI 2025 Generative AI Professional Certification covers:

LLM Fundamentals & Transformer Architecture
Prompt Engineering
Fine-Tuning Techniques
RAG Workflows & Vector Databases
Agent Development

Resources: Free tutorials, Coursera courses, Oracle MyLearn platform

Competitive Advantages

Model Choice: Access to Cohere, Meta, Google, xAI, OpenAI through unified platform
Enterprise-First: Security, compliance, data sovereignty
Cost-Effective: Better price-performance with transparent pricing
Database Integration: Unique integration with Oracle Database 23ai and MySQL HeatWave
Sovereign Cloud: Data residency guarantees

Getting Started

Create OCI account (free tier available)
Navigate to Analytics & AI → Generative AI
Choose approach: Playground, API/SDK, or Fine-tuning
Select model and configure parameters
Start building

Example: Building a Support Chatbot

Ingest documentation into Oracle Database 23ai vector store
Create embeddings using Cohere Embed models
Deploy RAG Agent
Implement chat interface with LangChain
Enable multi-turn conversations with tool calling

OCI Generative AI offers comprehensive, enterprise-focused LLM capabilities addressing real-world needs: security, sovereignty, choice, and integration. Whether building chatbots, implementing semantic search, or developing multi-agent systems, OCI provides the foundation for enterprise-grade AI applications.

Are you using OCI Generative AI? Share your experiences below

Coder Legion - Recent posts tagged oci-genai

OCI Generative AI Agents: Building Enterprise RAG Applications Without Code

What are OCI Generative AI Agents?

Core Capabilities

How Agents Work

Key Agent Components

Knowledge Base

Data Sources

Data Ingestion

Agent Concepts and Terminology

Session

Agent Endpoint

Trace

Citation

Content Moderation

Object Storage Guidelines for GenAI Agents

File Requirements

Chart and Table Guidelines

Hyperlinks and URLs

Planning for Future Data

Advanced Features (March 2025 Release)

Enhanced Response Quality

Hybrid Search

Improved Multi-Modal Parsing

Custom Instructions

Multi-Lingual Support

Multiple Knowledge Bases

Metadata Ingestion & Filtering

Building Your First Agent: Step-by-Step

Prerequisites

Step 1: Create Knowledge Base

Step 2: Monitor Ingestion

Step 3: Create Agent

Step 4: Create Agent Endpoint

Step 5: Test in Playground

Step 6: Integrate via API

Use Cases for OCI Generative AI Agents

1. Customer Support

2. Legal Research

3. Healthcare and Medical Guidance

4. Financial Analysis

5. Travel Planning

6. HR and Internal Knowledge

Best Practices for Production Agents

1. Organize Knowledge Bases Strategically

2. Optimize Document Preparation

3. Leverage Metadata Effectively

4. Implement Content Moderation

5. Monitor and Iterate

6. Plan for Scale

Limitations and Considerations

Regional Availability

Resource Limits

File Type Restrictions

Security and Compliance

Data Privacy

Enterprise Features

Conclusion

OCI Generative AI and LangChain: Building Enterprise AI Applications with Oracle

What is LangChain?

OCI Generative AI Integration with LangChain

The Official langchain-oci Package

Available Model Classes

Advanced Features

LangChain Prompt Templates

1. String Prompt Template

2. Chat Prompt Template

Creating Chains with LCEL

Declarative Chains with LCEL

Traditional Python Classes

LangChain Memory: Maintaining Conversation Context

Memory Types

Oracle Database 23ai as a Vector Store

Why Oracle Database 23ai for Vector Storage?

LangChain Oracle Vector Store Integration

Creating HNSW Index for Performance

Similarity Search

Accessing OCI Generative AI from Oracle Database

Using DBMS_CLOUD for REST API Access

Select AI: Natural Language to SQL