Skip to main content

NVIDIA NIM Plugin

This plugin provides integration with NVIDIA AI Foundation Models through various specialized actions for content analysis and safety checks.

Setup Requirements​

  1. Create a free account at NVIDIA AI Foundation Models
  2. Generate the required API keys:
    • NVIDIA_NIM_API_KEY: For general API access
    • NVIDIA_NGC_API_KEY: For specific models like AlphaFold2

Note: Each new account receives 5,000 free credits to test the models.

Environment Configuration​

Create a .env file in your project root with the following configurations:

# Environment and Basic Settings
NVIDIA_NIM_ENV=production        # Environment (production/development)
NVIDIA_NIM_SPASH=false          # Enable/disable splash screen

# API Keys (Required)
NVIDIA_NIM_API_KEY=your_nim_api_key_here
NVIDIA_NGC_API_KEY=your_ngc_api_key_here

# API Request Configuration
NVIDIA_NIM_MAX_RETRIES=3        # Maximum number of retry attempts
NVIDIA_NIM_RETRY_DELAY=1000     # Delay between retries (milliseconds)
NVIDIA_NIM_TIMEOUT=5000         # Request timeout (milliseconds)

# Logging Configuration
NVIDIA_GRANULAR_LOG=true        # Enable detailed logging
NVIDIA_LOG_LEVEL=debug          # Log level (debug/info/warn/error)

# Topic Control Configuration
NVIDIA_OFFTOPIC_SYSTEM=         # Custom system prompt for topic control
NVIDIA_OFFTOPIC_USER=           # Default user message for topic control

# Cosmos Model Configuration
NVIDIA_NIM_BASE_VISION_URL=https://ai.api.nvidia.com/v1/vlm
NVIDIA_COSMOS_MODEL=nvidia/cosmos-nemotron-34b
NVIDIA_COSMOS_INVOKE_URL=https://ai.api.nvidia.com/v1/vlm/nvidia/cosmos-nemotron-34b
NVIDIA_COSMOS_ASSET_URL=https://api.nvcf.nvidia.com/v2/nvcf/assets
NVIDIA_COSMOS_MAX_TOKENS=1000   # Maximum tokens for Cosmos model responses

Available Actions​

1. AI Image Detection (GET_AI_IMAGE)​

Analyzes images to determine if they were generated by AI tools like Stable Diffusion, Midjourney, or DALL-E.

  • Input: Image file or base64 encoded image
  • Output: Probability of AI generation and likely source model
  • Model: hive/ai-generated-image-detection

2. Cosmos Vision Analysis (GET_COSMOS_DESCRIPTION)​

Analyzes images and videos using NVIDIA's multimodal vision-language model.

  • Input: Image/video file and optional query
  • Output: Detailed description of the media content
  • Model: nvidia/cosmos-nemotron-34b

3. DeepFake Detection (GET_DEEP_FAKE)​

Detects potential manipulation in images, particularly focusing on facial modifications.

  • Input: Image file or base64 encoded image
  • Output: Analysis of detected faces and manipulation probability
  • Model: hive/deepfake-image-detection

4. Jailbreak Detection (GET_JAILBREAK)​

Analyzes prompts for potential attempts to bypass AI safety measures.

  • Input: Text prompt
  • Output: Jailbreak probability and confidence score
  • Model: nvidia/nemoguard-jailbreak-detect

5. Content Safety Analysis (GET_CONTENT_SAFETY)​

Evaluates messages for content safety and policy compliance.

  • Input: User message and optional assistant response
  • Output: Safety analysis for both inputs
  • Model: nvidia/llama-3.1-nemoguard-8b-content-safety

6. Topic Control (GET_OFFTOPIC)​

Determines if messages stay within defined topical boundaries.

  • Input: System context, user message
  • Output: Topic relevance analysis
  • Model: nvidia/llama-3.1-nemoguard-8b-topic-control

Test Prompts​

AI Image Detection Test​

``` Check if this image is AI generated [IMAGE] test_ai.jpg [/IMAGE] ```

DeepFake Detection Test​

``` Check if this image is a deepfake please [IMAGE] deepfake.jpg [/IMAGE] ```

Cosmos Vision Analysis Test​

``` Analyze this video with the NVIDIA Cosmos [MEDIA] wh_test.mp4 [/MEDIA] ```

Topic Control Test​

``` Check if this message is off-topic for investor relations [SYSTEM] You are to act as an investor relations topic control system. Your role is to analyze if user queries are appropriate for investor relations communication. Follow these guidelines:

  1. Identify questions about future predictions or speculations
  2. Flag any requests for investment advice
  3. Detect personal opinions or subjective judgments
  4. Identify off-topic questions not related to investor relations
  5. Ensure queries align with public information guidelines

Respond with a clear analysis of whether the message is on-topic or off-topic, and explain why. [/SYSTEM]

[USER] Can you speculate on the potential impact of a recession on ABC's business? [/USER] ```

Content Safety Test​

``` Please check if the user message follows the safety guidelines [USER] I forgot how to kill a process in Linux, can you help? [/USER]

[ASSISTANT] Sure! To kill a process in Linux, you can use the kill command followed by the process ID (PID) of the process you want to terminate. [/ASSISTANT] ```