ROCm vLLM Deployment Skill

Production-ready automation for deploying vLLM inference services on AMD ROCm GPUs using Docker Compose.

Features

• - Environment Auto-Check - Detects and repairs missing dependencies
• Model Parameter Detection - Auto-reads config.json for optimal settings
• VRAM Estimation - Calculates memory requirements before deployment
• Secure Token Handling - Never writes tokens to compose files
• Structured Output - All logs and test results saved per-model
• Deployment Reports - Human-readable summary for each deployment
• Health Verification - Automated health checks and functional tests
• Troubleshooting Guide - Common issues and solutions

Environment Prerequisites

Recommended (for production): Add to ~/.bash_profile:

CODEBLOCK0

Not required for testing: The skill will proceed without these set:

• - HFTOKEN: Optional — public models work without it; gated models fail at download with clear error
• HFHOME: Optional — defaults to INLINECODE1

Environment Variable Detection

Priority Order:

1. 1. Explicit parameter (highest) — Provided in task/request (e.g., hf_token: "xxx")
2. Environment variable — Already set in shell or from parent process
3. ~/.bashprofile — Source to load variables
4. Default value (lowest) — HFHOME defaults to INLINECODE3

Variable	Required	If Missing
INLINECODE4	Conditional	Continue without token (public models work; gated models fail at download with clear error)
INLINECODE5

No | Warning + Default — Use /root/.cache/huggingface/hub |

Philosophy: Fail fast for configuration errors, fail at download time for authentication errors.

---

Helper Scripts

Location: INLINECODE7

check-env.sh

Validate and load environment variables before deployment.

Usage:
CODEBLOCK1

Exit Codes:

Code	Meaning
0	Environment check completed (variables loaded or defaulted)
2

Critical error (e.g., cannot source ~/.bash_profile) |

Note: This script is optional. You can also directly run source ~/.bash_profile.

---

generate-report.sh

Generate human-readable deployment report after successful deployment.

Usage:
CODEBLOCK2

Parameters:

Parameter	Required	Description
INLINECODE9	Yes	Model ID (with / replaced by -)
INLINECODE12

Yes | Docker container name |
| port | Yes | Host port for API endpoint |
| status | Yes | Deployment status (e.g., "✅ Success") |
| model-load-time | No | Model loading time in seconds |
| memory-used | No | Memory consumption in GiB |

Output: INLINECODE17

Exit Codes:

Code	Meaning
0	Report generated successfully
1

Missing required parameters |
| 2 | Output directory not found |

Integration: This script is automatically called in Phase 7 of the deployment workflow.

---

Input Schema

Parameter	Type	Required	Default	Description
modelid	String	Yes	-	HuggingFace model ID
dockerimage

String | No | rocm/vllm-dev:nightly | vLLM Docker image |
| tensorparallelsize | Integer | No | 1 | Number of GPUs |
| port | Integer | No | 9999 | API server port |
| hf_home | String | No | ${HF_HOME} or /root/.cache/huggingface/hub | Model cache directory |
| hf_token | Secret | Conditional | ${HF_TOKEN} | HuggingFace token (optional for public models, required for gated models) |
| maxmodellen | Integer | No | Auto-detect | Maximum sequence length |
| gpumemoryutilization | Float | No | 0.85 | GPU memory utilization |
| auto_install | Boolean | No | true | Auto-install dependencies |
| log_level | String | No | INFO | Logging verbosity |

Output Structure

All deployment artifacts MUST be saved to:
CODEBLOCK3

Convert model ID to directory name by replacing / with -:

• - openai/gpt-oss-20b → INLINECODE24
• INLINECODE25 → INLINECODE26

Per-model directory structure:
CODEBLOCK4

File requirements:

• - deployment.log — Capture ALL container logs during deployment
• INLINECODE28 — Save API response from functional test request
• INLINECODE29 — Generated in Phase 7
• All three files MUST exist before marking deployment as complete

Execution Workflow

Phase 0: Environment Check & Auto-Repair

Step 0.1: Load Environment Variables

CODEBLOCK5

If HFHOME is not defined in ~/.bashprofile, it defaults to /root/.cache/huggingface/hub.

Step 0.2: Create Output Directory

• - Create: INLINECODE31

Step 0.3: Initialize Logging

• - All output → INLINECODE32

Step 0.4: System Checks

• - Detect OS and package manager
• Check Python, pip, huggingface_hub
• Check Docker, docker compose
• Check ROCm tools (rocm-smi/amd-smi)
• Check GPU access (/dev/kfd, /dev/dri)
• Check disk space (20GB minimum)

Phase 1: Model Download

Use HF_HOME from Phase 0 (environment variable or default):

CODEBLOCK6

Authentication Handling:

Scenario	Behavior
Public model + no token	✅ Download succeeds
Public model + token provided

✅ Download succeeds |
| Gated model + no token | ❌ Download fails with "authentication required" error |
| Gated model + invalid token | ❌ Download fails with "invalid token" error |
| Gated model + valid token | ✅ Download succeeds |

On Authentication Failure:
CODEBLOCK7

• - Locate model path in HF cache: INLINECODE33
• Log download progress to INLINECODE34

Phase 2: Model Parameter Detection

• - Read config.json from model
• Auto-detect: maxmodellen, hiddensize, numattentionheads, numhiddenlayers, vocabsize, dtype
• Validate TP size divides attention heads
• Estimate VRAM requirement

Phase 3: Docker Compose Configuration

Generate files in output directory:

• - docker-compose.yml → INLINECODE35

- Mount HF_HOME as volume (read-only for models) - NO hardcoded tokens in compose file

• - .env → $HOME/vllm-compose/<model-id>/.env (optional)

- Contains: HF_TOKEN=<value> - Permissions: chmod 600 - Only created if user explicitly requests persistent token storage

Volume mount example:
CODEBLOCK8

Important: Docker Compose reads ${HF_HOME} from the host environment at runtime. Before running docker compose, source ~/.bash_profile: INLINECODE40

Phase 4: Container Launch

Important: Before deploying, pull the latest image to ensure updates:
CODEBLOCK9

Note: Default port is 9999. Before running docker compose, check if port is available: ss -tlnp | grep :<port>. If port is in use, specify a different port in docker-compose.yml.

• - Pass HFTOKEN at runtime: HFTOKEN=$HF_TOKEN docker compose up -d
• Wait for container initialization

Phase 5: Health Verification

• - Check container status
• Test /health endpoint
• Test /v1/models endpoint

Phase 6: Functional Testing

• - Run completion test via /v1/chat/completions API
• Save response to: INLINECODE43
• Verify response contains valid completion
• Log deployment complete → Append to INLINECODE44
• Deployment is complete only when both files exist:

- deployment.log - INLINECODE46

Phase 7: Deployment Report

Generate human-readable deployment report using the helper script.

Step 7.1: Extract Deployment Metrics

CODEBLOCK10

Step 7.2: Generate Report

CODEBLOCK11

Output: INLINECODE47

Report Contents:

• - Output structure verification (file checklist)
• Deployment summary table (health, test, metrics)
• Test results (request/response preview)
• Environment configuration
• Quick commands for operations

Completion Criteria:

• - DEPLOYMENT_REPORT.md exists in output directory
• Report contains all required sections
• All file checks show ✅

Security Best Practices

1. 1. Never commit tokens to version control — Add .env to INLINECODE50
2. Use .env files with chmod 600 — Restrict access to owner only
3. Mask tokens in logs — Show only first 10 chars: INLINECODE51
4. Pass tokens at runtime — INLINECODE52
5. Store tokens in ~/.bashprofile — For production environments, set HF_TOKEN in user's shell config
6. Set token for gated models — HFTOKEN is validated at download time; set in ~/.bash_profile for production

Troubleshooting

Environment Variables

Issue	Solution
INLINECODE54	Add export HF_TOKEN="hf_xxx" to ~/.bash_profile, then source ~/.bash_profile. Or provide via parameter.
INLINECODE58

defaults to /root/.cache/huggingface/hub. For production, add export HF_HOME="/path" to ~/.bash_profile. | | ~/.bash_profile not found | Create ~/.bash_profile and add environment variables. | | Changes not taking effect | Run source ~/.bash_profile or restart terminal. | | HF_TOKEN provided but download still fails | Token may be invalid or lack access to the model. Verify token at https://huggingface.co/settings/tokens |

Model Download

Issue	Solution
INLINECODE67 (gated model)	Set HF_TOKEN in ~/.bash_profile or provide via parameter. Ensure token has access to the model.
INLINECODE70

Verify model ID is correct (case-sensitive). Check model exists on HuggingFace. | | Download timeout | Check network connection. Large models may take time. |

Deployment

Issue	Solution
hf CLI not found	INLINECODE72
Docker Compose fails

Use docker compose (no hyphen) | | GPU access fails | Add user to render group: sudo usermod -aG render $USER | | Port in use | Change port parameter | | OOM | Reduce gpu_memory_utilization |

Cleanup

CODEBLOCK12

Status Check

Check deployment status and logs:

CODEBLOCK13

Quick Start (Production)

Step 1: Add environment variables to ~/.bash_profile

CODEBLOCK14

Step 2: Verify environment is ready

CODEBLOCK15

Step 3: Run deployment

CODEBLOCK16

Version History

Version	Changes
1.0.0	Initial release