Skip to content

RHIDP-14083: extend customization profile for safety shield prompts #2374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Jdubrick wants to merge 2 commits into
base: main
Choose a base branch
from
Open

RHIDP-14083: extend customization profile for safety shield prompts #2374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f24922f
Resolve question_validity shield prompts from customization profile.
Jdubrick Aug 5, 2026
c48a776
address coderabbit
Jdubrick Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 48 additions & 9 deletions docs/devel_doc/openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1440,7 +1440,7 @@
"config": {
"invalid_question_response": "I can only answer questions about the product.",
"model_id": "openai/gpt-4o-mini",
"model_prompt": "Is this question valid?"
"model_prompt": "Is this question valid? ${message}"
},
"name": "question-validity",
"provider_id": "question_validity",
Expand Down Expand Up @@ -13887,7 +13887,34 @@
"type": "object",
"title": "System prompts",
"description": "Dictionary containing map of system prompts",
"default": {}
"default": {},
"readOnly": true
},
"validation": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Question validity classifier prompt",
"description": "Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG['system_prompts']['validation']), not from lightspeed-stack.yaml. Used when a question_validity shield omits model_prompt.",
"readOnly": true
},
"invalid_resp": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Invalid question response",
"description": "Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG['query_responses']['invalid_resp']), not from lightspeed-stack.yaml. Used when a question_validity shield omits invalid_question_response.",
"readOnly": true
}
},
"type": "object",
Expand Down Expand Up @@ -18804,16 +18831,28 @@
"description": "The model_id to use for the guard"
},
"model_prompt": {
"type": "string",
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Model prompt",
"description": "The default prompt sent to the LLM used to validate the Users' question.",
"default": "\nInstructions:\n- You are a question classifying tool\n- You are an expert in kubernetes and openshift\n- Your job is to determine where or a user's question is related to kubernetes and/or openshift technologies and to provide a one-word response.\n- If a question appears to be related to kubernetes or openshift technologies, answer with the word ${allowed}, otherwise answer with the word ${rejected}.\n- Do not explain your answer, just provide the one-word response. Do not give any other response.\n- If the given question is an empty string, answer with the word ${rejected}\n\n\nExample Question:\nWhy is the sky blue?\nExample Response:\n${rejected}\n\nExample Question:\nWhy is the grass green?\nExample Response:\n${rejected}\n\nExample Question:\nWhy is sand yellow?\nExample Response:\n${rejected}\n\nExample Question:\nCan you help configure my cluster to automatically scale?\nExample Response:\n${allowed}\n\nQuestion:\n${message}\nResponse:\n"
"description": "Classifier prompt for the agent / wrap_run path. Null/omitted at load is filled from the profile module's system_prompts.validation, then the LCORE default; an explicit empty string is kept. Include $message or ${message} so the user question is substituted. Not used by responses-path run(), which sends raw user input."
},
"invalid_question_response": {
"type": "string",
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Invalid question response",
"description": "The default response when the Users' question is determined to be invalid.",
"default": "\nHi, I'm the OpenShift Lightspeed assistant, I can help you with questions about OpenShift, \nplease ask me a question related to OpenShift.\n"
"description": "Refusal text used on both agent and responses / run() paths. Null/omitted at load is filled from the profile module's query_responses.invalid_resp, then the LCORE default; an explicit empty string is kept."
}
},
"additionalProperties": false,
Expand Down Expand Up @@ -21233,7 +21272,7 @@
"config": {
"invalid_question_response": "I can only answer questions about the product.",
"model_id": "openai/gpt-4o-mini",
"model_prompt": "Is this question valid?"
"model_prompt": "Is this question valid? ${message}"
},
"name": "question-validity",
"provider_id": "question_validity",
Expand Down
37 changes: 28 additions & 9 deletions docs/models/successful_responses.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1524,8 +1524,25 @@
},
"default": {},
"description": "Dictionary containing map of system prompts",
"readOnly": true,
"title": "System prompts",
"type": "object"
},
"validation": {
"type": "string",
"nullable": true,
"default": null,
"description": "Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG['system_prompts']['validation']), not from lightspeed-stack.yaml. Used when a question_validity shield omits model_prompt.",
"readOnly": true,
"title": "Question validity classifier prompt"
},
"invalid_resp": {
"type": "string",
"nullable": true,
"default": null,
"description": "Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG['query_responses']['invalid_resp']), not from lightspeed-stack.yaml. Used when a question_validity shield omits invalid_question_response.",
"readOnly": true,
"title": "Invalid question response"
}
},
"required": [
Expand Down Expand Up @@ -4473,16 +4490,18 @@
"type": "string"
},
"model_prompt": {
"default": "\nInstructions:\n- You are a question classifying tool\n- You are an expert in kubernetes and openshift\n- Your job is to determine where or a user's question is related to kubernetes and/or openshift technologies and to provide a one-word response.\n- If a question appears to be related to kubernetes or openshift technologies, answer with the word ${allowed}, otherwise answer with the word ${rejected}.\n- Do not explain your answer, just provide the one-word response. Do not give any other response.\n- If the given question is an empty string, answer with the word ${rejected}\n\n\nExample Question:\nWhy is the sky blue?\nExample Response:\n${rejected}\n\nExample Question:\nWhy is the grass green?\nExample Response:\n${rejected}\n\nExample Question:\nWhy is sand yellow?\nExample Response:\n${rejected}\n\nExample Question:\nCan you help configure my cluster to automatically scale?\nExample Response:\n${allowed}\n\nQuestion:\n${message}\nResponse:\n",
"description": "The default prompt sent to the LLM used to validate the Users' question.",
"title": "Model prompt",
"type": "string"
"type": "string",
"nullable": true,
"default": null,
"description": "Classifier prompt for the agent / wrap_run path. Null/omitted at load is filled from the profile module's system_prompts.validation, then the LCORE default; an explicit empty string is kept. Include $message or ${message} so the user question is substituted. Not used by responses-path run(), which sends raw user input.",
"title": "Model prompt"
},
"invalid_question_response": {
"default": "\nHi, I'm the OpenShift Lightspeed assistant, I can help you with questions about OpenShift, \nplease ask me a question related to OpenShift.\n",
"description": "The default response when the Users' question is determined to be invalid.",
"title": "Invalid question response",
"type": "string"
"type": "string",
"nullable": true,
"default": null,
"description": "Refusal text used on both agent and responses / run() paths. Null/omitted at load is filled from the profile module's query_responses.invalid_resp, then the LCORE default; an explicit empty string is kept.",
"title": "Invalid question response"
}
},
"required": [
Expand Down Expand Up @@ -5838,7 +5857,7 @@
"config": {
"invalid_question_response": "I can only answer questions about the product.",
"model_id": "openai/gpt-4o-mini",
"model_prompt": "Is this question valid?"
"model_prompt": "Is this question valid? ${message}"
},
"name": "question-validity",
"provider_id": "question_validity",
Expand Down
6 changes: 4 additions & 2 deletions docs/models/successful_responses.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,8 @@ Custom profile customization for prompts and validation.
|-------|------|-------------|
| path | string | Path to Python modules containing custom profile. |
| prompts | object | Dictionary containing map of system prompts |
| validation | string | Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG["system_prompts"]["validation"]), not from lightspeed-stack.yaml. Used when a question_validity shield omits model_prompt. |
| invalid_resp | string | Read-only. Loaded from the custom profile Python module (PROFILE_CONFIG["query_responses"]["invalid_resp"]), not from lightspeed-stack.yaml. Used when a question_validity shield omits invalid_question_response. |


## Customization
Expand Down Expand Up @@ -1951,8 +1953,8 @@ Configuration for the question validity guardrail.
| Field | Type | Description |
|-------|------|-------------|
| model_id | string | The model_id to use for the guard |
| model_prompt | string | The default prompt sent to the LLM used to validate the Users' question. |
| invalid_question_response | string | The default response when the Users' question is determined to be invalid. |
| model_prompt | string | Classifier prompt for the agent / wrap_run path. Null/omitted at load is filled from the profile module's system_prompts.validation, then the LCORE default; an explicit empty string is kept. Include $message or ${message} so the user question is substituted. Not used by responses-path run(), which sends raw user input. |
| invalid_question_response | string | Refusal text used on both agent and responses / run() paths. Null/omitted at load is filled from the profile module's query_responses.invalid_resp, then the LCORE default; an explicit empty string is kept. |


## QuestionValidityShieldConfiguration
Expand Down
14 changes: 8 additions & 6 deletions docs/user_doc/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,10 +281,12 @@ Conversation history configuration.
Custom profile customization for prompts and validation.


| Field | Type | Description |
|---------|--------|---------------------------------------------------|
| path | string | Path to Python modules containing custom profile. |
| prompts | object | Dictionary containing map of system prompts |
| Field | Type | Description |
|--------------|--------|---------------------------------------------------|
| path | string | Path to Python modules containing custom profile. |
| prompts | object | Read-only map of system prompts loaded from the profile module |
| validation | string | Read-only. Loaded from profile module `PROFILE_CONFIG["system_prompts"]["validation"]` (not a YAML `customization` field). Used when a question_validity shield omits `model_prompt`. |
| invalid_resp | string | Read-only. Loaded from profile module `PROFILE_CONFIG["query_responses"]["invalid_resp"]` (not a YAML `customization` field). Used when a question_validity shield omits `invalid_question_response`. |


## Customization
Expand Down Expand Up @@ -767,8 +769,8 @@ Configuration for the question validity guardrail.
| Field | Type | Description |
|---------------------------|--------|---------------------------------------------------------------|
| model_id | string | The model_id to use for the guard |
| model_prompt | string | Prompt sent to the LLM used to validate the user's question |
| invalid_question_response | string | Response when the user's question is determined to be invalid |
| model_prompt | string | Optional classifier prompt (`null` when omitted in YAML). Filled at load from the profile module's `system_prompts.validation`, then the LCORE default. Explicit YAML (including `""`) is kept. Used on the agent / `wrap_run` path (with `$message` / `${message}` substitution). Not applied by responses-path `run()`. After load / on `GET /v1/shields`, the effective string is returned. |
| invalid_question_response | string | Optional refusal text (`null` when omitted in YAML). Filled at load from the profile module's `query_responses.invalid_resp`, then the LCORE default. Explicit YAML (including `""`) is kept. Used on both agent and responses / `run()` paths. After load / on `GET /v1/shields`, the effective string is returned. |


## QuestionValidityShieldConfiguration
Expand Down
Loading