What you’ll take away
- Define where prompts, documents, model files, logs, and backups may travel before calling the deployment private.
- Verify model provenance and integrity, then separately evaluate loading risks, behavior, and fitness for the intended use.
- Release against scoped acceptance evidence with named operating owners, update procedures, and a tested fallback.
In this guide
Define what private means for this service
Moving a large language model onto company-controlled infrastructure changes who operates the service. It does not, by itself, establish who can use it, where supporting data goes, or whether the software is safe to run. Before buying capacity or opening a pilot, state what the private deployment must achieve. The answer might concern document access, external processing, administrator control, or availability when an outside service cannot be reached.
Consider a fictional engineering firm building an internal assistant for design documents. Engineers want to find relevant project material and draft summaries. The firm plans to run inference in its own environment and limit retrieval to documents each employee may already access. This is an illustrative planning example, not an SPS client outcome. The assistant supports research and drafting; an engineer remains responsible for checking the original material before using it in project work.
Write the intended users, approved tasks, data classes, hosting arrangement, and prohibited uses into a short service description. Include the business and technical owners. Joint government guidance on secure AI deployment emphasizes that safeguards depend on the deployment environment and infrastructure. [1] The practical consequence is that “self-hosted” needs a supporting design. It is an operating choice, not a security assurance or a claim that particular compliance obligations have been satisfied.
Map the boundary beyond the model endpoint
Follow one request from the employee’s browser through sign-in, the application gateway, document retrieval, the inference service, and the returned answer. Then map the supporting flows: document ingestion, model downloads, software updates, monitoring, backups, and administrator access. Record where each component runs, who controls it, and what information crosses each connection. A private inference endpoint describes only one part of this system.
For the fictional firm, the design documents may stay on internal storage while an embedding service or diagnostic feature sends their contents elsewhere. Check those dependencies explicitly. Identify whether embedding generation is local, where the search index is stored, and whether any feature uses an external model as a fallback. Distinguish a deliberate, approved connection from an unknown default. Do not label the whole workflow offline merely because the main model server has no public address.
Agree the allowed destinations and data for every outbound route, including telemetry and support exports. If the deployment must operate without network connectivity to external services, plan controlled import of model files, dependencies, and security updates, plus the approved export of operational evidence. If a managed AI service is part of the design, review its actual configuration and terms alongside the self-hosted components. Keep unresolved boundary questions visible before real project documents enter the pilot.
The model is one part of the boundary.
Fictional engineering firm: an internal design-document assistant.
- Staff request
Named user identity
- Authenticated gateway
Access and request limits
- Local model service
Controlled inference endpoint
Permitted document retrieval → model contextEnforce the user’s access before retrieving records.
- Artifact flowApproved registry → model service
- Retained dataApplication + stores → logs / backups
- Network egressOutbound policy → allowed destinations
Treat model intake as a controlled release process
Create an approved intake path for models and their supporting files. Record the publisher, source location, exact version or revision, intended use, file inventory, and review owner. Include associated tokenizer files, configuration, adapters, and any custom code. The engineering firm should be able to identify the complete approved package behind a deployed assistant, rather than point to a public model name whose contents may change.
Joint deployment guidance recommends cryptographic checks for artifact origin and integrity, alongside version control and testing. [1] Compare files with trusted reference values and verify signatures where available using an approved trust source. A matching hash means the file matches the reference; it does not establish that the reference came from a trustworthy publisher. Even verified origin and unchanged contents do not establish safe behavior, acceptable accuracy, or permission to use the model for your purpose.
Keep licensing and usage review separate from technical verification. Record applicable terms and route questions about permitted use to the appropriate reviewer. Store approved packages in a controlled internal registry and restrict who can replace them. For the fictional firm, a new model revision should enter evaluation before it reaches the document assistant. Preserve the intake record and release decision so an operator can later explain which package was approved and whether the running service matches it.
Approve the model before production loads it.
- Approved source
Publisher, version, and intended use
- Verify the artifact
Provenance, integrity, and license review
- Isolated review
Controlled loading and scoped evaluation
- Approved registry
Record the accepted artifact and checks
- Controlled release
Deploy the approved version with rollback
A matching fingerprint supports integrity. It does not establish model safety, publisher trust, or license rights.
Review how the package loads, not just where it came from
Model files are part of a software supply chain. Some loading paths can execute code, and a package may depend on libraries or custom components beyond its weights. PyTorch’s loading documentation explicitly warns against loading data from an untrusted source. [2] Treat first-time loading as a security-relevant operation. The person reviewing the package needs to understand the format, loader, dependencies, and privileges involved.
Hugging Face recommends the safetensors format to avoid the code-execution risks of unsafe weight formats and separately advises reviewing remote modeling code and pinning its revision. [3] These address specific loading risks. A safer weight format does not validate surrounding application code, eliminate vulnerabilities in dependencies, or establish that model outputs are reliable. Do not interpret a successful file scan or a loader option as approval of the complete service.
Evaluate a candidate package in a separated environment with limited privileges, controlled network access, and synthetic or approved test data. Record unexpected file writes, attempted connections, loading failures, and resource use alongside the functional results. For the engineering firm, evaluation should not need production document credentials. If the package requires additional executable code or broader access than expected, resolve that request through review before promoting it into the production registry.
Make the gateway enforce who can use the service
Put the inference service behind an explicitly designed access path. Identify the employee or calling application, authorize the permitted operation, and protect the connection. Joint AI deployment guidance calls for authentication and authorization at exposed APIs. [1] Ask the platform team to demonstrate what prevents a caller from reaching the model server directly and bypassing the application’s controls, including through internal network routes.
Separate the identities used for staff sign-in, application-to-model requests, document retrieval, and administration. In the fictional firm, membership in the assistant’s user group should not grant permission to upload a replacement model or read every project folder. The application must enforce document permissions when retrieving content; telling the model to respect project boundaries is not an access-control mechanism. Keep service credentials scoped to their purpose and document their storage, rotation, and revocation.
Set limits that match the workload: supported operations, request sizes, concurrency, and resource consumption. Review unused interfaces and administrative endpoints rather than exposing every capability of the serving software. Test both allowed and denied paths with representative identities, including direct requests that omit the normal user interface. Include a practical question for operators: if one account is disabled, which sessions or credentials could still reach the service, and how is their termination verified?
Protect the copies created during normal operation
Inventory what persists after a request completes: prompts, responses, conversation history, retrieval snippets, caches, temporary files, traces, and backups. Decide which records are necessary, who can access them, how long they remain, and how removal is handled. For the fictional firm, a design excerpt in a troubleshooting trace may need the same access restrictions as the source document. Moving it into an operational tool should not silently broaden its audience.
Use operational logging that supports investigation without collecting document content by default when it is unnecessary. Useful evidence may include the requesting identity, request identifier, model version, authorization result, and configuration changes. Review error handling and diagnostic modes, where additional content may appear. Test redaction against representative synthetic documents and inspect the resulting records. Document the cases it does not cover so operators know when an export needs additional handling.
Apply the boundary rules to backup and monitoring systems too. Verify where copies are stored, who can restore them, and how restored data inherits access restrictions. Agree how deletion or access changes affect indexes, cached responses, and retained copies; avoid promising instant deletion from every backup. Observe outbound behavior during ordinary use, failures, and updates, then compare it with the approved destinations. A blocked internet route is useful only when the team understands which necessary operations depend on it.
Give updates, recovery, and shutdown named owners
Before launch, assign responsibility for the model registry, platform patches, access reviews, document ingestion, monitoring, and incident escalation. Name a service owner who can pause the assistant when evidence or operational conditions warrant it. Establish how users continue their work while the service is unavailable. The engineering firm might return to searching the original document repository, with a clear message that generated summaries are temporarily unavailable.
Treat an update as a change to a versioned system. Model weights, serving software, prompt templates, retrieval configuration, and document indexes can all affect behavior. Record what changed and repeat the relevant security and functional checks in a staging environment. Keep the previous approved configuration and artifacts available, but do not assume that restoring an old model also restores compatible permissions, indexes, or application settings.
Rehearse rollback and recovery with the operators who will perform them. Define the trigger, approving role, steps, expected interruption, and evidence of a successful return to service. NIST’s Generative AI Profile recommends testing and managing fallback arrangements, including manual processing where appropriate. [4] A fallback that sends prompts to an unreviewed external service would change the privacy boundary. Evaluate that path explicitly, and record what users and support staff should do when recovery cannot proceed as planned.
Make the go-live decision from scoped evidence
Agree acceptance criteria before the pilot ends. For the fictional assistant, evidence might demonstrate that an unauthorized employee cannot reach the endpoint, one project’s user cannot retrieve another project’s test documents, deployed files match the approved registry, and observed outbound traffic matches the allowed destinations. Add a recovery result and confirmation that the responsible operators can find the runbook. Record the environment, versions, identities, and limitations of each check.
NIST cautions that generative AI testing can fail to reflect the deployment context. [4] Use representative tasks and data, evaluate the whole application, and describe what remains untested. Record unresolved issues with owners and a decision about whether they block release or restrict the pilot. Evidence supports the defined release decision; it does not guarantee that every attack or future change has been addressed. Expansion to more users, data, or capabilities should trigger another review.
SPS’s Private & Self-Hosted LLM Deployment service covers model intake, inference-server hardening, GPU-host review, segmentation, access design, and validation evidence. You receive a reference architecture, implementation guidance, and control records; your administrators apply changes. For an initial discussion, bring the proposed workload, data-flow diagram, model inventory, hosting choices, and operating owners. That makes it possible to scope the work around the deployment you intend to run and the evidence needed to approve it.
Make go-live a scoped decision.
- OWNER
- Identity + application owners
- EVIDENCE
- Tested user permissions, denied access, and administrative separation
- OWNER
- Data + platform owners
- EVIDENCE
- Reviewed retrieval, retention, backups, and outbound routes
- OWNER
- Platform + service owners
- EVIDENCE
- Documented updates, monitoring, restore checks, and rollback
Document accepted scope, unresolved conditions, accountable approval, and rollback.
Define the security boundary for your private LLM.
Bring your model, data flows, and proposed hosting design. SPS can scope the architecture, hardening guidance, and validation evidence needed before release.
Discuss private LLM deploymentReferences & further reading
- Joint guidance: Deploying AI Systems Securely — Best Practices for Deploying Secure and Resilient AI Systems
- PyTorch documentation: torch.load security warning
- Hugging Face Transformers: Security Policy for Remote Artifacts and Code
- NIST AI 600-1: Artificial Intelligence Risk Management Framework — Generative Artificial Intelligence Profile