ZIV LABS
Engineering notes

Treat model output as untrusted input

By ZIV LABS · Written and checked October 21, 2026 · Node.js v24.7.0

Whatever a language model returns, the code that receives it must treat it as untrusted input. Applications now pass model output to shells, file systems, web pages, and databases. OWASP lists this under "Improper Output Handling" (LLM05:2025) and describes it as weak validation, sanitization, and handling of model output before other components consume it. This note shows what that looks like in code: four consumers that use a field from a model's output directly, an attack string for each, and the fix. Everything was run on Node.js v24.7.0.

No model is called here. A fixed string stands in for an output that an attacker influenced. We are not claiming that any particular model produces such output. The point is that you cannot rule it out, so the consumer must not depend on the model behaving.

Why the output is untrusted

A model's output depends on its input, and its input often includes text the application did not write: what a user types, a web page it was asked to summarize, a document it retrieved, a tool's response. Anyone who controls any of that text can try to steer the output. OWASP's guidance is to treat the model as an untrusted user and to validate its responses before they reach backend functions. That is the same discipline as for form fields and query parameters, applied to a new source.

Four consumers that trust the output

Suppose an assistant returns JSON with four fields: a branch name, a filename, a summary, and a title. Each of the functions below uses one field directly. All four are wrong.

import { exec } from "node:child_process";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { promisify } from "node:util";

const run = promisify(exec);

// Each function takes one field of the model's output and uses it directly. All four are wrong.

// 1. Shell: the field becomes part of a command line.
export async function createBranch(workdir, branch) {
  const { stdout } = await run(`echo creating branch ${branch}`, { cwd: workdir });
  return stdout.trim();
}

// 2. File path: the field is joined to a directory and read.
export function readNote(baseDir, filename) {
  return readFileSync(join(baseDir, filename), "utf8").trim();
}

// 3. HTML: the field is placed into markup.
export function renderSummary(summary) {
  return `<p class="summary">${summary}</p>`;
}

// 4. SQL: the field is placed into a query string.
export function findNotes(db, title) {
  return db.prepare(`SELECT title FROM notes WHERE title = '${title}'`).all().map((r) => r.title);
}

The attack string for each field is below. The demo feeds them to the flawed consumers, then to the fixed ones from the next section:

model output: {"branch":"feature-x; touch PWNED","filename":"../secret.txt","summary":"<img src=x onerror=alert(1)>","title":"' OR '1'='1"} 

== flawed consumers ==
shell : creating branch feature-x | PWNED file created: true
path  : TOP-SECRET
html  : <p class="summary"><img src=x onerror=alert(1)></p>
sql   : [ 'public note', 'private note' ]

== validation, then safe consumers ==
validation rejects the whole output: branch name rejected
shell : creating branch feature-x; touch PWNED | PWNED file created: false
path  : path escapes the base directory
html  : <p class="summary">&lt;img src=x onerror=alert(1)&gt;</p>
sql   : []

Reading the flawed side:

  • Shell. The branch name contained ; touch PWNED. The shell ran the second command and created the file, which the output confirms (PWNED file created: true). We used touch because it is harmless; any command would have run the same way.
  • Path. ../secret.txt left the notes directory and returned the contents of a file outside it.
  • HTML. The summary reached the markup as a live tag with an onerror handler. We only printed the string; we did not render it in a browser, but this is the markup a browser would act on.
  • SQL. The title ' OR '1'='1 turned the filter into a condition that is always true, so the query returned both rows, including the private one.

The fixes

import { execFile } from "node:child_process";
import { readFileSync } from "node:fs";
import { resolve, sep } from "node:path";
import { promisify } from "node:util";

const runFile = promisify(execFile);

// Validate the shape first: known keys only, each value checked against what the consumer needs.
const BRANCH = /^[A-Za-z0-9][A-Za-z0-9._\/-]{0,62}$/;
const FILENAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,62}$/;

export function parseModelOutput(text) {
  const data = JSON.parse(text);
  const allowed = ["branch", "filename", "summary", "title"];
  for (const key of Object.keys(data)) if (!allowed.includes(key)) throw new Error(`unexpected key ${key}`);
  if (!BRANCH.test(data.branch ?? "")) throw new Error("branch name rejected");
  if (!FILENAME.test(data.filename ?? "")) throw new Error("filename rejected");
  if (typeof data.summary !== "string" || data.summary.length > 500) throw new Error("summary rejected");
  if (typeof data.title !== "string" || data.title.length > 100) throw new Error("title rejected");
  return data;
}

// 1. Shell: no shell at all. The value is one argument, never parsed as a command line.
export async function createBranch(workdir, branch) {
  const { stdout } = await runFile("echo", ["creating branch", branch], { cwd: workdir });
  return stdout.trim();
}

// 2. File path: resolve, then require the result to stay inside the base directory.
export function readNote(baseDir, filename) {
  const base = resolve(baseDir);
  const target = resolve(base, filename);
  if (!target.startsWith(base + sep)) throw new Error("path escapes the base directory");
  return readFileSync(target, "utf8").trim();
}

// 3. HTML: encode for the context where the text is displayed.
const escapeHtml = (s) => s.replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[c]);
export function renderSummary(summary) {
  return `<p class="summary">${escapeHtml(summary)}</p>`;
}

// 4. SQL: a parameterized query keeps the value as data.
export function findNotes(db, title) {
  return db.prepare("SELECT title FROM notes WHERE title = ?").all(title).map((r) => r.title);
}

The principles, matching the four examples:

  • Validate the shape first. parseModelOutput accepts known keys only and checks each value against what its consumer needs. In the demo it rejects the whole output ("branch name rejected").
  • Do not use a shell for arguments. execFile passes the value as one argument. In the last demo the shell-metacharacter string is printed literally and no file is created.
  • Resolve paths, then check containment. The resolved path must stay inside the base directory.
  • Encode for the output context. HTML-encode text shown in a page. The tag is now inert text.
  • Parameterize queries. The title stays data, and the query returns nothing.

Each consumer is safe even if the validation step were removed. That is deliberate: validation can be bypassed, changed, or forgotten, and the consumer that performs the dangerous operation is the right place for the final protection.

What this does not cover

  • Argument injection. execFile avoids shell parsing, but the program may still treat an argument that starts with - as an option. Our branch pattern requires an alphanumeric first character. We did not test option injection against real tools.
  • Excessive agency. Clean handling of output does not help if the model is given tools it should not have. Least privilege, confirmation for destructive actions, and scoped credentials are separate controls (OWASP LLM06:2025).
  • Other consumers. Markdown renderers, email templates, URL fetchers, and templating engines need the same treatment, with the encoding or validation that suits each context.
  • Browser behavior. A Content-Security-Policy can limit the damage of injected markup. We did not test any policy.

Checklist

  • List every place the model's output is used: shell, file system, HTML or Markdown, SQL, URLs, templates, other services.
  • For each, apply the control for that context: no shell, containment checks, output encoding, parameterized queries.
  • Validate the structure of structured output before using any field, and reject unknown keys.
  • Run the model's tools with the least privilege that works.
  • Keep the attack strings above as tests for your own consumers.

Reproduce it

Save flawed.mjs and safe.mjs from above and the demo below in one folder, and run node demo.mjs. Node prints an experimental-warning message for node:sqlite on stderr; the results above are the standard output. The demo creates temporary directories under the system temp folder.

demo.mjs
import { mkdtempSync, mkdirSync, writeFileSync, existsSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { DatabaseSync } from "node:sqlite";
import * as flawed from "./flawed.mjs";
import * as safe from "./safe.mjs";

// No model is called. This string stands in for what an attacker-influenced model could return.
const modelOutput = JSON.stringify({
  branch: "feature-x; touch PWNED",
  filename: "../secret.txt",
  summary: "<img src=x onerror=alert(1)>",
  title: "' OR '1'='1",
});

const root = mkdtempSync(join(tmpdir(), "llm05-"));
const work = join(root, "work-flawed");
const workSafe = join(root, "work-safe");
const notes = join(root, "notes");
mkdirSync(work);
mkdirSync(workSafe);
mkdirSync(notes);
writeFileSync(join(root, "secret.txt"), "TOP-SECRET");
writeFileSync(join(notes, "a.txt"), "a normal note");

const db = new DatabaseSync(":memory:");
db.exec("CREATE TABLE notes (title TEXT); INSERT INTO notes VALUES ('public note'), ('private note');");

const data = JSON.parse(modelOutput);
console.log("model output:", modelOutput, "\n");

console.log("== flawed consumers ==");
console.log("shell :", await flawed.createBranch(work, data.branch), "| PWNED file created:", existsSync(join(work, "PWNED")));
console.log("path  :", flawed.readNote(notes, data.filename));
console.log("html  :", flawed.renderSummary(data.summary));
console.log("sql   :", flawed.findNotes(db, data.title));

console.log("\n== validation, then safe consumers ==");
try {
  safe.parseModelOutput(modelOutput);
} catch (e) {
  console.log("validation rejects the whole output:", e.message);
}
// Even without the validation step, each consumer is safe on its own:
console.log("shell :", await safe.createBranch(workSafe, data.branch), "| PWNED file created:", existsSync(join(workSafe, "PWNED")));
try {
  safe.readNote(notes, data.filename);
} catch (e) {
  console.log("path  :", e.message);
}
console.log("html  :", safe.renderSummary(data.summary));
console.log("sql   :", safe.findNotes(db, data.title));

What we did not verify

We ran the code only on Node.js v24.7.0. We did not call a model, so we make no claim about how any model behaves or how often its output contains attack strings. We did not render the HTML in a browser, test other languages or frameworks, test argument injection, or test Content-Security-Policy. The examples are small by design; real systems have more consumers than these four.

References

Related tools

  • JSON Validator — Checks JSON syntax. Syntax is not shape: checking the fields of a model's output is your code's job, as shown above.

Small Tools is built and operated by ZIV LABS. These links go to tools.ziv-labs.com.

Related notes

  • JWT verification mistakes — Four tokens a flawed JWT verifier accepts and a strict one rejects: alg none, RS256-to-HS256 confusion, and unchecked claims, reproduced with runnable Node.js code.
  • Does this online tool upload your input? — How to test the claim that a tool runs in your browser, what we observed on our own tool, and why a clean result is evidence rather than proof.

Further reading