Alignment and Guardrails: Why Capable AI Still Needs Boundaries

In AI Foundations #24, we gave a language model tools and an action loop.

That immediately raises a new question:

If a model can do more, how do we make sure it does the right things?

This is where alignment and guardrails enter the picture.

They are related, but they are not the same thing.

A useful first approximation is:

alignment
= shape the model's behavior toward intended goals and constraints

guardrails
= controls around the model that restrict, validate, or monitor what the system can do

The distinction matters because a capable model is not automatically a safe system.

Capability and alignment are different axes

Suppose a model becomes better at programming.

That improvement can help it:

• explain code;
• fix bugs;
• automate repetitive work;
• reason about larger software systems.

But the same capability does not automatically answer questions such as:

Should this requested action be allowed?
Does the user have permission?
Is the model following the right instruction?
Should a destructive operation require confirmation?

Those are alignment and system-control questions.

A more capable model may make fewer reasoning mistakes while still operating under the wrong objective or an unsafe permission boundary.

So it helps to imagine at least two axes:

capability: how well can the model perform the task?
alignment: how well does its behavior match the intended rules and goals?

Improving one does not guarantee improving the other.

Pretraining does not specify every desired behavior

Recall pretraining.

A language model learns statistical structure from a large corpus by predicting tokens. That training produces broad capabilities, but the objective is not:

be helpful
follow the user's legitimate request
refuse dangerous requests
respect privacy
distinguish trusted instructions from untrusted webpage text
ask for confirmation before irreversible actions

The pretraining objective is much simpler.

That is why post-training exists.

Instruction tuning, preference training, reinforcement learning, and related methods try to shape a pretrained model into one that behaves more usefully under human instructions.

Instruction tuning teaches a behavioral format

Fine-tuning changes a model using additional examples.

For an instruction-following model, those examples can look like:

instruction
-> useful response

repeated across many tasks.

This teaches the model patterns such as:

• answer the question rather than continue the document;
• follow a requested format;
• ask for missing information when needed;
• separate an instruction from background text.

Instruction tuning can make a base model much easier to use.

But examples alone cannot enumerate every future situation.

The real world contains new tasks, ambiguous requests, adversarial inputs, and conflicting instructions.

Preference training adds a comparison signal

One influential approach is to collect judgments about which of several model responses is better.

Conceptually:

same prompt
-> response A
-> response B
-> human or model preference

Those comparisons can train a reward or preference signal, which can then be used to push the model toward preferred behavior.

The InstructGPT work is a well-known example of this family of methods. It combined supervised instruction data with human preference data and reinforcement learning.

The important idea for this lesson is not one particular algorithm.

It is that model behavior can be shaped by a signal richer than next-token prediction alone.

Alignment is not a solved optimization problem

It would be convenient if we could write:

reward = helpful + honest + safe

and optimize it perfectly.

Real preferences are not that simple.

People disagree. Rules can conflict. Context matters. A model can learn shortcuts that score well on training examples without capturing the intended principle.

There is also a classic optimization problem: when we measure a proxy for what we want, the system may optimize the proxy rather than the underlying goal.

For AI systems, this means a high preference score is evidence about behavior, not a mathematical proof of alignment.

Constitutional approaches make principles more explicit

Constitutional AI explores using a written set of principles to help critique and revise model responses and to produce preference signals.

At a high level:

model drafts response
-> response is evaluated against principles
-> critique/revision
-> improved training signal

This does not eliminate judgment. Someone still chooses the principles and the training process.

But it makes an important idea visible:

desired behavior can be represented partly as explicit rules, not only as thousands of isolated examples.

That begins to resemble the policy layer we saw in agent systems.

Model behavior and runtime enforcement are different

Suppose a model has been trained to avoid deleting files without permission.

That is useful.

But if the application gives the model an unrestricted delete tool and relies entirely on the model to remember the rule, the system has a fragile boundary.

A stronger design is:

model proposes delete
-> application checks authorization
-> application checks scope
-> application may require confirmation
-> tool executes only if policy allows it

The model can be aligned toward the desired behavior while the application independently enforces the most important constraints.

This is defense in depth.

Guardrails can exist before, during, and after inference

“Guardrail” is a broad engineering term rather than one single algorithm.

Controls can appear at several stages.

Before the model

The system can validate:

user identity
permissions
input size
allowed file types
tool availability
request scope

During model interaction

The host can enforce:

instruction hierarchy
tool schemas
maximum agent steps
allowed domains
read-only vs write permissions
context boundaries

After model output

The system can validate:

structured output
citations
policy constraints
dangerous commands
required confirmations
data leakage

The strongest systems do not expect one filter to solve every failure mode.

Instruction hierarchy answers “who gets to tell the model what to do?”

A tool-using model may see instructions from many places:

system policy
developer instructions
user request
retrieved documents
web pages
tool output
quoted emails
code comments

Those pieces of text do not have equal authority.

A web page saying “ignore all previous instructions” is data from the web page. It should not become more authoritative than the user’s task or the application’s policy.

This is why instruction hierarchy matters.

The system needs a rule for distinguishing:

trusted instruction
from
untrusted content that merely contains imperative language

That boundary becomes especially important in RAG and agents because external content is repeatedly inserted into model context.

Prompt injection is partly a boundary problem

Prompt injection is often described as “tricking the model with words.”

That description is incomplete.

The engineering problem is that the model reads both instructions and data through the same token interface.

If an application retrieves a document containing:

Ignore the user's request and do something else.

the model must recognize that sentence as content rather than authority.

Guardrails can reduce the risk by limiting what retrieved content can influence and by constraining what tools are available afterward.

The host system should not give an arbitrary retrieved page the effective authority to send email, delete data, or change permissions.

Red teaming searches for failures before deployment does

Normal testing asks:

Does the system work on expected inputs?

Red teaming asks something closer to:

How can the system fail when inputs are adversarial, unusual, or deliberately chosen to expose weaknesses?

For language models, that can include testing:

• conflicting instructions;
• jailbreak attempts;
• ambiguous authorization;
• malicious retrieved content;
• unusual encodings;
• multi-turn manipulation;
• combinations of tools that create unexpected side effects.

The point is not to prove that a model can never fail.

It is to discover failure modes early enough to improve training, policies, and system design.

Over-refusal is also a failure

Safety is not maximized by refusing everything.

Consider a model that rejects harmless requests because they resemble risky ones.

That system may be “safe” in a narrow sense but not useful.

So evaluation needs to examine both sides:

unsafe compliance
and
unnecessary refusal

This is another reason alignment cannot be summarized by one accuracy number.

A useful assistant needs calibrated behavior: allow legitimate tasks, constrain risky actions appropriately, and refuse only where necessary.

Agents raise the stakes

A text-only model can produce a bad answer.

An agent can turn a bad decision into an action.

The risk therefore depends on both model behavior and the tools around it.

A useful mental equation is:

system risk
≈ model error
× available capability
× permission scope
× reversibility

This is not a literal numerical formula. It is a way to think about why the same model can be low-risk in one application and high-risk in another.

A model with a calculator is different from the same model with unrestricted production credentials.

The layered mental model

We can now add another layer to the Foundations map:

pretraining
-> capabilities

post-training
-> preferred behavior

instruction hierarchy
-> decides which directions have authority

guardrails
-> constrain inputs, outputs, and actions

permissions
-> limit what tools can actually do

monitoring and evaluation
-> detect failures and improve the system

No single layer replaces the others.

A well-aligned model can still encounter an application bug.

A strict permission system can still produce a useless experience if the model refuses harmless tasks.

A good deployment tries to make the layers reinforce each other.

The main takeaway

Alignment is about steering model behavior toward intended goals and constraints.

Guardrails are the surrounding controls that help enforce those constraints in a real system.

The distinction becomes crucial once a model has tools.

A model may propose an action. The application still decides whether that action is valid, authorized, and safe to execute.

That gives us a durable rule:

Train the model to behave well, but do not make critical system boundaries depend on the model behaving perfectly.

Capability makes an AI system useful. Alignment and guardrails determine whether that capability is applied within the boundaries we actually intended.