Building the Fortress

Invisible Unicode is only half the story. To defend your system, you need a proper normalisation pipeline — not the “strip whitespace and pray” approach most apps use.

Here’s the C# blueprint.

Normalize(NormalizationForm.FormC)

This is your first line of defence.

FormC canonicalisation quietly removes or folds several problematic characters:

• U+2061 FUNCTION APPLICATION
• U+2062 INVISIBLE TIMES
• U+2063 INVISIBLE SEPARATOR
• U+2064 INVISIBLE PLUS
• U+200C (ZWNJ)
• U+200D (ZWJ)
• U+00AD (soft hyphen)
• U+2060 (word joiner)

These are invisible or semi‑invisible troublemakers. FormC neutralises them.

If you don’t want to rely on FormC, you can replicate the behaviour manually — but don’t skip it.

Enumerate Runes, Drop the Dangerous Ones

After canonicalisation, you still need a second pass.

Strip:

• Unicode Tag characters
• Variation Selectors
• Variation Selector Supplement
• All UnicodeCategory.Format characters
• Control characters (except newline)

Then normalise:

• multiple kinds of “space” → ' '
• fancy question marks → '?'
• curly quotes → straight quotes
• em/en dashes → hyphen
• box‑drawing characters → removed

This ensures consistency and prevents logit drift — because yes, adding random spaces does change model behaviour.

Confusables: The Silent Assassins

This is the part most engineers miss.

LLMs treat:

⟨tag⟩

and

<tag>

as the same thing.

They do not care that one is a mathematical angle bracket and one is ASCII. They do not care that one is CJK and one is deprecated. They do not care that one looks like it escaped from a 1980s typesetting machine.

To the model, they are interchangeable.

This matters because LLMs use <|...|> as boundary tokens.

If you don’t normalise confusables, attackers can jailbreak your system using:

⟨|system|⟩

instead of:

<|system|>

Your guardrails won’t trigger. Your filters won’t match. Your system will happily treat the fake tag as real.

The fix is simple: Replace all confusable delimiters with their ASCII equivalents.

The Confusable List

Here are the usual suspects:

• ⟨ ⟩
• 〈 〉
• ＜ ＞
• ﹤ ﹥
• ［ ］
• ﹇ ﹈
• ⦃ ⦄
• ｛ ｝
• ｜ ¦
• ： ﹕ ︓ ꞉

If your app uses tags, boundaries, or structured prompts, normalise these.

Your future self will thank you.

Final Advice

Normalisation isn’t glamorous. It isn’t fun. It isn’t the part of the project you brag about on LinkedIn.

But it’s the difference between:

• a secure AI system
• and a raccoon deleting your inbox because someone hid instructions inside invisible Unicode

If you want your apps to be safe, predictable, and jailbreak‑resistant, you need fortress‑grade normalisation.

It’s not hard. It’s just… overlooked.

Let’s fix that.