REST APIs move data between systems, and every handoff point is an opportunity for encoding issues to appear. Query parameters get double-encoded. Binary attachments arrive garbled. Tokens get corrupted in transit. Most of these failures follow a predictable pattern: the data was encoded correctly for one context but not transformed correctly when it crossed a boundary.
This guide walks through encoding decisions at each layer of a typical REST API interaction. The goal is not to cover every edge case, but to give you a clear mental model of what encoding is required at each step and why.
Step 1: Encode Query Parameters Before Appending Them to URLs
When you build a URL with dynamic data, the values that go into query parameters must be percent-encoded. The characters ?, &, =, +, #, and / all have structural roles in URLs. If your data contains any of them, the URL parser misinterprets them as URL structure unless they are encoded first.
The correct approach is to use a URL encoding function rather than building the query string by hand. In JavaScript, encodeURIComponent encodes a single value correctly for use as a query parameter value. Python's urllib.parse.urlencode handles a dictionary of parameters. PHP's http_build_query does the same. The MDN documentation on URLs and encoding covers the JavaScript functions with examples.
Avoid building query strings with string concatenation like "?q=" + userInput. This works until the user input contains a & character, at which point the URL structure breaks.
Step 2: Set and Respect the Content-Type Header
The Content-Type header tells the receiving system how to interpret the request body. For JSON payloads, it should be application/json. For form data, application/x-www-form-urlencoded or multipart/form-data depending on whether you have file uploads.
This header matters for encoding because the body encoding rules differ by content type. A JSON body must be valid JSON, which means string values are JSON-escaped (backslash sequences for special characters). A form-encoded body uses the + convention for spaces and percent-encoding for everything else. A multipart body uses MIME boundary encoding.
If the Content-Type header mismatches the actual body format, the server may fail to parse the request silently, return a generic error, or parse it incorrectly and produce unexpected behavior.
Step 3: Handle Binary Data with Base64
REST APIs that use JSON or XML for their bodies cannot directly include binary data. Binary bytes include zero values and control characters that are not valid in JSON strings. The solution is to Base64-encode binary data before placing it in a JSON body and decode it after extraction.
A common pattern is to accept file uploads as Base64-encoded strings in a JSON field:
{
"filename": "avatar.png",
"content": "iVBORw0KGgoAAAANSUhEUgAA..."
}
The receiver Base64-decodes the content field to get the original binary file. This pattern avoids the complexity of multipart form data for APIs that are primarily JSON.
The tradeoff is size: Base64 encoding adds approximately 33% to the data size. For large files, multipart form data is more efficient because it sends the binary directly without the encoding overhead.
Step 4: Use Base64URL for Tokens and Identifiers in URLs
If you need to include binary or arbitrary data as a URL path segment or query parameter, use Base64URL encoding rather than standard Base64. Standard Base64 uses + and / as characters 62 and 63. Both of these have structural meaning in URLs, so a standard Base64 string embedded in a URL can break the URL parser.
Base64URL substitutes - for + and _ for /. It also omits the = padding characters, which are structural in query string context. The result is a string that can be embedded anywhere in a URL without additional percent-encoding.
JWT tokens use Base64URL encoding for this reason. If your API issues tokens or opaque identifiers that will appear in URLs, Base64URL is the correct encoding to use. The Wikipedia article on Base64 covers the URL-safe variant in the section on implementations.
Step 5: HTML-Encode API Responses Before Rendering Them in a Browser
This step happens on the client side, but it is worth including because API developers often think about encoding only in terms of request handling. When a browser application receives API data and renders it in the DOM, every string value from the API must be HTML-encoded before insertion into HTML.
An API that returns user-supplied text (comments, usernames, descriptions) returns exactly what was stored. If a user submitted a string containing HTML tags, the API returns those tags. If the client renders the API response using innerHTML, the tags execute. If the client uses a framework's data-binding syntax (React's JSX, Angular's {{ }}, Vue's v-bind), the framework handles HTML encoding automatically in the default rendering mode.
The safety rule: never use raw HTML insertion for API-supplied strings. Use the framework's data-binding system or a trusted HTML sanitizer if rich text rendering is required.
Step 6: Validate Encoding at System Boundaries
The most reliable place to enforce encoding rules is at the boundary where data enters or leaves a system. Validate that query parameters are correctly percent-encoded before processing them. Validate that JSON bodies are valid JSON before parsing them. Validate that Base64 strings are valid Base64 before decoding them.
The OWASP Cheat Sheet Series on input validation covers the broader input handling recommendations. For encoding specifically, validation prevents the double-encoding problems that arise when intermediate layers apply additional transformations to data that was already correctly encoded.
Step 7: Test Your Encoding With Real Data
The encoding bugs that reach production are almost always caused by test data that did not include the edge cases. When you test query parameter encoding, include values with spaces, ampersands, hash characters, and international characters. When you test Base64, include binary data with null bytes and high-byte values. When you test HTML rendering, include inputs with angle brackets, ampersands, and script tags.
The free encoding toolkit by EvvyTools makes it fast to generate test cases across all encoding formats. Paste a value, check the encoded form in each format, and use those outputs as your test fixtures. The Encoding Toolkit supports URL encoding, HTML entities, Base64, Hex, and more with an auto-detection mode for decoding.
For a comprehensive reference on how each encoding format works with concrete examples, the guide on text encoding formats at EvvyTools covers Base64, URL encoding, HTML entities, Hex, Binary, Unicode escapes, JWT decoding, and ROT13 in a single place.
Summary
The encoding decisions in a REST API follow a consistent pattern:
- Query parameters: percent-encode values with
encodeURIComponentor equivalent - Request bodies: set the Content-Type header and encode the body to match
- Binary data: Base64-encode before including in JSON
- Tokens in URLs: use Base64URL, not standard Base64
- Client rendering: HTML-encode API strings before DOM insertion
- Boundary validation: check encoding validity at entry and exit points
Each rule applies to a specific context. Knowing which context you are working in makes the encoding choice automatic, and having a reliable tool to test the exact output of each transformation quickly removes most of the guesswork from API debugging and integration work.
Top comments (0)