Character Sets

Percent-encode a string

Encode an input string with percent-encoding using the utf8_percent_encode function from the percent-encoding crate. Then decode using the percent_decode function.

use percent_encoding::{utf8_percent_encode, percent_decode, AsciiSet, CONTROLS};
use std::str::Utf8Error;

/// https://url.spec.whatwg.org/#fragment-percent-encode-set
const FRAGMENT: &AsciiSet = &CONTROLS.add(b' ').add(b'"').add(b'<').add(b'>').add(b'`');

fn main() -> Result<(), Utf8Error> {
    let input = "confident, productive systems programming";

    let iter = utf8_percent_encode(input, FRAGMENT);
    let encoded: String = iter.collect();
    assert_eq!(encoded, "confident,%20productive%20systems%20programming");

    let iter = percent_decode(encoded.as_bytes());
    let decoded = iter.decode_utf8()?;
    assert_eq!(decoded, "confident, productive systems programming");

    Ok(())
}

The encode set defines which bytes (in addition to non-ASCII and controls) need to be percent-encoded. The choice of this set depends on context. For example, url encodes ? in a URL path but not in a query string.

The return value of encoding is an iterator of &str slices which collect into a String.

Encode a string as application/x-www-form-urlencoded

Encodes a string into application/x-www-form-urlencoded syntax using the form_urlencoded::byte_serialize and subsequently decodes it with form_urlencoded::parse. Both functions return iterators that collect into a String.

use url::form_urlencoded::{byte_serialize, parse};

fn main() {
    let urlencoded: String = byte_serialize("What is ❤?".as_bytes()).collect();
    assert_eq!(urlencoded, "What+is+%E2%9D%A4%3F");
    println!("urlencoded:'{}'", urlencoded);

    let decoded: String = parse(urlencoded.as_bytes())
        .map(|(key, val)| [key, val].concat())
        .collect();
    assert_eq!(decoded, "What is ❤?");
    println!("decoded:'{}'", decoded);
}

Encode and decode hex

The data_encoding crate provides a HEXUPPER::encode method which takes a &[u8] and returns a String containing the hexadecimal representation of the data.

Similarly, a HEXUPPER::decode method is provided which takes a &[u8] and returns a Vec<u8> if the input data is successfully decoded.

The recipe below converts &[u8] data to hexadecimal equivalent. Compares this value to the expected value.

use data_encoding::{HEXUPPER, DecodeError};

fn main() -> Result<(), DecodeError> {
    let original = b"The quick brown fox jumps over the lazy dog.";
    let expected = "54686520717569636B2062726F776E20666F78206A756D7073206F76\
        657220746865206C617A7920646F672E";

    let encoded = HEXUPPER.encode(original);
    assert_eq!(encoded, expected);

    let decoded = HEXUPPER.decode(&encoded.into_bytes())?;
    assert_eq!(&decoded[..], &original[..]);

    Ok(())
}

Generate OTP with a Base32 set

The data_encoding crate provides a BASE32_NOPAD::encode method which takes a &[u8] and returns a String containing the base32 representation of the data.

Similarly, a BASE32_NOPAD::decode method is provided which takes a &[u8] and returns a Vec<u8> if the input data is successfully decoded. There is also a BASE32_NOPAD_VISUAL::decode (resp. BASE32_NOPAD_NOCASE::decode) method which accepts characters outside base32 that can easily be confused with characters inside base32 and understands them accordingly (resp. accepts lowercase characters and understands them as their uppercase version).

The recipe below generates a human-readable one-time password by converting random bytes to base32. It then assumes the password was entered by a human with some visual errors. It finally makes sure the entered password decodes back to the initial random bytes.

use data_encoding::{BASE32_NOPAD, BASE32_NOPAD_VISUAL, DecodeError};

fn main() -> Result<(), DecodeError> {
    // Generate the password from random bytes.
    let initial_entropy = rand::random::<[u8; 10]>(); // 80 bits of entropy
    let shown_password = make_password(&initial_entropy);

    // Show the password to the user.
    println!("{shown_password}");

    // Read the password from the user (may have visual mistakes).
    let entered_password = alter_visually(&shown_password);
    println!("{entered_password}");

    // Reconstitute the random bytes (correcting the visual errors).
    let final_entropy = read_entropy(&entered_password)?;
    assert_eq!(final_entropy, initial_entropy);

    Ok(())
}

fn make_password(entropy: &[u8]) -> String {
    BASE32_NOPAD.encode(entropy)
}

fn read_entropy(password: &str) -> Result<Vec<u8>, DecodeError> {
    BASE32_NOPAD_VISUAL.decode(password.as_bytes())
}

fn alter_visually(input: &str) -> String {
    fn alter_char(c: char) -> char {
        match c {
            // O may be confused with 0
            'O' if rand::random::<bool>() => '0',
            // I may be confused with 1 or l
            'I' if rand::random::<bool>() => {
                if rand::random::<bool>() {
                    '1'
                } else {
                    'l'
                }
            }
            // B may be confused with 8
            'B' if rand::random::<bool>() => '8',
            _ => c,
        }
    }
    input.chars().map(alter_char).collect()
}

Encode and decode base64

Encodes byte slice into base64 String using encode and decodes it with decode.

use anyhow::Result;
use std::str;
use base64::prelude::{Engine as _, BASE64_STANDARD};

fn main() -> Result<()> {
    let hello = b"hello rustaceans";
    let encoded = BASE64_STANDARD.encode(hello);
    let decoded = BASE64_STANDARD.decode(&encoded)?;

    println!("origin: {}", str::from_utf8(hello)?);
    println!("base64 encoded: {}", encoded);
    println!("back to origin: {}", str::from_utf8(&decoded)?);

    Ok(())
}