Utf8

Utf8 :: # (opaque)

Incremental decoding of arbitrary UTF-8 byte chunks.

Use this module at file, network, and other untrusted-byte boundaries. A valid Roc Str needs no decode error channel; use Scalar.iter there.

Problem

:= [
    InvalidStartByte,
    UnexpectedEndOfSequence,
    ExpectedContinuation,
    OverlongEncoding,
    EncodesSurrogateHalf,
    CodePointTooLarge,
]

is_eq : _

Compare two stable malformed-input classifications.

DecodeError : {
    problem : Problem,
    offset : U64,
    sequence_start : U64,
}

A malformed sequence with absolute logical-source coordinates.

sequence_start identifies its leading byte. offset identifies the byte where malformed input was detected, or the logical end offset where a continuation byte was missing. For errors concerning the completed value, both offsets equal sequence_start.

Cursor

Utf8.Cursor :: # (opaque)

A raw UTF-8 stream decoder with at most three pending bytes.

The cursor is sealed and must be explicitly finished. Its representation keeps only a partial scalar, absolute coordinates, and terminal status; a returned cursor never retains a consumed chunk.

init : {  } -> Cursor

Begin decoding at byte offset and scalar index zero.

This is constant time and does not allocate.

push : Cursor, List(U8), state, (state, LocatedScalar -> [Continue(state), Stop(state)]) -> [
    Pushed({ cursor : Cursor, state : state, consumed : U64 }),
    Stopped({ cursor : Cursor, state : state, consumed : U64 }),
    Failed({ cursor : Cursor, state : state, consumed : U64, error : Error }),
]

Decode one arbitrary byte chunk and fold every completed scalar into caller state.

The callback returns Continue or Stop. Stopped returns at the first requested scalar boundary without decoding the chunk suffix; consumed is the number of bytes accepted from this chunk, so the caller can resume with chunk.drop_first(consumed). The cursor does not retain that remainder. Pushed means the whole chunk was accepted but does not mean end of text. Failed is terminal and returns the state containing any earlier, irrevocably decoded scalars; callers must not present that state as a complete decode. Every result's consumed is relative to this chunk. On malformed input it excludes the offending byte, while earlier bytes of a partial sequence remain consumed by the terminal cursor.

Work is O(B) in chunk bytes with constant stack and auxiliary state. The decoder allocates no list or string and does not retain chunk. The callback controls any allocation in its own state.

finish : Cursor -> [
    End({ cursor : Cursor, byte_offset : U64, scalar_count : U64 }),
    Failed({ cursor : Cursor, error : Error }),
]

Explicitly mark the logical end of the byte source.

End reports the absolute byte length and decoded scalar count. An incomplete trailing sequence becomes UnexpectedEndOfSequence at the end offset. Calling finish twice, finishing a failed cursor, or pushing after either terminal state returns a typed state error. This is constant time and does not allocate.