esc_json_lex.c File Reference

The first stage of reading an ESC profile from JSON: the text cut into tokens. More...

#include "esc_json_int.h"
#include <string.h>

Macros
#define	MAX_DEPTH   16
	Nesting deeper than any profile needs (five levels) is refused rather than followed, so a hostile file cannot run the stack down.
#define	MAX_MEMBERS   64u
	No profile object has more than 14 members.

Functions
static void	ws (lex_t *l)
	Skip white space.
static int64_t	add (lex_t *l, ttype_t t, uint32_t start)
	Add a token that starts at a byte.
static void	close_tok (lex_t *l, int64_t i, uint32_t end, uint32_t size)
	Set the end, size and next link of a token.
static bool	lex_value (lex_t *l, int depth)
	Lex one value of any type.
static int	hexval (char c)
	Value of one hexadecimal digit.
static int32_t	hex4 (const char *s)
	The four hex digits at s, or -1.
static uint32_t	utf8_len (const unsigned char *s, uint32_t avail)
	The length of the well-formed UTF-8 sequence at s, or 0: no stray continuation byte, no overlong form, no encoded surrogate, nothing past U+10FFFF.
static bool	lex_string (lex_t *l)
	A string, with every escape checked here, in the one pass that sees every string – including those in fields the panel never reads. \u0000 is refused (it would end a C string early), and so is half a surrogate pair: json.loads() keeps a lone half, which no UTF-8 can carry.
unsigned	esc_json_decode_unit (const char *s, uint32_t *i, uint8_t out[4])
	The UTF-8 bytes of the character at s[*i], into out; *i moves past it.
static bool	same_text (const char *s, const tok_t *a, const tok_t *b)
	Whether two string tokens' texts decode to the same bytes, so "schema" and "sch\u0065ma" are one key, as json.loads() has them.
bool	esc_json_text_is (const char *s, const tok_t *a, const char *lit)
	Whether a string token's text decodes to the plain lit.
static bool	is_digit (char c)
	Whether a character is an ASCII decimal digit.
static bool	lex_number (lex_t *l)
	-? (0 | [1-9][0-9]*) (. [0-9]+)? ([eE] [+-]? [0-9]+)?, as JSON has it and as the generator's json.loads() takes it, so a file one refuses the other refuses too, whichever field the number sits in.
static bool	lex_word (lex_t *l, const char *w, ttype_t t)
	Lex a literal word: true, false or null.
static bool	key_seen (const lex_t *l, uint32_t obj, uint32_t members)
	Whether the key just read is one of the members before it in the object at token obj. json.loads() keeps the last of two and a reader that stops at the first keeps the first, so a file with a key twice is refused rather than read two ways.
static bool	lex_container (lex_t *l, int depth, bool obj)
	Lex an object or an array.
bool	esc_json_lex_all (lex_t *l)
	Cut lexer l's text into tokens.
const char *	esc_json_lex_why_text (lex_why_t why)
	Why the lexer refused, as the text after "%s near byte".

Detailed Description

The first stage of reading an ESC profile from JSON: the text cut into tokens.

See esc_json.c for the second stage, and esc_json_int.h for the four functions this file gives it.

One token per value, each with the index one past its last descendant, so a reader skips a subtree in one step and never recurses to find a key. The reader runs this twice over the same text: once to count the tokens, once into an array of that size, where the duplicate-key check also runs. Every string is checked here, including those in fields the panel never reads.

The rules are tools/gen_esc_profiles.py's. The host suite parses every profile of record and compares it with the generated table, so a rule that differs between the two fails there.

SPDX-License-Identifier: MIT

Macro Definition Documentation

MAX_MEMBERS

#define MAX_MEMBERS   64u

No profile object has more than 14 members.

The duplicate check compares each key with those before it, so this bounds it at 64 x 63 / 2 comparisons an object rather than letting a 64KiB file of short keys ask for tens of millions.

Function Documentation

add()

int64_t add ( lex_t * l, ttype_t t, uint32_t start )

static

Add a token that starts at a byte.

Stores the token when the lexer has a token array; counts it always.

Parameters

l	The lexer state.
t	Type of the token.
start	First byte of the token.

Returns

Index of the token, or -1 when the array is full.

close_tok()

void close_tok ( lex_t * l, int64_t i, uint32_t end, uint32_t size )

static

Set the end, size and next link of a token.

Does nothing when the lexer only counts tokens.

Parameters

l	The lexer state.
i	Index of the token.
end	One past the last byte of the token.
size	Members of an object, elements of an array, or 0.

esc_json_decode_unit()

unsigned esc_json_decode_unit	(	const char *	s,
		uint32_t *	i,
		uint8_t	out[4] )

The UTF-8 bytes of the character at s[*i], into out; *i moves past it.

Only for text esc_json_lex_all() accepted. Returns the byte count, 1 to 4.

Parameters

s	The text.
i	Position of the character; moves past it.
out	Receives the UTF-8 bytes.

Returns

The byte count, 1 to 4.

esc_json_lex_all()

bool esc_json_lex_all

(

lex_t *

l

)

Cut lexer l's text into tokens.

With l->tok NULL it only counts them into l->n; with l->tok set to l->cap tokens it stores them and refuses a key an object already had. False on a refusal, with l->pos and l->why saying where and why.

Parameters

l	The lexer state.

Returns

true when the text was accepted.

esc_json_lex_why_text()

const char * esc_json_lex_why_text

(

lex_why_t

why

)

Why the lexer refused, as the text after "%s near byte".

Parameters

why	The refusal reason.

Returns

A static text.

esc_json_text_is()

bool esc_json_text_is	(	const char *	s,
		const tok_t *	a,
		const char *	lit )

Whether a string token's text decodes to the plain lit.

Parameters

s	The text.
a	The string token.
lit	The literal to compare with.

Returns

true when the decoded text equals lit.

hex4()

int32_t hex4 ( const char * s )

static

The four hex digits at s, or -1.

Parameters

s	The text; four digits are read.

Returns

The 16-bit value, or -1 when a digit is not hexadecimal.

hexval()

int hexval ( char c )

static

Value of one hexadecimal digit.

Parameters

c	The digit character.

Returns

0 to 15, or -1 when c is not a hexadecimal digit.

is_digit()

bool is_digit ( char c )

static

Whether a character is an ASCII decimal digit.

Parameters

c	The character.

Returns

true for '0' to '9'.

key_seen()

bool key_seen ( const lex_t * l, uint32_t obj, uint32_t members )

static

Whether the key just read is one of the members before it in the object at token obj. json.loads() keeps the last of two and a reader that stops at the first keeps the first, so a file with a key twice is refused rather than read two ways.

Only once the tokens exist: the counting pass has nothing to compare.

Parameters

l	The lexer state; the key is its newest token.
obj	Index of the object token.
members	Members of the object read before this key.

Returns

true when an earlier member has the same key.

lex_container()

bool lex_container ( lex_t * l, int depth, bool obj )

static

Lex an object or an array.

Parameters

l	The lexer state, at the opening bracket.
depth	Nesting depth of this container.
obj	true for an object, false for an array.

Returns

true when the container was lexed.

lex_number()

bool lex_number ( lex_t * l )

static

-? (0 | [1-9][0-9]*) (. [0-9]+)? ([eE] [+-]? [0-9]+)?, as JSON has it and as the generator's json.loads() takes it, so a file one refuses the other refuses too, whichever field the number sits in.

Parameters

l	The lexer state, at the first character of the number.

Returns

true when a well-formed number was lexed.

lex_string()

bool lex_string ( lex_t * l )

static

A string, with every escape checked here, in the one pass that sees every string – including those in fields the panel never reads. \u0000 is refused (it would end a C string early), and so is half a surrogate pair: json.loads() keeps a lone half, which no UTF-8 can carry.

What passes is what esc_json_decode_unit() decodes without looking again.

Parameters

l	The lexer state, at the opening quote; moves past the closing quote on success.

Returns

true when the string is well formed.

lex_value()

bool lex_value ( lex_t * l, int depth )

static

Lex one value of any type.

Parameters

l	The lexer state.
depth	Nesting depth of the value.

Returns

true when the value was lexed.

lex_word()

bool lex_word ( lex_t * l, const char * w, ttype_t t )

static

Lex a literal word: true, false or null.

Parameters

l	The lexer state.
w	The word to match.
t	The token type for the word.

Returns

true when the word matched.

same_text()

bool same_text ( const char * s, const tok_t * a, const tok_t * b )

static

Whether two string tokens' texts decode to the same bytes, so "schema" and "sch\u0065ma" are one key, as json.loads() has them.

The two decoded streams are compared byte by byte, not unit by unit: a raw "\xC3\xA9" comes a byte at a time and "\u00e9" as both bytes at once, and they are the same character.

Parameters

s	The text both tokens point into.
a	The first string token.
b	The second string token.

Returns

true when the decoded texts are equal.

utf8_len()

uint32_t utf8_len ( const unsigned char * s, uint32_t avail )

static

The length of the well-formed UTF-8 sequence at s, or 0: no stray continuation byte, no overlong form, no encoded surrogate, nothing past U+10FFFF.

What Python's UTF-8 decoder takes, so a file the generator would refuse to read is refused here too.

Parameters

s	The bytes, starting at the lead byte.
avail	Bytes available from s.

Returns

The sequence length, 2 to 4, or 0 when it is not well formed.

ws()

void ws ( lex_t * l )

static

Skip white space.

Skips space, tab, LF and CR.

Parameters

l	The lexer state; its position moves past the white space.