log_csv.h File Reference

CSV (comma-separated values) and TSV (tab-separated values) import for a board that cannot hold the file it is reading. More...

#include <stdbool.h>
#include <stddef.h>
#include "log_numbers.h"

Go to the source code of this file.

Data Structures
struct	log_source_t
	A byte source that can be read from the top more than once. More...
struct	log_mem_ctx_t
	State of the memory source. More...
struct	log_reader_t
	Streaming row splitter: quotes, doubled quotes, and CRLF. More...
struct	log_column_t
	Analysis of one column. More...
struct	log_analysis_t
	Result of the analysis of a whole file. More...
struct	log_csv_opts_t
	Options of the CSV import. More...
struct	log_field_t
	One plotted field. More...
struct	log_data_t
	Materialised data of the selected columns. More...

Macros
#define	LOG_MAX_COLUMNS   25
	Ceilings.
#define	LOG_MAX_SERIES   4
	Most series the build materialises.
#define	LOG_NAME_MAX   28
	Room for a column name, bytes, with its terminator.
#define	LOG_SAMPLE_MAX   16
	Room for one sample cell, bytes, with its terminator.
#define	LOG_SAMPLES   3
	Number of sample cells kept per column.
#define	LOG_ROW_MAX   1024
	Most bytes in one row.
#define	LOG_DEFAULT_MAX_ROWS   60000
	Row cap used when the options give 0, rows.
#define	LOG_TIME_AUTO   (-2)
	time_index value that detects the time column.

Enumerations
enum	log_err_t {   LOG_OK = 0 , LOG_ERR_READ , LOG_ERR_NO_HEADER , LOG_ERR_NO_ROWS ,   LOG_ERR_NO_NUMERIC , LOG_ERR_TOO_MANY_COLUMNS , LOG_ERR_MEMORY , LOG_ERR_ARG }
	Result of a CSV import step.

Functions
void	log_source_memory (log_source_t *src, log_mem_ctx_t *ctx, const char *text, size_t len)
	A source over a block of memory.
void	log_source_stdio (log_source_t *src, void *file)
	A source over an open stdio stream. file is a FILE*.
void	log_reader_init (log_reader_t *r, log_source_t *src)
	Set up a row splitter.
int	log_reader_row (log_reader_t *r, char delimiter, char *store, size_t store_size, char **fields, int max_fields, int *total_fields)
	Read one row.
const char *	log_err_str (log_err_t err)
	Text for an import error.
void	log_csv_opts_default (log_csv_opts_t *opts)
	Set the default import options.
const char *	log_csv_delimiter_label (char delimiter)
	Label of a delimiter.
char	log_csv_sniff (const char *sample, size_t len)
	Guess the delimiter from a text sample: the most consistent, widest split.
log_err_t	log_csv_analyse (log_source_t *src, const log_csv_opts_t *opts, log_analysis_t *out)
	Two streaming passes: geometry and evidence, then per-column statistics.
log_err_t	log_csv_build (log_source_t *src, const log_analysis_t *a, const int *columns, int n_columns, log_data_t *out)
	Materialise the selected columns.
void	log_data_free (log_data_t *data)
	Free the sample arrays of a build.
void	log_set_allocator (void *(*alloc_fn)(size_t), void(*free_fn)(void *))
	Where the sample arrays come from.

Variables
const char	log_csv_delimiters [4]
	The candidate delimiters, in the order they are scored.

Detailed Description

CSV (comma-separated values) and TSV (tab-separated values) import for a board that cannot hold the file it is reading.

The source is a card, a log can be tens of megabytes, and the 8MB of PSRAM (pseudo-static random-access memory) already hold two framebuffers. So the reader works through a callback and never keeps more than one row: analysis is two streaming passes, and only the columns the user plots are materialised as arrays.

Two-phase: log_csv_analyse reports what it detected without committing to it, so the import screen can show a preview and let the user override the delimiter, the decimal convention and the time column before log_csv_build runs.

Pure C, no ESP-IDF.

SPDX-License-Identifier: MIT

Macro Definition Documentation

LOG_MAX_COLUMNS

#define LOG_MAX_COLUMNS   25

Ceilings.

25 columns: the fullest file the bench writes, a SERVO run's with three INA3221 groups (log_writer.h); the import view scrolls its list. A wider file is refused, and reporting the limit beats a silent truncation.

Function Documentation

log_csv_analyse()

log_err_t log_csv_analyse	(	log_source_t *	src,
		const log_csv_opts_t *	opts,
		log_analysis_t *	out )

Two streaming passes: geometry and evidence, then per-column statistics.

Parameters

src	The input.
opts	The import options.
out	Receives the analysis.

Returns

LOG_OK, or the reason the analysis failed.

log_csv_build()

log_err_t log_csv_build	(	log_source_t *	src,
		const log_analysis_t *	a,
		const int *	columns,
		int	n_columns,
		log_data_t *	out )

Materialise the selected columns.

Parameters

src	The input.
a	The analysis.
columns	column indices to plot, or NULL to take the first numeric ones that are not the time axis
n_columns	Number of entries in columns.
out	Receives the data.

Returns

LOG_OK, or the reason the data was not built.

log_csv_delimiter_label()

const char * log_csv_delimiter_label

(

char

delimiter

)

Label of a delimiter.

Parameters

delimiter

The delimiter.

Returns

A static text.

log_csv_opts_default()

void log_csv_opts_default

(

log_csv_opts_t *

opts

)

Set the default import options.

Parameters

opts	Receives the defaults.

log_csv_sniff()

char log_csv_sniff	(	const char *	sample,
		size_t	len )

Guess the delimiter from a text sample: the most consistent, widest split.

Parameters

sample	The sample text.
len	Length of sample, bytes.

Returns

The delimiter.

log_data_free()

void log_data_free

(

log_data_t *

data

)

Free the sample arrays of a build.

Parameters

data	The data to free.

log_err_str()

const char * log_err_str

(

log_err_t

err

)

Text for an import error.

Parameters

err	The error.

Returns

A static text.

log_reader_init()

void log_reader_init	(	log_reader_t *	r,
		log_source_t *	src )

Set up a row splitter.

Parameters

r	The splitter.
src	The input.

log_reader_row()

int log_reader_row	(	log_reader_t *	r,
		char	delimiter,
		char *	store,
		size_t	store_size,
		char **	fields,
		int	max_fields,
		int *	total_fields )

Read one row.

A row ends at its line feed (0x0A). Input that ends inside a row – a write cut off by a power loss or a full card – leaves a row nothing proves complete: a number cut after two of its four digits still parses, and the field count can still match the header. That row is not returned: the call answers -1 as at the end of input and sets r->cut.

Parameters

r	The splitter.
delimiter	The field delimiter.
store	scratch for the field text; fields point into it
store_size	Size of store, bytes.
fields	receives one pointer per field, up to max_fields
max_fields	Capacity of fields, in pointers.
total_fields	receives the true field count, which may exceed max_fields; that is how a ragged row is spotted

Returns

the number of fields stored, or -1 at end of input

log_set_allocator()

void log_set_allocator	(	void *	alloc_fn (*)(size_t),
		void(*	free_fn )(void *) )

Where the sample arrays come from.

The firmware points this at PSRAM; the default is malloc/free, which is what the host tests want.

Parameters

alloc_fn	Allocates a block of bytes.
free_fn	Frees a block from alloc_fn.

log_source_memory()

void log_source_memory	(	log_source_t *	src,
		log_mem_ctx_t *	ctx,
		const char *	text,
		size_t	len )

A source over a block of memory.

Parameters

src	Receives the source.
ctx	Holds the read position; set to the start of text.
text	The bytes read.
len	Length of text, bytes.

log_source_stdio()

void log_source_stdio	(	log_source_t *	src,
		void *	file )

A source over an open stdio stream. file is a FILE*.

Parameters

src	Receives the source.
file	An open stream, a FILE *.