TokMesh Lite — Session 01 Field Notes

01 / 10

What we achieved

TokMesh Lite can now express a categorical rule in YAML and enforce it against CSV rows. It also rejects a contract that declares the rule without listing any permitted values.

The new contract vocabulary

quality:
  - column: status
    rule: accepted_values
    values:
      - operating
      - suspended
      - closed

In plain language: the status column may contain only one of those three strings. A value such as unknown creates a validation finding.

Why this matters for a data product. The expectation is no longer trapped in prose or in someone’s memory. It travels with the product contract and can be checked by a person, a local command, or a future automated pipeline.

02 / 10

Our starting proof

Before adding a new rule, we proved that the initial vertical slice already handled valid products, valid data, and invalid contracts.

Valid product contract

$ cargo run -- product validate examples/copper-mines/product.yaml
Finished `dev` profile [unoptimized + debuginfo]
Running `target/debug/tokmesh-lite product validate ...`
VALID — no validation issues found

Valid CSV against its contract

$ cargo run -- data validate \
    examples/copper-mines/product.yaml \
    examples/copper-mines/data.csv
Finished `dev` profile [unoptimized + debuginfo]
Running `target/debug/tokmesh-lite data validate ...`
VALID — no validation issues found

Deliberately invalid contract

$ cargo run -- product validate \
    examples/invalid-products/duplicate-columns.yaml

INVALID — 3 issue(s)
- [product.version.invalid] version: version must follow semantic versioning,
  for example 1.0.0
- [data.column.name.duplicate] data.schema[1].name:
  column name 'mine_id' appears more than once
- [quality.column.unknown] quality[0].column:
  quality rule references unknown column 'missing_column'

The terminal’s non-zero status was intentional. Validation completed correctly and found contract violations. Exit code 1 lets automation stop an invalid product.

03 / 10

What TDD means

Test-Driven Development describes one missing behavior before implementing it. The failure becomes evidence that the test can see the missing capability; the smallest implementation then turns that same test green.

NeedState the data-product expectation.
TestObserve behavior through a public seam.
REDConfirm failure for the expected reason.
GREENAdd only enough Rust to satisfy it.
VerifyRun formatting, linting, and all tests.

The public seams we tested

Interface	What a caller gets	Why we test here
`load_manifest(&path)`	YAML becomes a typed `ProductManifest`, or a useful error.	Tests use the same loading path as the CLI.
`validate_product(&manifest)`	A report about whether the contract itself is sensible.	An empty accepted-values list is a contract problem.
`validate_data(&manifest, &path)`	A report about whether CSV data obeys the contract.	An unlisted row value is a dataset problem.

We did not test private helpers. Tests should survive an internal rewrite as long as the caller-visible promises remain true.

04 / 10

Cycle one: reject an unlisted value

Dataset behavior

unknown must fail when only three statuses are accepted.

The behavioral test

#[test]
fn data_validation_rejects_value_outside_accepted_values() {
    let directory =
        tempfile::tempdir().expect("temporary directory should be created");
    let manifest_path = directory.path().join("product.yaml");
    let data_path = directory.path().join("data.csv");

    std::fs::write(
        &manifest_path,
        r"
id: tokmesh.ng-lithium-sites
name: Nigeria Lithium Sites
version: 1.0.0
owner:
  country: Nigeria
  organization: TokMesh Learning Laboratory
data:
  format: csv
  schema:
    - name: site_id
      type: string
      required: true
    - name: status
      type: string
      required: true
quality:
  - column: status
    rule: accepted_values
    values:
      - operating
      - suspended
      - closed
sovereignty:
  classification: national
  allowed_countries:
    - Nigeria
",
    )
    .expect("manifest fixture should be written");

    std::fs::write(
        &data_path,
        "site_id,status\nNG-LI-000001,operating\nNG-LI-000002,unknown\n",
    )
    .expect("CSV fixture should be written");

    let manifest = load_manifest(&manifest_path).expect("manifest should parse");
    let report =
        validate_data(&manifest, &data_path).expect("CSV should be readable");

    assert_eq!(report.issues.len(), 1);
    assert_eq!(
        report.issues[0].code,
        "quality.accepted_values.rejected"
    );
    assert_eq!(report.issues[0].location, "row[3].status");
}

RED: the vocabulary did not exist yet

FAILED
manifest should parse: Manifest {
  source: Error(
    "quality[0].rule: unknown variant `accepted_values`,
     expected `unique` or `range`"
  )
}

This was a useful RED: YAML parsing reached the exact new rule and proved that TokMesh did not already support it.

GREEN part A: teach the manifest model

pub struct QualityRule {
    pub column: String,
    pub rule: RuleType,
    #[serde(default)]
    pub minimum: Option<f64>,
    #[serde(default)]
    pub maximum: Option<f64>,
    #[serde(default)]
    pub values: Vec<String>,
}

#[serde(rename_all = "lowercase")]
pub enum RuleType {
    Unique,
    Range,
    #[serde(rename = "accepted_values")]
    AcceptedValues,
}

GREEN part B: enforce the rule for each row

RuleType::AcceptedValues => {
    if !rule.values.iter().any(|accepted| accepted == value) {
        report.error(
            "quality.accepted_values.rejected",
            &location,
            format!(
                "value '{value}' is not one of: {}",
                rule.values.join(", ")
            ),
        );
    }
}

Read this Rust aloud. Borrow each accepted value with .iter(); ask whether .any(...) equals the CSV value; negate that answer with !; report an issue when none match.

GREEN output

$ cargo test --test validation \
    data_validation_rejects_value_outside_accepted_values \
    -- --exact

running 1 test
test data_validation_rejects_value_outside_accepted_values ... ok

test result: ok. 1 passed; 0 failed; 0 ignored;
2 filtered out; finished in 0.00s

05 / 10

Cycle two: reject an empty declaration

Contract behavior

values: [] must fail before a CSV is read.

The behavioral test

#[test]
fn product_validation_rejects_empty_accepted_values() {
    let directory =
        tempfile::tempdir().expect("temporary directory should be created");
    let manifest_path = directory.path().join("product.yaml");

    std::fs::write(
        &manifest_path,
        r"
id: tokmesh.ng-lithium-sites
name: Nigeria Lithium Sites
version: 1.0.0
owner:
  country: Nigeria
  organization: TokMesh Learning Laboratory
data:
  format: csv
  schema:
    - name: site_id
      type: string
      required: true
    - name: status
      type: string
      required: true
quality:
  - column: status
    rule: accepted_values
    values: []
sovereignty:
  classification: national
  allowed_countries:
    - Nigeria
",
    )
    .expect("manifest fixture should be written");

    let manifest = load_manifest(&manifest_path).expect("manifest should parse");
    let report = validate_product(&manifest);

    assert_eq!(report.issues.len(), 1);
    assert_eq!(
        report.issues[0].code,
        "quality.accepted_values.empty"
    );
    assert_eq!(
        report.issues[0].location,
        "quality[0].values"
    );
}

RED output captured during the session

$ cargo test --test validation \
    product_validation_rejects_empty_accepted_values \
    -- --exact

running 1 test
test product_validation_rejects_empty_accepted_values ... FAILED

assertion `left == right` failed
  left: 0
 right: 1

test result: FAILED. 0 passed; 1 failed;
3 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test validation`

The manifest parsed and product validation ran. The only mismatch was behavioral: TokMesh found zero issues while our specification required one.

The minimal implementation

if rule.rule == RuleType::AcceptedValues && rule.values.is_empty() {
    report.error(
        "quality.accepted_values.empty",
        format!("quality[{index}].values"),
        "accepted-values rules require at least one value",
    );
}

Expression	Meaning
`rule.rule == RuleType::AcceptedValues`	The current quality rule is the new categorical rule.
`&&`	Both the left and right conditions must be true.
`rule.values.is_empty()`	The owned vector contains zero permitted values.
`format!("quality[{index}].values")`	Build a precise location such as `quality[0].values`.

GREEN output captured during the session

$ cargo test --test validation \
    product_validation_rejects_empty_accepted_values \
    -- --exact

running 1 test
test product_validation_rejects_empty_accepted_values ... ok

test result: ok. 1 passed; 0 failed; 0 ignored;
3 filtered out; finished in 0.00s

06 / 10

Where the behavior lives

One YAML rule crossed four files while preserving the two small validation interfaces used by callers.

src/product/manifest.rs Defines the typed vocabulary: values and AcceptedValues.

src/product/validate.rs Checks whether the declaration itself makes sense.

src/data/quality.rs Checks whether each non-empty, correctly typed CSV value obeys the declaration.

tests/validation.rs Specifies both promises through public interfaces.

product.yaml
    │
    ▼
load_manifest() ───────► ProductManifest
                              │
                    ┌─────────┴─────────┐
                    ▼                   ▼
          validate_product()     validate_data()
          Is the contract        Does the CSV obey
          internally valid?      the contract?
                    │                   │
                    └─────────┬─────────┘
                              ▼
                     ValidationReport
                   code · location · message

07 / 10

Repository-wide verification

Strict Clippy run

$ cargo clippy --all-targets --all-features -- -D warnings

Checking tokmesh-lite v0.1.0
Finished `dev` profile [unoptimized + debuginfo]

Result: passed with zero warnings.

Complete test suite

$ cargo test

tests/cli.rs
  invalid_product_command_uses_validation_exit_code ... ok
  valid_product_command_exits_successfully ........... ok

tests/validation.rs
  invalid_example_reports_stable_issue_codes ........ ok
  valid_example_manifest_and_data_pass .............. ok
  product_validation_rejects_empty_accepted_values .. ok
  data_validation_rejects_value_outside_accepted_values
                                                     ... ok

Doc-tests tokmesh_lite
  0 tests; ok

Overall: 6 executable tests passed; 0 failed.

Why did the complete suite print “INVALID — 3 issue(s)”?

One CLI test deliberately runs an invalid manifest and confirms that the command returns exit code 1. Seeing the invalid report inside a passing test run is expected: the test passed because the command failed validation exactly as designed.

Completion evidence. The feature passed its focused tests, all earlier behavior remained green, formatting completed, and strict Clippy reported no warnings.

08 / 10

Cargo command glossary

Cargo is Rust’s project tool. It downloads declared dependencies, compiles code, runs programs, executes tests, and coordinates other development checks.

Command	What it does	How to read success
`cargo run -- …`	Compiles the development build when needed, then runs TokMesh Lite. Everything after `--` is passed to TokMesh rather than interpreted by Cargo.	The program prints its report and returns exit code 0, 1, or 2.
`cargo fmt`	Rewrites Rust files into the standard style chosen by `rustfmt`. This removes debates about spacing and line wrapping.	Usually prints nothing. Files may be changed mechanically.
`cargo fmt --check`	Checks formatting without rewriting files. Useful before a commit and in CI.	No output and exit code 0 means everything is formatted.
`cargo test`	Builds test targets and runs the complete suite: library, binary, integration, CLI, and documentation test harnesses.	Every section ends with `test result: ok`.
`cargo test --test validation NAME -- --exact`	Runs one named test from `tests/validation.rs`. The second `--` sends `--exact` to Rust’s test runner.	One test runs; the others are reported as filtered out.
`cargo clippy`	Runs Rust-aware static analysis. Clippy finds suspicious, needlessly complex, or unidiomatic code beyond what compilation alone reports.	It finishes without warnings or errors.
`--all-targets`	Checks the library, executable, tests, examples, and benchmarks that exist in the package.	No target is silently omitted from linting.
`--all-features`	Enables every declared Cargo feature while checking. TokMesh Lite currently has no optional feature set, but the gate remains future-safe.	All feature combinations represented by “all enabled” compile cleanly.
`-- -D warnings`	Passes `-D warnings` to Clippy. “Deny warnings” makes any warning fail the command, preventing warning debt.	Exit code 0 means there were no warnings.
`cargo build --release`	Creates an optimized binary under `target/release/`. We will use this for million-row performance measurements.	The release profile finishes successfully.

Common output phrases

Phrase	Meaning
`Compiling`	Cargo is translating changed Rust source or dependencies into machine code.
`Finished dev profile`	The unoptimized, debugging-friendly build is ready.
`Finished in 0.00s`	The measured test execution was extremely quick; compilation time is reported separately.
`filtered out`	Tests existed but were intentionally skipped because a name filter selected one test.
`exit code 0`	Execution succeeded and validation passed.
`exit code 1`	Execution succeeded but the product or dataset was invalid.
`exit code 2`	TokMesh could not complete because input could not be read, parsed, or serialized.
`exit code 101`	The Rust test process had at least one failing test; expected during a deliberate RED step.

09 / 10

Git checkpoint

The completed feature became one focused commit and was pushed to the tracked main branch.

$ git status
$ git diff
$ git add src/product/manifest.rs
$ git add src/product/validate.rs
$ git add src/data/quality.rs
$ git add tests/validation.rs
$ git diff --cached
$ git commit -m "feat: add accepted-values quality validation"
$ git push

Recorded result

da81912 feat: add accepted-values quality validation

At documentation closeout, local main and origin/main both pointed to this commit.

Git command	Purpose
`git status`	Shows the branch and which files are modified, staged, or untracked.
`git diff`	Shows unstaged line-by-line changes.
`git add FILE`	Places one chosen file into the next commit.
`git diff --cached`	Shows exactly what the next commit will contain.
`git commit -m "…"`	Records the staged snapshot locally with a descriptive message.
`git push`	Sends local commits to the configured GitHub remote.

Resume here

Next session: a runnable Nigeria lithium product

Review and commit this session’s documentation.
Create a small, clearly labelled synthetic product manifest.
Create one valid and one deliberately invalid CSV.
Validate both through the CLI and record their outputs.
Only then design the deterministic large-dataset generator.

The scale lesson will generate 10K, 100K, 1M, and later 5M rows locally. Generated CSV files will stay out of Git. We will measure release-build duration, rows per second, file size, findings, and peak memory rather than guessing about performance.