🧪 Data Preparation & Updated Input Reference¶

This page documents the new preprocessing contract for EveNet. The reference schema in share/event_info/pretrain.yaml is bundled with the repository as an example, but you are free to adapt the feature lists to match your campaign. The goal of this guide is to help you build the .npz files that the EveNet converter ingests and to clarify how each tensor maps onto the model heads.

⚠️ Ownership reminder: Users are responsible for generating the .npz files. EveNet does not reorder or reshape features for you—the arrays must already follow the size and ordering implied by your event-info YAML. The converter simply validates the layout and writes the parquet outputs. Keep the YAML and the .npz dictionary in sync at all times.

• Config + CLI workflow
• Input tensor dictionary
• Supervision targets by head
• NPZ → Parquet conversion
• Runtime checklist

🛠️ Config + CLI Workflow¶

1. Start from an event-info YAML. The repository ships an example at share/event_info/pretrain.yaml; copy or extend it to describe the objects, global variables, and heads you plan to enable. The display names inside the INPUTS block are just labels used for logging and plotting—what matters is the order, which must match the tensor layout you write to disk.
2. Produce an event dictionary. For every event, assemble a Python dictionary that satisfies the shapes described below and append it to the archive you will write to disk. When you call numpy.savez (or savez_compressed), each key becomes an array with leading dimension N, the number of events in the file. Masks indicate which padded entries are valid and should contribute to the loss.
3. Run the EveNet converter. Point preprocessing/preprocess.py at your .npz bundle and pass the matching YAML so the loader can recover feature names, the number of sequential vectors, and the heads you are enabling. The converter assumes both artifacts describe the same structure—mismatches will surface as validation errors.
4. Train or evaluate. Training configs reference the resulting parquet directory via platform.data_parquet_dir and reuse the same YAML in options.Dataset.event_info.

✨ Normalization note. The normalize, log_normalize, and none tags in the YAML are metadata only. EveNet derives channel-wise statistics during conversion. The sole special case is normalize_uniform, which reserves a transformation for circular variables (φ); the model automatically maps to and from the wrapped representation.

📦 Input Tensor Dictionary¶

Each event is described by the following feature tensors. Shapes are shown with a leading N to indicate the number of stored events in a given .npz file. Masks share the same leading dimension as the value they gate.

🎯 Supervision Targets by Head¶

Only provide the tensors required for the heads you enable in your training YAML. Omit unused targets or set them to empty arrays so the converter skips unnecessary storage.

Classification Head¶

TruthGeneration Head¶

ReconGeneration Head¶

ReconGeneration is self-supervised: it perturbs the visible point-cloud channels and learns to denoise them. The target specification (which channels to regenerate) lives directly in the YAML under the ReconGeneration configuration. No additional tensors beyond the standard inputs are required.

Resonance Assignment Head¶

📐 Assignment internals: During conversion EveNet scans your assignment map to determine R and D, initialises arrays filled with -1, and then writes the actual child indices along with boolean masks. The snippet below mirrors the loader logic so you can generate matching tensors in your own pipeline:

full_indices = np.full((num_events, n_targets, max_daughters), -1, dtype=int)
full_mask = np.zeros((num_events, n_targets), dtype=bool)
index_mask = np.zeros((num_events, n_targets, max_daughters), dtype=bool)
# Fill with your resonance→daughter mappings; mark valid entries in the masks.

Segmentation Head¶

As with assignments, the converter computes S and T by scanning your segment-tag map. The internal helper creates zero-initialised arrays sized (num_events, max_event_segments, max_segment_tags) and fills them according to the resonance definitions. Any slots you leave unused remain associated with the catch-all background class (0).

If you disable the segmentation head, you can skip all four tensors.

Worked Input Example¶

import numpy as np

example = {
    "x": np.zeros((N, 18, 7), dtype=np.float32),  # fixed to (18, 7) for pretraining compatibility
    "x_mask": np.zeros((N, 18), dtype=bool),
    # Optional globals — drop both keys if unused
    "conditions": np.zeros((N, C), dtype=np.float32),
    "conditions_mask": np.ones((N, 1), dtype=bool),
    # Classification head (weights default to ones if omitted)
    "classification": np.zeros((N,), dtype=np.int32),
    "event_weight": np.ones((N,), dtype=np.float32),
    # Head-specific entries sized by your resonance/segment definitions
    "assignments-indices": np.full((N, R, D), -1, dtype=int),
    "assignments-mask": np.zeros((N, R), dtype=bool),
    "segmentation-data": np.zeros((N, S, 18), dtype=bool),
    # ... add heads you enabled ...
}

Feel free to adjust the head-specific dimensions (R, D, S, T) and the number of condition scalars C to match your physics process. The only fixed sizes are the point-cloud slots (18, 7) shared across datasets. Keep the YAML and the .npz dictionary in sync so the converter knows how many channels to expect and how to name them.

🔄 NPZ → Parquet Conversion¶

1. Assemble events into Python lists and save them with numpy.savez (or savez_compressed). Each key listed above becomes an array inside the archive.
2. Invoke the converter:

python preprocessing/preprocess.py \
  share/event_info/pretrain.yaml \
  --in_npz /path/to/events.npz \
  --store_dir /path/to/output \
  --cpu_max 32

The converter reads the YAML to recover feature names, masks, and head activation flags, then emits:

- `data_*.parquet` containing flattened tensors.
- `shape_metadata.json` with the original shapes (e.g., `(18, 7)` for `x`).
- `normalization.pt` with channel-wise statistics and class weights.

1. Inspect the logs. The script reports how many particles, invisible objects, and resonances were valid across the dataset—helpful when debugging mask alignment.

✅ Runtime Checklist¶

• platform.data_parquet_dir points to the folder with the generated parquet shards and shape_metadata.json.
• options.Dataset.event_info references the same YAML (share/event_info/pretrain.yaml or your copy).
• options.Dataset.normalization_file matches the normalization.pt produced during conversion.
• Only the heads you activated in the training YAML have matching supervision tensors in the parquet files.

With those pieces in place, EveNet will rebuild the full event dictionary on the fly, apply the appropriate circular normalization for normalize_uniform channels, and route each tensor to the corresponding head.