Getting Started

FlatMessage is a zero-copy, schema-less serialization library built for Rust, offering efficient and flexible data serialization.

It is designed to be fast, memory-efficient, and easy to use, making it suitable for a wide range of applications, from simple data storage to complex network protocols.

A schema-less library means that you don't need to define a schema before you can start serializing and deserializing data (all necessary information is stored in the data itself). This means that the output buffer is larger than the total size of the data, but you are more flexible in scenarios where your data changes often.

A zero-copy library means that you don't need to copy the data from the serialized buffer; you can use references and slices to access the data directly. This is useful for larger datasets where copying them would add a performance penalty.

Installation

This chapter will guide you through installing and setting up FlatMessage in your Rust project.

System Requirements

FlatMessage requires:

• Rust: Version 1.70 or later (2021 edition)
• Operating System: Cross-platform (Windows, macOS, Linux)
• Architecture: Support for 32-bit and 64-bit systems

Adding FlatMessage to Your Project

The easiest way to add FlatMessage to your project is through Cargo. Add the following dependency to your Cargo.toml file:

[dependencies]
flat_message = "*"

Use it

To use FlatMessage, define a structure and derive it from FlatMessage like in the following example:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
struct TestMessage {
    // fields
}
}

Deriving FlatMessage

To make a struct serializable with FlatMessage, simply add #[derive(FlatMessage)]:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
struct Person {
    name: String,
    age: u32,
    active: bool,
}
}

This automatically generates the necessary code to implement the FlatMessage trait for your struct.

Structure-Level Configuration (Optional)

You can configure the entire structure using the #[flat_message_options(...)] attribute in the following way:

#[flat_message_options(option-1 : value-1, option-2 : value-2, ...)]

or

#[flat_message_options(option-1 = value-1, option-2 = value-2, ...)]

For example:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = false, version = 1, checksum = true)]
struct OptimizedStruct {
    data: Vec<u8>,
}
}

Available Structure Options (Optional)

Remarks:

• The store_name option does not store the actual structure name, but a hash of it. That hash is being used to check if the structure you are deserializing into is the same as the one you serialized. However, this is not always neccesary (especially when talking about versioning and compabibility). If this is not needed, you should set the store_name option to false to save some space on the serialized buffer.
• You can read more about versioning and compabibility in the Versioning chapter.
• The checksum option is set to false by default. If set to true, the CRC32 checksum of the serialized data is being calculated and stored in the serialized buffer. This is useful to ensure data integrity during deserialization (usually when you are sending data over a network). You can read more on how checksums and validation work in the Checksum and Validation chapter.

Field-Level Configuration

Individual fields can be configured using #[flat_message_item(...)] in the following way:

#[flat_message_item(option-1 : value-1, option-2 : value-2, ...)]

or

#[flat_message_item(option-1 = value-1, option-2 = value-2, ...)]

This is useful when you want to specify how a field should be serialized / deserialized. The following example shows how to use the #[flat_message_item(repr = u8, kind = enum)] attribute to serialize an enum as a u8 value.

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Example {
    normal_field: u32,
    
    #[flat_message_item(repr = u8, kind = enum)]
    status: Status,
}

#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Status {
    Active = 1,
    Inactive = 2,
}
}

Remarks:

• You can read more on how enums are being serialized in the Enums chapter or the Flags chapter.

Field Options

Remarks:

• Fields of type PhantomData<T> are automatically ignored during serialization:

  #![allow(unused)]
  fn main() {
  use std::marker::PhantomData;
  
  #[derive(FlatMessage)]
  struct GenericStruct<T> {
      data: u32,
      _phantom: PhantomData<T>,  // This field is ignored
  }
  }

• You can use the ignore or skip option to ignore a field during serialization and deserialization. This is useful when you want to skip a field that is not part of the structure you are deserializing into. The fields have to implement the Default trait.

  #![allow(unused)]
  fn main() {
  use std::marker::PhantomData;
  
  #[derive(Default)]
  struct MyNonSerializableData {
      a: u8,
      b: u32,
  }
  #[derive(FlatMessage)]
  #[flat_message_options(store_name = false)]
  struct Data {
      x: u8,
      #[flat_message_item(ignore = true)]
      y: MyNonSerializableData,
  }
  }

• Mandatory fields are required for deserialization. If a mandatory field is not present in the serialized data, the deserialization will fail. On the other hand, if a field is not mandatory, and it is not found in the serialized data or there are some issues trying to deserialize it, it will be defaulted to the default value of the type. This implies that the trait Default is implemented for that type.
• By default, fields are mandatory, except for Option<T> fields that are marked with mandatory = false (unless you specify the attribute #[flat_message_item(mandatory = true)]).
• The default option can be used to provide a default value for a field. This is useful when you want to provide a default value for a field that is not mandatory. The default value can be a string literal, a function call, or a macro call.

  #![allow(unused)]
  fn main() {
  #[derive(FlatMessage)]
  #[flat_message_options(store_name = false)]
  struct MyDataV1 {
      a: u8,
  }
  #[derive(FlatMessage)]
  #[flat_message_options(store_name = false)]
  struct MyDataV2 {
      a: u8,
      // if the field is not present in the serialized data, it will be defaulted to vec![1,2,3]
      #[flat_message_item(mandatory = false, default = "vec![1,2,3]")]
      b: Vec<u8>,
  }
  }

Generated Code

When you derive FlatMessage, the following methods are automatically implemented:

#![allow(unused)]
fn main() {
impl<'a> FlatMessage<'a> for YourStruct {
    fn serialize_to(&self, output: &mut Storage, config: Config) -> Result<(), Error>;
    fn deserialize_from(input: &'a Storage) -> Result<Self, Error>;
    unsafe fn deserialize_from_unchecked(input: &'a Storage) -> Result<Self, Error>;
}
}

Complete Example

use flat_message::*;

#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Priority {
    Low = 1,
    Medium = 2,
    High = 3,
}

#[derive(FlatMessage, Debug, PartialEq)]
#[flat_message_options(version = 1, store_name = true, checksum = true)]
struct Task {
    title: String,
    description: Option<String>,
    completed: bool,
    
    #[flat_message_item(repr = u8, kind = enum)]
    priority: Priority,
    
    tags: Vec<String>,
    created: Timestamp,
    id: UniqueID,
}

fn main() {
    let task = Task {
        title: "Learn FlatMessage".to_string(),
        description: Some("Read the documentation".to_string()),
        completed: false,
        priority: Priority::High,
        tags: vec!["learning".to_string(), "rust".to_string()],
    };

    // Create a serialization storage buffer
    let mut storage = Storage::default();
    if let Err(e) = task.serialize_to(&mut storage, Config::default()) {
        panic!("Error serializing task: {}", e);
    }

    // print the buffer
    println!("Buffer: {:?}", storage.as_slice());

    // Deserialize from buffer
    match Task::deserialize_from(&storage) {
        Ok(restored_task) => {
            assert_eq!(task, restored_task);
            println!("Task serialized and deserialized successfully");
        }
        Err(e) => {
            panic!("Error deserializing task: {}", e);
        }
    }
}

Upon execution, the following output is being printed:

Buffer: [70, 76, 77, 1, 5, 0, 1, 12, 22, 82, 101, 97, 100, 32, 116, 104, 101, 
         32, 100, 111, 99, 117, 109, 101, 110, 116, 97, 116, 105, 111, 110, 113, 
         190, 208, 155, 3, 17, 76, 101, 97, 114, 110, 32, 70, 108, 97, 116, 77, 
         101, 115, 115, 97, 103, 101, 2, 8, 108, 101, 97, 114, 110, 105, 110, 103, 
         4, 114, 117, 115, 116, 0, 0, 0, 14, 59, 111, 52, 19, 227, 228, 148, 14, 
         181, 101, 152, 142, 235, 22, 244, 13, 129, 116, 244, 8, 31, 36, 54, 69, 
         108, 136, 72, 244, 245, 75, 146, 228]
Task serialized and deserialized successfully

Core Concepts

FlatMessage relies on a few core concepts / components that are used to build the serialization and deserialization logic.

Storage

Storage is FlatMessage's primary buffer type for holding serialized data. It provides efficient memory management optimized for serialization workloads.

#![allow(unused)]
fn main() {
use flat_message::*;

// Using Storage for serialization
let mut storage = Storage::default();
data.serialize_to(&mut storage, Config::default())?;
}

Storage advantages:

• Memory alignment: Uses Vec<u128> internally for better alignment
• Reduced allocations: More efficient memory growth patterns
• Optimized for serialization: Designed specifically for FlatMessage workloads

Storage API

#![allow(unused)]
fn main() {
impl Storage {
    // Create from existing byte buffer 
    pub fn from_buffer(input: &[u8]) -> Storage;
    
    // Create with a given capacity filled with zeros
    pub fn with_capacity(capacity: usize) -> Storage;

    // Get current size
    pub fn len(&self) -> usize;
}
}

Storage also implements Default trait, which creates an empty storage (with no allocated memory).

Config

Config controls serialization behavior and constraints:

#![allow(unused)]
fn main() {
use flat_message::*;

// Default configuration
let config = Config::default();

// Custom configuration
let config = ConfigBuilder::new()
    .max_size(1024 * 1024)  // 1MB limit
    .build();
}

Configuration Options

You can use ConfigBuilder to create a Config instance and provide a set of options (on how the serialization should be performed).

Using Config

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Data {
    content: Vec<u8>,
}

let data = Data { content: vec![1, 2, 3] };
let mut storage = Storage::default();

// Use custom size limit
let config = ConfigBuilder::new()
    .max_size(1024)  // Only allow 1KB
    .build();

match data.serialize_to(&mut storage, config) {
    Ok(()) => println!("Serialization successful"),
    Err(Error::ExceedMaxSize((actual, max))) => {
        println!("Data too large: {} bytes (max: {})", actual, max);
    }
    Err(e) => println!("Other error: {}", e),
}
}

FlatMessage Trait

The FlatMessage trait defines the core serialization interface:

#![allow(unused)]
fn main() {
pub trait FlatMessage<'a> {
    // Serialize data to a Storage buffer
    fn serialize_to(&self, output: &mut Storage, config: Config) -> Result<(), Error>;
    
    // Deserialize data from a buffer (with validation)
    fn deserialize_from(input: &'a Storage) -> Result<Self, Error>
    where
        Self: Sized;
    
    // Deserialize without validation (faster, but unsafe)
    unsafe fn deserialize_from_unchecked(input: &'a Storage) -> Result<Self, Error>
    where
        Self: Sized;
}
}

Serialization

Serialization transforms your struct into a byte buffer:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Point {
    x: f32,
    y: f32,
}

let point = Point { x: 1.0, y: 2.0 };
let mut storage = Storage::default();

// Serialize the point
point.serialize_to(&mut storage, Config::default())?;

// Access the raw bytes
let bytes = storage.as_slice();
println!("Serialized size: {} bytes", bytes.len());
}

Deserialization

Deserialization reconstructs your struct from bytes:

#![allow(unused)]
fn main() {
// Deserialize with validation (recommended)
let restored_point = Point::deserialize_from(&storage)?;

// Deserialize without validation (faster, but risky)
let restored_point = unsafe {
    Point::deserialize_from_unchecked(&storage)?
};
}

Zero-Copy Deserialization

FlatMessage's key feature is zero-copy deserialization - it doesn't copy data from the buffer when possible. This is however highly dependent on the data type you are deserializing. Some of them such as String or Vec<T> require allocation and copying of the data. Also, basic types such as u32, f32, bool etc. are not zero-copy types (but since they are small, the performance impact is negligible).

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Message<'a> {
    title: &'a str,        // Points directly into buffer
    tags: &'a [u32],       // Points directly into buffer
    owned_data: String,    // This still requires allocation
}
}

Zero-Copy vs Allocation

Lifetime Management

Zero-copy deserialization means your deserialized struct borrows from the original buffer:

#![allow(unused)]
fn main() {
// We assume that the `get_serialized_data` function returns a `Storage` object that contains the serialized data.
fn correct_process_message() -> Result<(), Error> {
    let storage = get_serialized_data();
    
    // This works - storage lives long enough
    let message = Message::deserialize_from(&storage)?;
    println!("Title: {}", message.title);
    
    Ok(())
} // storage and message both dropped here

fn broken_process_message() -> Result<Message<'static>, Error> {
    let storage = get_serialized_data();
    let message = Message::deserialize_from(&storage)?;
    
    // This won't compile - message can't outlive storage
    // Ok(message) // ❌ Compilation error
    
    unreachable!()
}
}

Performance Characteristics

Understanding performance implications helps you make good design decisions:

Serialization Performance

• Direct types (u32, f64, bool): Fastest, just memory copies
• Strings: Fast, just memory copies (for both &str and String)
• Vectors/Slices: Fast, just memory copies
• Enums: Fast, just the underlying representation

Deserialization Performance

• Zero-copy types: Fastest, just pointer adjustments
• Owned types: Slower, requires allocation and copying
• Validation: deserialize_from() validates data, deserialize_from_unchecked() skips validation

Memory Usage

Being a schemaless library, FlatMessage has to store information on the type of the data being serialized. This is done by storing a hash over then type name and the type size.

As a general rule, for a structure with n fields, the serialized size will:

• 8 bytes for the header
• 5 bytes x n for the fields (for each field, 5 bytes are used to store the field name and the field size)
• x bytes the actual content of the fields (where x is the sum of the sizes of all the fields)

Additionally, the following information is stored (if enabled)

• 4 bytes for the checksum (if enabled)
• 4 bytes for the name of the structure (if enabled)
• 8 bytes for an unique identifier (if enabled)
• 8 bytes for the timestamp of the serialization (if enabled)

This means that for the following structure:

#![allow(unused)]

fn main() {
#[derive(FlatMessage)]
struct Point {
    x: u32,
    y: u32,
}
}

The serialized size will be:

• 8 bytes for the header
• 5 bytes x 2 for the fields (for each field, 5 bytes are used to store the field name and the field size)
• 8 bytes for the actual content of the fields (both fields are u32, so 8 bytes each)
• 4 bytes for the name of the structure (enabled by default)

So a total of 30 bytes for the serialized size. The size could be 4 bytes smaller if we add #[flat_message_options(store_name = false)] to the structure.

Remarks:

• The increase of size is not linear, but rather it depends on the number of fields and their sizes. For example a structure with only one string field that holds 1000 characters will only add 13 ore characters on the serialized buffer (1013 bytes) witch is insignificant relative to the actual content of the field.

Best Practices

1. Use Storage for serialization: Storage is the only supported serialization target, optimized for FlatMessage workloads
2. Prefer zero-copy types: Use &str over String, &[T] over Vec<T> when possible
3. Validate when needed: Use deserialize_from() for untrusted data, deserialize_from_unchecked() for performance-critical trusted data
4. Set appropriate limits: Use Config to prevent excessive memory usage
5. Manage lifetimes carefully: Ensure buffers live long enough for zero-copy data

Example: Complete Workflow

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage, Debug, PartialEq)]
struct NetworkMessage<'a> {
    session_id: u64,
    command: &'a str,      // Zero-copy
    payload: &'a [u8],     // Zero-copy  
}

fn network_example() -> Result<(), flat_message::Error> {
    // Create message
    let original_message = NetworkMessage {
        session_id: 12345,
        command: "GET_USER",
        payload: &[1, 2, 3, 4, 5],
    };

    // Serialize with size limit
    let mut storage = Storage::default();
    let config = ConfigBuilder::new()
        .max_size(1024)  // 1KB limit
        .build();
    
    original_message.serialize_to(&mut storage, config)?;
    
    // Send over network (simulation)
    let network_bytes = storage.as_slice().to_vec();
    
    // Receive and deserialize
    let received_storage = Storage::from_buffer(&network_bytes);
    let received_message = NetworkMessage::deserialize_from(&received_storage)?;
    
    assert_eq!(original_message, received_message);
    println!("Message transmitted successfully!");
    
    Ok(())
}
}

Error Handling

FlatMessage provides comprehensive error handling through the Error enum. Understanding these errors helps you build robust serialization code.

Complete Error Reference

The following table lists all FlatMessage error types with their causes, typical scenarios, and recommended handling strategies:

Error Categories

Data Format Errors

• InvalidHeaderLength, InvalidMagic, InvalidSize, InvalidOffsetSize
• Cause: Malformed or corrupted data format
• Recovery: Validate data source, check file integrity

Data Integrity Errors

• InvalidHash, InvalidChecksum
• Cause: Data corruption during storage or transmission
• Recovery: Re-transmit data, use error correction

Structure Compatibility Errors

• IncompatibleVersion, FieldIsMissing, UnmatchedName
• Cause: Schema evolution, version mismatches
• Recovery: Migrate data, update compatibility rules

Configuration Errors

• NameNotStored, ChecksumNotStored, ExceedMaxSize
• Cause: Mismatched configuration between serialization and deserialization
• Recovery: Align configurations, adjust limits

Field-Level Errors

• InvalidFieldOffset, FailToDeserialize
• Cause: Field-specific corruption or type mismatches
• Recovery: Validate individual fields, check type compatibility

Example

The following example shows how to handle errors in a simple way when deserializing a struct.

#![allow(unused)]
fn main() {
use flat_message::*;

// consider that serialized_data is a buffer of the serialized data
let storage = Storage::from_buffer(serialized_data);

// Message is a struct that implements FlatMessage
match Message::deserialize_from(&storage) {
    Ok(restored_message) => {
        println!("Message serialized and deserialized successfully");
    }
    Err(e) => {
        panic!("Error deserializing message: {}", e);
    }
}
}

Serialization Model

FlatMessage uses a field-based serialization model that stores data in a structured binary format with hash tables for fast field lookup. The key advantage is zero-copy deserialization - instead of reconstructing objects in memory, it provides direct references to data within the serialized buffer.

This approach enables instant deserialization with no memory allocation, making it ideal for high-performance scenarios where read speed and memory efficiency are critical.

Binary Format

Remarks:

• The Magic field is used to identify the file format. It should always be 'FLM'.
• The Structure version field is used to indicate the version of the structure. Version 0 means that the structure has no versioning.
• The Serializarion flags field is a bitmask that provides information about the data. The following flags are currently supported:
  • TIMESTAMP: Indicates that the structure has a timestamp.
  • UNIQUEID: Indicates that the structure has a unique ID.
  • MAKEHASH: Indicates that the structure has a name hash.
  • CHECKSUM: Indicates that the structure has a checksum.
• The first 2 bits from the Serializarion flags field are use for offset size (1, 2 or 4 bytes). Smaller structus usually use 1 byte offset (meaning that the endire data is less than 255 bytes), while larger structs use a 2 or 4 bytes offset.

Supported Data Types

FlatMessage supports a variety of data types for serialization in the following way:

1. as a direct value:

   #![allow(unused)]
   fn main() {
   struct Name { value: T } 
   }

2. as a slice of values:

   #![allow(unused)]
   fn main() {
   struct Name { value: &[T] } 
   }

3. as a vector of values:

   #![allow(unused)]
   fn main() {
   struct Name { value: Vec<T> } 
   }

4. as an Option:

   #![allow(unused)]
   fn main() {
   struct Name { 
       value: Option<T>,
       slice: Option<&[T]>,
       vector: Option<Vec<T>>
   } 
   }

where T is the data type.

Remarks:

• The main difference between a slice and a vector is that a slice is a reference to an array of values, while a vector is an owned collection of values. Slices are more memory efficient, but vectors provide more flexibility in terms of resizing and ownership. You can use them interchangeably, meaning that you can serialize an object that has a vector field and deserialize it into a slice, or vice versa. FlatMessage will handle the conversion automatically.

Keep in mind that deserialization of a slice is a no-cost operation, while deserialization of a vector requires allocation and copying of the data, which may incur some performance overhead.

Basic Types

Remarks:

• for bool values, deserialization using deserialize_from will validate if the value is 0 or 1, and will return an error if the value is not valid. If you are certain that the value is valid, you can use deserialize_from_unchecked to skip the validation step. This will speed up the deserialization process, but it is your responsibility to ensure that the value is valid.

Example

1. Direct values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       boolean_value: bool,
       integer_value: u32,
       float_value: f64,
   }
   }

2. Slices of values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       boolean_values: &[bool],
       integer_values: &[u32],
       float_values: &[f64],
   }
   }

3. Vectors of values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       boolean_values: Vec<bool>,
       integer_values: Vec<u32>,
       float_values: Vec<f64>,
   }
   }

4. Option values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       boolean_value: Option<bool>,
       u32_vec: Option<Vec<u32>>,
       f32_slice: Option<&[f32]>,
   }
   }

Strings

Strings types are represented as UTF-8 encoded bytes.

Remarks:

• deserialization using deserialize_from will validate if the string is a correct UTF-8 buffer. If you are certain that format is valid, you can use deserialize_from_unchecked to skip the validation step. This will speed up the deserialization process, but it is your responsibility to ensure that the string is actually a valid UTF-8 format.
• You can use String and &str interchangeably, meaning that you can serialize an object that has a String field and deserialize it into a &str, or vice versa. This usually speeds up the deserialization process, as &str is a reference to a string slice and does not require allocation and copying of the data, while String is an owned collection of characters that requires allocation and copying of the data.

Example

1. Direct values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       string_value: String,
       str_value: &'a str,
   }
   }

2. Vectors of strings or string slices:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       string_values: Vec<String>,
       str_values: Vec<&'a str>,
   }
   }

3. Using Option values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       string_value: Option<String>,
       str_value: Option<&'a str>,
   }
   }

IP Addresses

Remarks:

• The serialization size for Ipv4Addr is 4 bytes, and for Ipv6Addr it is 16 bytes.
• The serialization size for IpAddr is 5 bybtes (if it is an Ipv4Addr) or 17 bytes (if it is an Ipv6Addr).

Example

1. Direct values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   use std::net::{Ipv4Addr, Ipv6Addr, IpAddr};
   
   #[derive(FlatMessage)]
   struct Example {
       ipv4_address: Ipv4Addr,
       ipv6_address: Ipv6Addr,
       ip_address: IpAddr,
   }
   }

2. Using Option values:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   use std::net::{Ipv4Addr, Ipv6Addr, IpAddr};
   
   #[derive(FlatMessage)]
   struct Example {
       ipv4_address: Option<Ipv4Addr>,
       ipv6_address: Option<Ipv6Addr>,
       ip_address: Option<IpAddr>,
   }
   }

Unique ID

Remarks:

• A UniqueID is a 64-bit value that must be non-zero and is intended to appear only once in a struct. It serves as a unique identifier for a message. The following example will not compile as it uses two fields with the type UniqueID:

  #![allow(unused)]
  fn main() {
  use flat_message::*;
  
  // code will not compile (two ids)
  #[derive(FlatMessage)]
  struct Example {
      id1: UniqueID
      id2: UniqueID
  }
  }

• The use of UniqueID is optional—you don't need to include it in your structure unless you want to store messages in a database or require a unique identifier.

• Since UniqueID can be used only once per struct, its field name is not stored; it will be automatically mapped to any field with the same type.

Methods

The following methods are available for an UniqueID:

Example

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
struct Example {
    name: String,
    grade: u32,
    id: UniqueID
}
}

Timestamp

Remarks:

• A Timestamp is a 64-bit value that represents time in milliseconds since the UNIX epoch (January 1, 1970, 00:00:00 UTC) and is intended to appear only once in a struct. The following example will not compile as it uses two fields with the type Timestamp:

  #![allow(unused)]
  fn main() {
  use flat_message::*;
  
  // code will not compile (two timestamps)
  #[derive(FlatMessage)]
  struct Example {
      created_at: Timestamp,
      updated_at: Timestamp
  }
  }

• The use of Timestamp is optional—you don't need to include it in your structure unless you want to track timing information.

• Since Timestamp can be used only once per struct, its field name is not stored; it will be automatically mapped to any field with the same type.

• The timestamp value is stored as an unsigned 64-bit integer, providing a range that can represent dates far into the future.

• When system time retrieval fails, the timestamp defaults to 0 (representing the UNIX epoch).

• The Timestamp type implements common traits like Copy, Clone, Debug, Eq, PartialEq, Ord, and PartialOrd, making it suitable for comparisons and sorting operations.

Methods

The following methods are available for a Timestamp:

Example

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
struct LogEntry {
    message: String,
    level: u8,
    created_at: Timestamp
}

// Usage examples
let entry = LogEntry {
    message: "Application started".to_string(),
    level: 1,
    created_at: Timestamp::now()
};

// Create with specific timestamp
let historical_entry = LogEntry {
    message: "System boot".to_string(),
    level: 0,
    created_at: Timestamp::with_value(1640995200000) // Jan 1, 2022 00:00:00 UTC
};
}

Enums

Enums with explicitly defined backing types are supported for serialization and deserialization.

Supported representations:

• u8, i8, u16, i16, u32, i32, u64, i64

Remarks:

• Enums must derive FlatMessageEnum and have an explicit #[repr(...)] attribute specifying the underlying primitive type.
• Enum variants can have explicit values assigned, or they will use the default incrementing values starting from 0.
• When using enums in structs, you must specify both the representation and kind in the field attribute: #[flat_message_item(repr = u8, kind = enum)].
• Enums can be marked as #[sealed] for stricter version compatibility. Sealed enums include all variant names and values in their hash, making them incompatible with any version that adds, removes, or modifies variants. Non-sealed enums only include the enum name in their hash, allowing forward compatibility when adding new variants.
• Both sealed and non-sealed enums validate that deserialized values match known variants in the current enum definition. The difference is in version compatibility: non-sealed enums allow adding variants without breaking hash compatibility, while sealed enums will fail to deserialize if the enum definition has changed in any way.
• Deserialization using deserialize_from validates enum variant values. If you are certain the data is valid, you can use deserialize_from_unchecked to skip validation and improve performance.

Example

1. Basic enum usage:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(u8)]
   enum Color {
       Red = 1,
       Green = 10,
       Blue = 100,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Example {
       #[flat_message_item(repr = u8, kind = enum)]
       color: Color,
   }
   }

2. Enum slices:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(u16)]
   enum Priority {
       Low = 1,
       Medium = 100,
       High = 1000,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Example<'a> {
       #[flat_message_item(repr = u16, kind = enum)]
       priorities: &'a [Priority],
   }
   }

3. Enum vectors:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(u32)]
   enum Status {
       Pending = 1,
       Processing = 2,
       Complete = 3,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Example {
       #[flat_message_item(repr = u32, kind = enum)]
       statuses: Vec<Status>,
   }
   }

4. Sealed enums for strict validation:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(u8)]
   #[sealed]
   enum Protocol {
       Http = 1,
       Https = 2,
       Ftp = 3,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Example {
       #[flat_message_item(repr = u8, kind = enum)]
       protocol: Protocol,
   }
   }

5. Using Option with enums:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(i16)]
   enum Temperature {
       Freezing = -100,
       Cold = -10,
       Warm = 20,
       Hot = 40,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Example {
       #[flat_message_item(repr = i16, kind = enum)]
       temperature: Option<Temperature>,
   }
   }

Version Compatibility

Enums support forward compatibility when adding new variants:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Color {
    Red = 1,
    Green = 10,
    Blue = 100,
}

// Version 2 - compatible with version 1 data
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Color {
    Red = 1,
    Green = 10,
    Blue = 100,
    Yellow = 200,  // New variant added
}
}

Data serialized with version 1 can be successfully deserialized with version 2 for non-sealed enums, as long as the specific variant values used in the serialized data exist in both versions. Sealed enums will fail to deserialize if any variants have been added, removed, or modified between versions.

Flags

Flags (bit fields) with explicitly defined backing types are supported for serialization and deserialization.

Supported representations:

• u8, u16, u32, u64, u128

Remarks:

• Flags must derive FlatMessageFlags and have an explicit #[repr(transparent)] attribute.
• Flags must declare available flag names in the #[flags(...)] attribute.
• Individual flag values are defined using the add_flag! macro or as public constants.
• When using flags in structs, you must specify both the representation and kind in the field attribute: #[flat_message_item(repr = u8, kind = flags)].
• Flags can be marked as #[sealed] for stricter version compatibility. Sealed flags include all flag names and values in their hash, making them incompatible with any version that adds, removes, or modifies flags. Non-sealed flags only include the flag struct name in their hash, allowing forward compatibility when adding new flags.
• Both sealed and non-sealed flags validate that deserialized values contain only valid flag combinations. If invalid flags are detected during deserialization, an error is returned.
• Deserialization using deserialize_from validates flag values. If you are certain the data is valid, you can use deserialize_from_unchecked to skip validation and improve performance.

Available Methods

Flags automatically implement the FlagsSupport trait, providing the following methods:

• from_value(value: T) - Creates flags from raw value, validates against known flags
• to_value(&self) - Returns the raw underlying value
• any_set(&self, flag: Self) - Checks if any of the specified flags are set
• all_set(&self, flag: Self) - Checks if all of the specified flags are set
• is_empty(&self) - Checks if no flags are set
• set(&mut self, flag: Self) - Sets the specified flags
• unset(&mut self, flag: Self) - Unsets the specified flags
• toggle(&mut self, flag: Self) - Toggles the specified flags
• clear(&mut self) - Clears all flags

Flags also support standard bitwise operations: | (OR), & (AND), ^ (XOR), and their assignment variants (|=, &=, ^=).

Example

1. Basic flags usage:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[flags(Read, Write, Execute)]
   struct Permissions(u8);
   
   impl Permissions {
       add_flag!(Read = 1);
       add_flag!(Write = 2);
       add_flag!(Execute = 4);
   }
   
   #[derive(Debug, FlatMessage)]
   struct FileInfo {
       #[flat_message_item(repr = u8, kind = flags)]
       permissions: Permissions,
   }
   }

2. Using flags with bit operations:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[flags(A, B, C)]
   struct Features(u32);
   
   impl Features {
       add_flag!(A = 1);
       add_flag!(B = 2);
       add_flag!(C = 4);
   }
   
   // Combining flags
   let mut features = Features::A | Features::B;
   
   // Checking flags
   assert!(features.all_set(Features::A));
   assert!(features.any_set(Features::A | Features::C));
   
   // Modifying flags
   features.set(Features::C);
   features.unset(Features::A);
   assert!(features.all_set(Features::B | Features::C));
   }

3. Flags slices:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[flags(Low, Medium, High)]
   struct Priority(u16);
   
   impl Priority {
       add_flag!(Low = 1);
       add_flag!(Medium = 2);
       add_flag!(High = 4);
   }
   
   #[derive(Debug, FlatMessage)]
   struct TaskList<'a> {
       #[flat_message_item(repr = u16, kind = flags)]
       priorities: &'a [Priority],
   }
   }

4. Flags vectors:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[flags(Debug, Info, Warning, Error)]
   struct LogLevel(u32);
   
   impl LogLevel {
       add_flag!(Debug = 1);
       add_flag!(Info = 2);
       add_flag!(Warning = 4);
       add_flag!(Error = 8);
   }
   
   #[derive(Debug, FlatMessage)]
   struct LogConfig {
       #[flat_message_item(repr = u32, kind = flags)]
       enabled_levels: Vec<LogLevel>,
   }
   }

5. Sealed flags for strict validation:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[sealed]
   #[flags(Http, Https, Ftp)]
   struct Protocol(u8);
   
   impl Protocol {
       add_flag!(Http = 1);
       add_flag!(Https = 2);
       add_flag!(Ftp = 4);
   }
   
   #[derive(Debug, FlatMessage)]
   struct Connection {
       #[flat_message_item(repr = u8, kind = flags)]
       supported_protocols: Protocol,
   }
   }

6. Using Option with flags:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
   #[repr(transparent)]
   #[flags(Cache, Compress, Encrypt)]
   struct Options(u64);
   
   impl Options {
       add_flag!(Cache = 1);
       add_flag!(Compress = 2);
       add_flag!(Encrypt = 4);
   }
   
   #[derive(Debug, FlatMessage)]
   struct RequestConfig {
       #[flat_message_item(repr = u64, kind = flags)]
       options: Option<Options>,
   }
   }

Version Compatibility

Flags support forward compatibility when adding new flags:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
#[repr(transparent)]
#[flags(Read, Write)]
struct Permissions(u8);

impl Permissions {
    add_flag!(Read = 1);
    add_flag!(Write = 2);
}

// Version 2 - compatible with version 1 data
#[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug)]
#[repr(transparent)]
#[flags(Read, Write, Execute)]
struct Permissions(u8);

impl Permissions {
    add_flag!(Read = 1);
    add_flag!(Write = 2);
    add_flag!(Execute = 4);  // New flag added
}
}

Data serialized with version 1 can be successfully deserialized with version 2 for non-sealed flags, as long as the specific flag values used in the serialized data are valid in both versions. However, if version 2 data contains new flags (like Execute) and is deserialized with version 1, it will fail validation since version 1 doesn't recognize the new flag.

Sealed flags will fail to deserialize if any flags have been added, removed, or modified between versions, providing stricter version control at the cost of forward compatibility.

Fixed Size Buffer

Fixed-size byte arrays with compile-time known length are supported for serialization and deserialization.

Remarks:

• Fixed-size byte arrays ([u8; N]) have their size known at compile time, where N is a constant value.
• References to fixed-size arrays (&[u8; N]) can be used for zero-copy deserialization, avoiding memory allocation.
• Deserialization validates that the stored size matches the expected compile-time size N. If the sizes don't match, deserialization will fail.
• You can use deserialize_from_unchecked to skip size validation if you are certain the data is valid, which improves performance.

Example

1. Direct fixed-size arrays:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       buffer_10: [u8; 10],
       buffer_4: [u8; 4],
       small_buffer: [u8; 2],
   }
   }

2. References to fixed-size arrays:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       buffer_ref: &'a [u8; 10],
   }
   }

3. Slices of fixed-size arrays:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       multiple_buffers: &'a [[u8; 3]],
   }
   }

4. Vectors of fixed-size arrays:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example {
       buffer_collection: Vec<[u8; 8]>,
   }
   }

5. Optional fixed-size arrays:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessage)]
   struct Example<'a> {
       optional_buffer: Option<[u8; 16]>,
       optional_buffer_ref: Option<&'a [u8; 12]>,
       optional_buffer_vec: Option<Vec<[u8; 4]>>,
   }
   }

Performance Considerations

• Fixed-size arrays are highly efficient as their size is known at compile time
• No dynamic memory allocation is needed for the arrays themselves
• Use references (&[u8; N]) when possible for zero-copy deserialization
• For collections of fixed arrays, the memory layout is contiguous can easily be readu using a slice (&[[u8; N]]) - meaning zero-copy deserialization.
• These types of buffer are often used for hash values (such as sha256 or sha512)

Structures

Custom structs with explicit alignment are supported for serialization and deserialization.

Supported alignments:

• 4-byte alignment (default)
• 8-byte alignment (if one of the fields requires 64 bits alignament - such as Vec)
• 16-byte alignment (if one of the fields requires 128 bits alignament - such as Vec)

Remarks:

• Structs must derive FlatMessageStruct to be used as nested structures within other FlatMessage types.
• When using structs in other structs, you must specify the alignment in the field attribute: #[flat_message_item(align = 4, kind = struct)].
• The alignment must match the struct's actual memory alignment requirements based on its fields.
• Structs automatically determine their required alignment based on their largest field's alignment requirements.
• This type of serialization does not support metadata fields like Timestamp and UniqueID. You can add them but they will ont be serialized and in deserialization phase they will be defaulted to 0.
• Fields can be marked with #[flat_message_item(ignore = true)] to exclude them from serialization.

Example

1. Basic struct usage:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct MyData {
       a: u8,
       b: u32,
       c: u16,
       d: String,
   }
   
   #[derive(Debug, FlatMessage)]
   #[flat_message_options(store_name = false)]
   struct Test {
       x: u8,
       #[flat_message_item(align = 4, kind = struct)]
       data: MyData,
       y: u8,
   }
   }

2. Struct with 8-byte alignment (contains u64 vectors or slices):

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct MyData {
       a: u8,
       b: u32,
       c: u16,
       d: String,
       values: Vec<u64>,  // Requires 8-byte alignment
   }
   
   #[derive(Debug, FlatMessage)]
   #[flat_message_options(store_name = false)]
   struct Test {
       x: u8,
       #[flat_message_item(align = 8, kind = struct)]
       data: MyData,
       y: u8,
   }
   }

3. Struct with 16-byte alignment (contains u128 vectors or slices):

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct MyData {
       a: u8,
       values: Vec<u128>,  // Requires 16-byte alignment
   }
   
   #[derive(Debug, FlatMessage)]
   #[flat_message_options(store_name = false)]
   struct Test {
       x: u8,
       #[flat_message_item(align = 16, kind = struct)]
       data: MyData,
       y: u8,
   }
   }

4. Structs with metadata fields:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct EventData {
       a: u8,
       b: u32,
       timestamp: Timestamp,  // ignored - will be defaulted to 0
       unique_id: UniqueID,   // ignored - will be defaulted to 0
   }
   
   #[derive(Debug, FlatMessage)]
   #[flat_message_options(store_name = false)]
   struct Event {
       #[flat_message_item(align = 4, kind = struct)]
       data: EventData,
   }
   }

5. Using Option with structs:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct Configuration {
       timeout: u32,
       retries: u8,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Request {
       #[flat_message_item(align = 4, kind = struct)]
       config: Option<Configuration>,
   }
   }

6. Structs with ignored fields:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct ProcessData {
       pid: u32,
       name: String,
       #[flat_message_item(ignore = true)]
       runtime_state: String,  // Not serialized
   }
   
   #[derive(Debug, FlatMessage)]
   struct ProcessList {
       #[flat_message_item(align = 4, kind = struct)]
       processes: Vec<ProcessData>,
   }
   }

7. Nested structs:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct LevelTwo {
       a: bool,
       l: Vec<i8>,
   }
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct LevelOne {
       a: u8,
       b: u32,
       c: u16,
       d: String,
       #[flat_message_item(align = 4, kind = struct)]
       e: LevelTwo,
   }
   #[derive(FlatMessage, Debug, PartialEq, Eq)]
   #[flat_message_options(store_name = false)]
   struct Test {
       x: u8,
       #[flat_message_item(align = 4, kind = struct)]
       d: LevelOne,
       a: u8,
   }
   }

Serialization Behavior

When structs are serialized:

1. Field Ordering: Fields are reordered during serialization based on their alignment requirements (largest alignment first) to optimize memory layout.

2. Hash Table: Each struct maintains a hash table of its fields for efficient deserialization and version compatibility.

3. Reference Table: Offset information for each field is stored to enable random access during deserialization.

Packed Structs

Packed structs are a high-performance, memory-efficient serialization format optimized for sequential data layouts and minimal overhead. Unlike regular structs, packed structs store data in a continuous memory layout without hash tables or field lookups.

Supported alignments:

• 1-byte alignment (fields requiring only byte alignment)
• 2-byte alignment (fields requiring 16-bit alignment)
• 4-byte alignment (fields requiring 32-bit alignment - default for most cases)
• 8-byte alignment (fields requiring 64-bit alignment - such as Vec)
• 16-byte alignment (fields requiring 128-bit alignment - such as Vec)

Key Characteristics:

• High Performance: Sequential memory layout provides optimal cache performance
• Low Overhead: No hash tables or field lookup structures
• Compact Size: Minimal metadata, only structure hash for validation
• Field Reordering: Fields are automatically reordered by alignment (largest first) for optimal packing
• Version Compatibility: Uses structure hash for version validation

Restrictions:

• No Option<T> types supported
• No Timestamp or UniqueID metadata fields
• All fields must be mandatory (no mandatory = false)
• No default values on deserialization failure (no validate = fallback)
• Maximum 65,535 fields per struct
• Fields can be ignored with #[flat_message_item(ignore = true)]

Usage

Packed structs must derive FlatMessagePacked and are used with kind = packed in field attributes:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessagePacked, Debug, PartialEq, Eq)]
struct MyPackedData {
    x: i32,
    y: u32,
    label: String,
}

#[derive(FlatMessage, Debug)]
struct Container {
    #[flat_message_item(kind = packed, align = 1)]
    data: MyPackedData,
    other_field: String,
}
}

Examples

1. Basic Packed Struct (1-byte alignment)

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
struct Point {
    x: i32,
    y: u32,
    label: String,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
struct Test {
    #[flat_message_item(kind = packed, align = 1)]
    point: Point,
    description: String,
}

fn example() {
    let test_data = Test {
        point: Point {
            x: 10,
            y: 20,
            label: "Origin".to_string(),
        },
        description: "Test point".to_string(),
    };
    
    // Serialize and deserialize
    let mut storage = Storage::default();
    test_data.serialize_to(&mut storage, Config::default()).unwrap();
    let deserialized = Test::deserialize_from(&storage).unwrap();
    
    assert_eq!(test_data, deserialized);
}
}

2. Packed Struct with 4-byte Alignment

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
struct DataSet {
    data: Vec<u32>,
    index: u8,
}

#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
struct ComplexPoint {
    x: i8,
    y: i8,
    #[flat_message_item(kind = packed, align = 4)]
    dataset1: DataSet,
    #[flat_message_item(kind = packed, align = 4)]
    dataset2: DataSet,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
struct Container {
    #[flat_message_item(kind = packed, align = 4)]
    point: ComplexPoint,
    name: String,
}

fn example() {
    let container = Container {
        point: ComplexPoint {
            x: 10,
            y: 20,
            dataset1: DataSet {
                data: vec![1, 2, 3],
                index: 1,
            },
            dataset2: DataSet {
                data: vec![4, 5, 6],
                index: 2,
            },
        },
        name: "Complex data".to_string(),
    };
    
    // Efficient serialization and deserialization
    let mut storage = Storage::default();
    container.serialize_to(&mut storage, Config::default()).unwrap();
    let result = Container::deserialize_from(&storage).unwrap();
    
    assert_eq!(container, result);
}
}

3. Packed Struct with Ignored Fields

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
struct ProcessInfo {
    pid: u32,
    name: String,
    memory_usage: u64,
    #[flat_message_item(ignore = true)]
    runtime_data: String,  // Not serialized, will be default on deserialization
}

#[derive(Debug, FlatMessage)]
struct SystemSnapshot {
    #[flat_message_item(kind = packed, align = 8)]
    processes: Vec<ProcessInfo>,
    timestamp: u64,
}

fn example() {
    let snapshot = SystemSnapshot {
        processes: vec![
            ProcessInfo {
                pid: 1234,
                name: "my_app".to_string(),
                memory_usage: 1024 * 1024 * 50, // 50MB
                runtime_data: "This will be ignored".to_string(),
            },
        ],
        timestamp: 1640995200,
    };
    
    let mut storage = Storage::default();
    snapshot.serialize_to(&mut storage, Config::default()).unwrap();
    let result = SystemSnapshot::deserialize_from(&storage).unwrap();
    
    // runtime_data will be empty string (default) after deserialization
    assert_eq!(result.processes[0].runtime_data, String::default());
    assert_eq!(result.processes[0].pid, 1234);
}
}

Serialization Behavior

Packed structs use a fundamentally different serialization approach compared to regular structs:

1. Sequential Layout

Fields are stored sequentially in memory without indirection:

[Structure Hash][Field1][Padding if required][Field2][Padding if required][Field3]...

2. Automatic Field Reordering

Fields are automatically reordered by alignment requirements (largest alignment first) to minimize padding and optimize memory layout.

3. Minimal Metadata

• Only a structure hash (4 bytes) is stored for version validation
• No field hash tables or lookup structures
• No field offset tables

4. Alignment-Based Padding

• Padding is inserted between fields as needed for proper alignment
• The struct's overall alignment is determined by its largest field alignment requirement
• Padding follows standard C struct alignment rules

5. Structure Hash Validation

• A FNV-32 hash of the structure definition is stored at the beginning
• Hash includes field names, types, data formats, and alignment requirements
• Deserialization fails if the hash doesn't match the expected structure

Performance Characteristics

When to Use Packed Structs

Choose packed structs when:

• Performance is critical and you need maximum speed
• Memory usage must be minimized
• Data is accessed sequentially or entirely
• Structure is stable and doesn't change frequently
• You don't need optional fields or metadata

Choose regular structs when:

• You need flexibility with optional fields
• Structure evolves frequently
• You need metadata fields (Timestamp, UniqueID)
• Random field access is important
• Backward/forward compatibility is required

Comparison with Regular Structs

Variant Enums

Variant enums (also known as algebraic data types or tagged unions) are supported for serialization and deserialization. They allow you to define an enum where each variant can contain different types of data.

Supported variant types:

• Basic types: u8, i8, u16, i16, u32, i32, u64, i64, u128, i128, f32, f64, bool
• Strings: String, &str
• Collections: Vec<T>, &[T] (where T is a supported type)
• Custom enums with #[derive(FlatMessageEnum)]
• Flags with #[derive(FlatMessageFlags)]
• Structs with #[derive(FlatMessageStruct)]
• Nested variant enums
• Unit variants (variants without data)
• Options: Option<T> for any supported type T

Memory alignment:

• Variant enums automatically determine their alignment based on the largest alignment requirement of their variants
• Supported alignments: 1, 2, 4, 8, 16 bytes
• When using variant enums in structs, you need to specify the alignment: #[flat_message_item(kind = variant, align = N)]

Remarks:

• Variant enums must derive FlatMessageVariant
• Each variant can contain at most one field (tuple-style variants)
• Named struct-style variants are not supported
• Unit variants (no associated data) are supported
• Variant enums can be marked as #[sealed] for stricter version compatibility
• Sealed variant enums include all variant names and types in their hash, making them incompatible with any version that adds, removes, or modifies variants
• Non-sealed variant enums only include the enum name in their hash, allowing forward compatibility when adding new variants
• When using complex types (enums, flags, structs) within variants, you must specify additional attributes: #[flat_message_item(kind = enum/flags/struct, repr = type, align = N)]
• Deserialization using deserialize_from validates variant types and data. If you are certain the data is valid, you can use deserialize_from_unchecked to skip validation and improve performance

Examples

1. Basic variant enum with primitive types:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum Value {
       Integer(i32),
       Float(f64),
       Text(String),
       Flag(bool),
   }
   
   #[derive(Debug, FlatMessage)]
   struct Message {
       #[flat_message_item(kind = variant, align = 8)]
       data: Value,
   }
   }

2. Variant enum with collections:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum Container {
       Numbers(Vec<i32>),
       Words(Vec<String>),
       Bytes(Vec<u8>),
       Empty,  // Unit variant
   }
   
   #[derive(Debug, FlatMessage)]
   struct Document {
       #[flat_message_item(kind = variant, align = 4)]
       content: Container,
   }
   }

3. Variant enum with custom enums:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
   #[repr(u8)]
   enum Priority {
       Low = 1,
       Medium = 2,
       High = 3,
   }
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum TaskData {
       Text(String),
       #[flat_message_item(kind = enum, repr = u8)]
       PriorityLevel(Priority),
       Urgent,  // Unit variant for urgent tasks
   }
   
   #[derive(Debug, FlatMessage)]
   struct Task {
       id: u32,
       #[flat_message_item(kind = variant, align = 1)]
       data: TaskData,
   }
   }

4. Variant enum with flags:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
   #[repr(transparent)]
   #[flags(Read, Write, Execute)]
   struct Permissions(u8);
   impl Permissions {
       add_flag!(Read = 1);
       add_flag!(Write = 2);
       add_flag!(Execute = 4);
   }
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum FileInfo {
       Name(String),
       Size(u64),
       #[flat_message_item(kind = flags, repr = u8)]
       Access(Permissions),
   }
   
   #[derive(Debug, FlatMessage)]
   struct FileEntry {
       #[flat_message_item(kind = variant, align = 8)]
       info: FileInfo,
   }
   }

5. Variant enum with structs:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct Point {
       x: f32,
       y: f32,
   }
   
   #[derive(FlatMessageStruct, Debug, PartialEq, Eq)]
   struct Color {
       r: u8,
       g: u8,
       b: u8,
   }
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum ShapeData {
       #[flat_message_item(kind = struct, align = 4)]
       Position(Point),
       #[flat_message_item(kind = struct, align = 1)]
       Appearance(Color),
       Label(String),
   }
   
   #[derive(Debug, FlatMessage)]
   struct Shape {
       #[flat_message_item(kind = variant, align = 4)]
       data: ShapeData,
   }
   }

6. Variant enum with Option types:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum OptionalData {
       Text(Option<String>),
       Number(Option<i32>),
       Present,
       Absent,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Record {
       #[flat_message_item(kind = variant, align = 4)]
       data: OptionalData,
   }
   }

7. Sealed variant enum for strict validation:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   #[sealed]
   enum Protocol {
       Http(String),
       Https(String),
       Ftp(String),
       Unknown,
   }
   
   #[derive(Debug, FlatMessage)]
   struct Connection {
       #[flat_message_item(kind = variant, align = 1)]
       protocol: Protocol,
   }
   }

8. Using Option with variant enums:

   #![allow(unused)]
   fn main() {
   use flat_message::*;
   
   #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
   enum Response {
       Success(String),
       Error(u32),
       Timeout,
   }
   
   #[derive(Debug, FlatMessage)]
   struct ApiCall {
       endpoint: String,
       #[flat_message_item(kind = variant, align = 4)]
       response: Option<Response>,  // Optional response
   }
   }

Version Compatibility

Variant enums support forward compatibility when adding new variants:

#![allow(unused)]
fn main() {
// Version 1
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
enum Message {
    Text(String),
    Number(i32),
}

// Version 2 - compatible with version 1 data
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
enum Message {
    Text(String),
    Number(i32),
    Binary(Vec<u8>),  // New variant added
    Timestamp(u64),   // Another new variant
}
}

Compatibility rules:

• Non-sealed variants: Data serialized with version 1 can be successfully deserialized with version 2, as long as the specific variants used in the serialized data exist in both versions
• Sealed variants: Will fail to deserialize if any variants have been added, removed, or modified between versions
• Existing variants: Must maintain the same type signature and attributes across versions
• New variants: Can be safely added to non-sealed variant enums without breaking compatibility

Breaking changes:

• Removing variants
• Changing the type of data in existing variants
• Changing attributes (kind, repr, align) of existing variants
• Making a non-sealed variant enum sealed

Serialization Compatibility

FlatMessage provides comprehensive compatibility features that allow your data structures to evolve over time while maintaining interoperability between different versions. This chapter covers the key mechanisms for ensuring your serialized data can be safely shared between applications using different versions of the same message structures.

Overview

When working with serialized data in production systems, you often need to handle scenarios where:

• Newer applications need to read data serialized by older versions
• Older applications need to read data serialized by newer versions
• Different services are running different versions of your data structures
• Data migration requires careful compatibility planning

FlatMessage addresses these challenges through several key features:

• Version Control: Every FlatMessage structure can be assigned a version number and specify which versions it's compatible with. This provides explicit control over backward and forward compatibility.

• Flexible Field Management: Fields can be marked as mandatory or optional, with support for default values when fields are missing. This enables safe addition and modification of structure fields.

• Sealed vs Non-Sealed Types: Enums, flags, and variants can be sealed (strict compatibility) or non-sealed (forward compatible), giving you control over how these types evolve.

Versioning

FlatMessage provides explicit version control for your data structures through the version and compatible_versions attributes. This system allows you to evolve your data formats safely while maintaining precise control over compatibility. Understanding how versioning interacts with field requirements is crucial for designing robust, evolvable data structures.

Basic Version Declaration

Every FlatMessage structure can declare a version number using the version attribute:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
#[flat_message_options(version = 1)]
struct UserProfile {
    name: String,
    email: String,
}
}

Key Points:

• Version numbers range from 1 to 255 (version 0 means no versioning)
• The version is stored in the 8-byte header of serialized data
• During deserialization, version compatibility is checked BEFORE field validation

Version Compatibility Control

Use compatible_versions to specify which data versions your structure can deserialize:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(version = 2, compatible_versions = "1,2")]
struct UserProfileV2 {
    name: String,
    email: String,
    age: u32,  // New field - but is it compatible with v1 data?
}
}

This configuration means:

• The structure declares itself as version 2
• It accepts data from versions 1 and 2
• Attempting to deserialize version 3+ data will fail with Error::IncompatibleVersion

Critical: Version compatibility only controls which data versions are accepted - it does NOT automatically handle field differences.

Deserialization Process Order

Understanding the deserialization process is crucial:

#![allow(unused)]
fn main() {
fn deserialize_example() {
    // 1. Header validation (magic number, size checks)
    // 2. Version compatibility check
    if header.version not in compatible_versions {
        return Err(Error::IncompatibleVersion(header.version));
    }
    // 3. Field validation and deserialization
    for each mandatory field {
        if field not found in data {
            return Err(Error::FieldIsMissing(field_hash));
        }
    }
    // 4. Struct construction
}
}

Version Compatibility Syntax

The compatible_versions attribute supports flexible range expressions:

Specific Versions

#![allow(unused)]
fn main() {
#[flat_message_options(version = 3, compatible_versions = "1,2,3")]
// Accepts exactly versions 1, 2, and 3
}

Range Operators

#![allow(unused)]
fn main() {
#[flat_message_options(version = 5, compatible_versions = "<5")]
// Accepts versions 1, 2, 3, 4 (less than 5)

#[flat_message_options(version = 4, compatible_versions = "<=4")]
// Accepts versions 1, 2, 3, 4 (less than or equal to 4)
}

Interval Ranges

#![allow(unused)]
fn main() {
#[flat_message_options(version = 10, compatible_versions = "5-10")]
// or "5:10" or "5..10"
// Accepts versions 5, 6, 7, 8, 9, 10
}

Combined Expressions

#![allow(unused)]
fn main() {
#[flat_message_options(version = 8, compatible_versions = "1,3,5-8")]
// Accepts versions 1, 3, 5, 6, 7, 8
}

Real-World Version Evolution Examples

Scenario 1: Version Check Without Field Compatibility

#![allow(unused)]
fn main() {
mod v1 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 1, compatible_versions = "1")]
    struct Config {
        value: u8,
    }
}

mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2, compatible_versions = "1,2")]
    struct Config {
        value: u8,
        value2: u16,  // New mandatory field
    }
}
}

Results:

• v1 data → v2 struct: ❌ Fails with FieldIsMissing (value2 not in v1 data)
• v2 data → v1 struct: ❌ Fails with IncompatibleVersion(2) (v1 only accepts version 1)
• v2 data → v2 struct: ✅ Works

Key Insight: Version compatibility and field compatibility are separate concerns!

Scenario 2: Forward Compatible Versioning

#![allow(unused)]
fn main() {
mod v1 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 1)]  // No compatible_versions = accepts any version
    struct Config {
        value: u8,
    }
}

mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]  // No compatible_versions = accepts any version
    struct Config {
        value: u8,
        value2: u16,  // New mandatory field
    }
}
}

Results:

• v1 data → v2 struct: ❌ Fails with FieldIsMissing (value2 not in v1 data)
• v2 data → v1 struct: ✅ Works (v1 ignores extra fields it doesn't need)

Scenario 3: Safe Evolution with Optional Fields

#![allow(unused)]
fn main() {
mod v1 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 1)]
    struct Config {
        value: u8,
    }
}

mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]
    struct Config {
        value: u8,
        #[flat_message_item(mandatory = false, default = "3")]
        value2: u16,  // Optional field with default
    }
}
}

Results:

• v1 data → v2 struct: ✅ Works (value2 = 3 default)
• v2 data → v1 struct: ✅ Works (v1 ignores extra fields)

Scenario 4: Safe Evolution with Option Fields

#![allow(unused)]
fn main() {
mod v1 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 1)]
    struct Config {
        value: u8,
    }
}

mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]
    struct Config {
        value: u8,
        value2: Option<u16>,  // Option<T> automatically optional
    }
}
}

Results:

• v1 data → v2 struct: ✅ Works (value2 = None default)
• v2 data → v1 struct: ✅ Works (v1 ignores extra fields)

Key Insight: Option<T> fields are automatically optional, making them ideal for safe version evolution.

Scenario 5: Explicit Mandatory Option

#![allow(unused)]
fn main() {
mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]
    struct Config {
        value: u8,
        #[flat_message_item(mandatory = true)]
        value2: Option<u16>,  // Explicitly mandatory Option<T>
    }
}
}

Results:

• v1 data → v2 struct: ❌ Fails with FieldIsMissing (explicit mandatory override)
• v2 data → v1 struct: ✅ Works (v1 ignores extra fields)

Common Version Compatibility Patterns

Strict Version Matching

#![allow(unused)]
fn main() {
#[flat_message_options(version = 2, compatible_versions = "2")]
// Only accepts exactly version 2 data
}

Use case: When you need strict control and no backward compatibility.

Backward Compatibility

#![allow(unused)]
fn main() {
#[flat_message_options(version = 3, compatible_versions = "1-3")]
// Version 3 struct can read data from versions 1, 2, and 3
}

Use case: When newer code needs to read older data formats. Requires careful field design with optional fields for additions. Option<T> fields are automatically optional, making this easier.

Forward Compatibility

#![allow(unused)]
fn main() {
#[flat_message_options(version = 1, compatible_versions = "1-5")]
// Version 1 struct can read data up to version 5
}

Use case: When older code needs to read newer data formats. Only works if newer versions only add optional fields.

Version Windows

#![allow(unused)]
fn main() {
#[flat_message_options(version = 5, compatible_versions = "3-7")]
// Accepts versions 3, 4, 5, 6, 7 but not 1, 2, or 8+
}

Use case: When you want to support a sliding window of versions, dropping support for very old formats.

Version Introspection

You can check the version of serialized data without full deserialization:

#![allow(unused)]
fn main() {
use flat_message::*;

fn check_version(storage: &Storage) -> Result<(), Error> {
    let info = StructureInformation::try_from(storage)?;
    
    match info.version() {
        Some(version) => {
            println!("Data version: {}", version);
            
            // Make decisions based on version
            if version < 2 {
                println!("Legacy format detected");
            } else if version > 5 {
                println!("Future version - may need migration");
            }
        }
        None => println!("No version information available"),
    }
    
    Ok(())
}
}

Error Handling

Version-related errors provide specific information:

#![allow(unused)]
fn main() {
match Config::deserialize_from(&storage) {
    Err(Error::IncompatibleVersion(found_version)) => {
        eprintln!("Cannot read version {} data with this struct", found_version);
        // Could attempt migration or request data in supported format
    }
    Err(Error::FieldIsMissing(field_hash)) => {
        eprintln!("Missing required field (hash: 0x{:08X})", field_hash);
        // Field compatibility issue, not version issue
    }
    Err(Error::InvalidHeaderLength(_)) => {
        eprintln!("Corrupted or invalid data");
    }
    Ok(config) => {
        // Successfully deserialized
    }
}
}

Advanced Version Handling

For complex migration scenarios, implement version-aware deserialization:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(version = 3, compatible_versions = "1-3")]
struct Config {
    host: String,
    port: u16,
    
    #[flat_message_item(mandatory = false, default = "30")]
    timeout: u32,    // Optional for v1/v2 compatibility
    
    #[flat_message_item(mandatory = false, default = "3")]
    retries: u8,     // Optional for v1/v2 compatibility
}

impl Config {
    pub fn from_any_version(storage: &Storage) -> Result<Self, Error> {
        // Check version before attempting deserialization
        let info = StructureInformation::try_from(storage)?;
        
        match info.version() {
            Some(1) => {
                println!("Migrating from v1 format");
                // v1 data should work with optional fields
                Self::deserialize_from(storage)
            }
            Some(2) => {
                println!("Reading v2 format");
                Self::deserialize_from(storage)
            }
            Some(3) => {
                println!("Reading current v3 format");
                Self::deserialize_from(storage)
            }
            Some(version) if version > 3 => {
                Err(Error::IncompatibleVersion(version))
            }
            None => {
                println!("No version info - assuming v1");
                Self::deserialize_from(storage)
            }
            _ => unreachable!(),
        }
    }
}
}

Best Practices

1. Version from the start: Always include version = 1 on new structures
2. Plan compatibility strategy: Decide upfront whether you need backward, forward, or bidirectional compatibility
3. Use optional fields for additions: New fields should be optional (mandatory = false) to maintain backward compatibility
4. Test all compatibility scenarios: Include tests for all supported version combinations
5. Understand the two-phase validation: Version check happens before field validation
6. Document breaking changes: Clearly mark when mandatory fields are added
7. Use version introspection: Check versions before deserialization in multi-version systems
8. Plan deprecation cycles: Allow time for systems to upgrade before removing compatibility

Common Pitfalls

1. Assuming version compatibility handles fields: compatible_versions only controls version acceptance, not field compatibility
2. Adding mandatory fields to backward-compatible versions: This breaks compatibility even with version ranges
3. Not understanding Option default behavior: Option<T> fields are automatically optional unless explicitly marked with mandatory = true
4. Not testing field compatibility: Version compatibility tests must also verify field-level compatibility

Key Takeaway

This versioning system, combined with careful field management and the automatic optional behavior of Option<T> fields, provides the foundation for robust data evolution in FlatMessage. The automatic optional behavior of Option<T> fields makes version evolution much safer and easier - when adding new fields to existing structures, prefer Option<T> types as they automatically provide backward compatibility without requiring explicit mandatory = false attributes.

Mandatory Fields and Default Values

FlatMessage fields are mandatory by default, meaning they must be present in the serialized data during deserialization. However, you can mark fields as optional using the mandatory = false attribute and provide default values. Understanding this system is crucial for designing evolvable data structures.

Understanding Mandatory vs Optional Fields

Mandatory Fields (Default Behavior)

By default, all fields in a FlatMessage structure are mandatory:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
struct UserProfile {
    name: String,      // Mandatory
    email: String,     // Mandatory
    age: u32,          // Mandatory
}
}

What happens during deserialization:

• FlatMessage searches for each mandatory field's hash in the serialized data
• If any mandatory field is missing, deserialization fails with Error::FieldIsMissing(hash)
• This happens regardless of version compatibility settings

Optional Fields

Use mandatory = false to make fields optional:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct UserProfile {
    name: String,      // Mandatory
    email: String,     // Mandatory
    
    #[flat_message_item(mandatory = false)]
    age: u32,          // Optional - defaults to 0 if missing
    
    #[flat_message_item(mandatory = false)]
    bio: String,       // Optional - defaults to "" if missing
}
}

What happens during deserialization:

• FlatMessage searches for the optional field's hash in the serialized data
• If found, the field is deserialized normally
• If not found, the field uses its default value (Type::default() or custom default)
• No error is thrown for missing optional fields

Default Value Behavior

Type Defaults

When mandatory = false is specified without a custom default, the field uses the type's Default::default() implementation:

Custom Default Values

You can specify custom default values using the default attribute:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct ServerConfig {
    host: String,      // Mandatory
    
    #[flat_message_item(mandatory = false, default = "8080")]
    port: u16,         // Optional with custom default
    
    #[flat_message_item(mandatory = false, default = "30")]
    timeout: u32,      // Optional with custom default
    
    #[flat_message_item(mandatory = false, default = "\"production\"")]
    environment: String, // Optional with custom default (note quotes)
}
}

String Default Syntax:

• For string literals, use double quotes: default = "\"production\""
• The system expects a valid Rust expression that evaluates to the field's type

Advanced Default Values

You can use constants, expressions, or function calls:

#![allow(unused)]
fn main() {
const DEFAULT_TIMEOUT: u32 = 60;
const DEFAULT_RETRIES: u8 = 3;

#[derive(FlatMessage)]
struct ApiConfig {
    endpoint: String,  // Mandatory
    
    #[flat_message_item(mandatory = false, default = "DEFAULT_TIMEOUT")]
    timeout: u32,      // Uses constant
    
    #[flat_message_item(mandatory = false, default = "DEFAULT_RETRIES")]
    retries: u8,       // Uses constant
    
    #[flat_message_item(mandatory = false, default = "vec![8080, 8081, 8082]")]
    allowed_ports: Vec<u16>, // Uses expression
}
}

Important: Option Fields Are Optional by Default

Key Change: Option<T> fields are automatically treated as optional (mandatory = false) by default:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Config {
    host: String,           // Mandatory
    port: Option<u16>,      // Automatically optional! Uses None if missing
    
    #[flat_message_item(mandatory = true)]
    timeout: Option<u32>,   // Explicitly mandatory - must be present in data
}
}

Behavior:

• Option<T> fields without explicit attributes: Optional (use None if missing)
• Option<T> fields with mandatory = true: Mandatory (cause Error::FieldIsMissing if not present)
• Option<T> fields with mandatory = false: Optional (use None if missing - same as default)

This makes Option<T> fields naturally compatible for version evolution since they default to being optional.

Relationship with Versioning

Mandatory fields interact with versioning in specific ways:

Version Compatibility is Checked First

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(version = 2, compatible_versions = "1,2")]
struct Config {
    host: String,      // Mandatory
    port: u16,         // Mandatory
    timeout: u32,      // Mandatory - added in v2
}
}

Deserialization process:

1. Version check: Is the data version in compatible_versions?
   • If not → Error::IncompatibleVersion(version)
2. Field validation: Are all mandatory fields present?
   • If not → Error::FieldIsMissing(hash)

Adding Mandatory Fields Breaks Compatibility

#![allow(unused)]
fn main() {
// Version 1 data serialized with this structure
#[derive(FlatMessage)]
#[flat_message_options(version = 1)]
struct Config {
    host: String,
    port: u16,
}

// Version 2 structure tries to read v1 data
#[derive(FlatMessage)]
#[flat_message_options(version = 2, compatible_versions = "1,2")]
struct Config {
    host: String,
    port: u16,
    timeout: u32,      // New mandatory field
}
}

Result: Even though version compatibility allows reading v1 data, deserialization will fail with Error::FieldIsMissing because timeout is mandatory but not present in v1 data.

Adding Optional Fields Maintains Compatibility

#![allow(unused)]
fn main() {
// Version 2 structure can successfully read v1 data
#[derive(FlatMessage)]
#[flat_message_options(version = 2, compatible_versions = "1,2")]
struct Config {
    host: String,
    port: u16,
    
    #[flat_message_item(mandatory = false, default = "30")]
    timeout: u32,      // Optional field with default
}
}

Result: v1 data → v2 struct works (timeout = 30), v2 data → v1 struct works (ignores timeout)

Real-World Compatibility Scenarios

Based on the test scenarios, here are the actual compatibility behaviors:

Scenario 1: Adding Mandatory Fields

#![allow(unused)]
fn main() {
mod v1 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 1)]
    struct Config { value: u8 }
}

mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]
    struct Config { 
        value: u8,
        value2: u16,  // New mandatory field
    }
}
}

• ✅ v2 data → v1 struct: Works (v1 ignores extra field)
• ❌ v1 data → v2 struct: Fails with FieldIsMissing (value2 not in v1 data)

Scenario 2: Adding Optional Fields

#![allow(unused)]
fn main() {
mod v2 {
    #[derive(FlatMessage)]
    #[flat_message_options(version = 2)]
    struct Config { 
        value: u8,
        #[flat_message_item(mandatory = false, default = "3")]
        value2: u16,  // Optional field
    }
}
}

• ✅ v2 data → v1 struct: Works (v1 ignores extra field)
• ✅ v1 data → v2 struct: Works (value2 = 3 default)

Scenario 3: Option Fields (Automatically Optional)

#![allow(unused)]
fn main() {
mod v2 {
    #[derive(FlatMessage)]
    struct Config { 
        value: u8,
        value2: Option<u16>,  // Automatically optional (new default behavior)
    }
}
}

• ✅ v2 data → v1 struct: Works (v1 ignores extra field)
• ✅ v1 data → v2 struct: Works (value2 = None default)

Scenario 4: Option with Explicit mandatory = true

#![allow(unused)]
fn main() {
mod v2 {
    #[derive(FlatMessage)]
    struct Config { 
        value: u8,
        #[flat_message_item(mandatory = true)]
        value2: Option<u16>,  // Explicitly mandatory Option
    }
}
}

• ✅ v2 data → v1 struct: Works (v1 ignores extra field)
• ❌ v1 data → v2 struct: Fails with FieldIsMissing (explicitly mandatory Option)

Best Practices

1. Default to mandatory: Make fields mandatory unless you specifically need them optional
2. Plan for evolution: New fields should be optional to maintain backward compatibility
3. Leverage Option for new fields: Option<T> fields are automatically optional, making them ideal for version evolution
4. Use meaningful defaults: Provide sensible default values that won't break application logic
5. Test compatibility: Always test deserialization across supported version combinations
6. Version carefully: Consider mandatory field additions as breaking changes

Understanding mandatory fields and the automatic optional behavior of Option<T> is essential for designing robust, evolvable FlatMessage structures that can grow over time while maintaining compatibility with existing data.

Field Value Validation

Field value validation in FlatMessage provides a powerful mechanism for handling deserialization failures gracefully, making it particularly useful for versioning scenarios. The validate attribute allows you to specify how the system should behave when a field cannot be deserialized successfully.

Validation Modes

FlatMessage supports two validation modes:

• validate = strict (default): Deserialization fails immediately if any field cannot be deserialized
• validate = fallback: Falls back to the field's default value if deserialization fails

Syntax

The validate attribute can be applied at two levels:

Structure Level

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(validate = fallback)]
struct MyStruct {
    // All fields will use fallback validation by default
}
}

Field Level

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct MyStruct {
    #[flat_message_item(validate = strict)]
    critical_field: u32,
    
    #[flat_message_item(validate = fallback)]
    optional_field: Color,
}
}

Remarks:: The structure level validate attribute can be overridden at the field level (by useing #[flat_message_item(validate = "...")]).

Common Use Cases

1. Enum Evolution with Backward Compatibility

When adding new variants to enums, validate = fallback allows older code to handle newer enum values gracefully:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug, Default)]
#[repr(u8)]
pub enum Color {
    #[default]
    Red = 1,
    Green = 10,
    Blue = 100,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(repr = u8, kind = enum, validate = fallback)]
    pub color: Color,
}
}

#![allow(unused)]
fn main() {
// Version 2 - adds Yellow variant
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
pub enum Color {
    Red = 1,
    Green = 10,
    Blue = 100,
    Yellow = 200,  // New variant
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(repr = u8, kind = enum)]
    pub color: Color,
}
}

When v1 code tries to deserialize data containing Yellow (value 200), it will fallback to the default Red value instead of failing:

#![allow(unused)]
fn main() {
// v2 serializes data with Yellow
let d_v2 = v2::TestStruct { value: 1, color: v2::Color::Yellow };
d_v2.serialize_to(&mut storage, Config::default()).unwrap();

// v1 deserializes successfully with fallback to default
let d_v1 = v1::TestStruct::deserialize_from(&storage).unwrap();
assert_eq!(d_v1.value, 1);
assert_eq!(d_v1.color, v1::Color::Red); // Fallback to default
}

2. Flags Evolution with Backward Compatibility

Similar to enums, flags can evolve by adding new flag variants. Using validate = fallback allows older code to handle newer flag combinations gracefully:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug, Default)]
#[repr(transparent)]
#[flags(A,B)]
pub struct Flags(u8);
impl Flags {
    add_flag!(A = 1);
    add_flag!(B = 2);
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(repr = u8, kind = flags, validate = fallback)]
    pub flags: Flags,
}
}

#![allow(unused)]
fn main() {
// Version 2 - adds C flag
#[derive(Copy, Clone, FlatMessageFlags, PartialEq, Eq, Debug, Default)]
#[repr(transparent)]
#[flags(A,B,C)]
pub struct Flags(u8);
impl Flags {
    add_flag!(A = 1);
    add_flag!(B = 2);
    add_flag!(C = 4);  // New flag
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(repr = u8, kind = flags)]
    pub flags: Flags,
}
}

When v1 code tries to deserialize data containing the new C flag, it will fallback to the default empty flags instead of failing:

#![allow(unused)]
fn main() {
// v2 serializes data with C flag (unknown to v1)
let d_v2 = v2::TestStruct { 
    value: 1, 
    flags: v2::Flags::C | v2::Flags::B 
};
d_v2.serialize_to(&mut storage, Config::default()).unwrap();

// v1 deserializes successfully with fallback to default
let d_v1 = v1::TestStruct::deserialize_from(&storage).unwrap();
assert_eq!(d_v1.value, 1);
assert!(d_v1.flags.is_empty()); // Fallback to default (empty flags)
}

3. Variant Evolution with Backward Compatibility

Variants (also known as tagged unions or sum types) can evolve by adding new variants. Using validate = fallback allows older code to handle newer variant types gracefully by falling back to a default variant:

#![allow(unused)]
fn main() {
// Version 1
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
pub enum DataVariant {
    Byte(u8),
    String(String),
}

impl Default for DataVariant {
    fn default() -> Self {
        DataVariant::Byte(0)
    }
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(kind = variant, align = 1, validate = fallback)]
    pub data: DataVariant,
}
}

#![allow(unused)]
fn main() {
// Version 2 - adds DWord variant
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
pub enum DataVariant {
    Byte(u8),
    String(String),
    DWord(u32),  // New variant
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
pub struct TestStruct {
    pub value: u8,
    #[flat_message_item(kind = variant, align = 1)]
    pub data: DataVariant,
}
}

When v1 code tries to deserialize data containing the new DWord variant, it will fallback to the default variant instead of failing:

#![allow(unused)]
fn main() {
// v2 serializes data with DWord variant (unknown to v1)
let d_v2 = v2::TestStruct { 
    value: 1, 
    data: v2::DataVariant::DWord(12345) 
};
d_v2.serialize_to(&mut storage, Config::default()).unwrap();

// v1 deserializes successfully with fallback to default
let d_v1 = v1::TestStruct::deserialize_from(&storage).unwrap();
assert_eq!(d_v1.value, 1);
assert_eq!(d_v1.data, v1::DataVariant::Byte(0)); // Fallback to default variant
}

Important Notes for Variants:

• The variant type must implement the Default trait for fallback validation to work
• Unlike enums, variants cannot use the #[default] attribute on individual variants due to their complex data structure
• The Default implementation should return a sensible default variant with appropriate default values for its contained data

4. Graceful Schema Evolution

Fallback validation enables smooth transitions when field types change or become incompatible:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(validate = fallback)] // Structure-level fallback validation
struct Configuration {
    #[flat_message_item(default = 30)]
    timeout_seconds: u32,
    
    log_level: LogLevel, // Uses structure-level fallback validation
    
    #[flat_message_item(validate = strict)] // Override to strict for critical field
    database_url: String, // Critical field - must not fail
}
}

How Fallback Validation Works

When validate = fallback is specified:

1. Normal Operation: If the field can be deserialized normally, it uses the stored value
2. Fallback Trigger: If deserialization fails (e.g., enum variant not found, type mismatch), the system:
   • Uses the default value if specified in the attribute
   • Uses the type's Default::default() implementation
   • For Option<T> types, defaults to None

Best Practices

When to Use Strict Validation

• Critical fields where data integrity is essential
• When you need to detect and handle incompatibilities explicitly
• Fields where a default value doesn't make semantic sense

When to Use Fallback Validation

• Optional or non-critical fields
• Enum/Flags/Variant fields that may evolve over time
• During data migration periods
• When maintaining backward compatibility is important

Combining with Other Attributes

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct FlexibleStruct {
    // Critical field - must be present and valid
    #[flat_message_item(mandatory = true, validate = strict)]
    user_id: u64,
    
    // Optional field with fallback and custom default
    #[flat_message_item(
        mandatory = false, 
        validate = fallback, 
        default = "guest"
    )]
    username: String,
    
    // Enum that can evolve safely
    #[flat_message_item(
        repr = u8, 
        kind = enum, 
        validate = fallback
    )]
    user_type: UserType,
}
}

Implementation Notes

• The validate attribute affects only the deserialization process
• Serialization is not affected by validation settings
• Fallback validation requires the type to implement Default trait
• For enums, the #[default] attribute must be specified on one variant
• The validation behavior is determined at compile time through proc macros

Sealed vs Non-Sealed Types

FlatMessage supports the #[sealed] attribute for enums, flags, and variants, providing two different approaches to type evolution and compatibility. This attribute fundamentally changes how hash values are computed and affects compatibility behavior when types are modified.

Understanding the Sealed Attribute

The #[sealed] attribute controls whether a type's hash includes all its internal structure:

• Non-sealed (default): Hash includes only the type name
• Sealed: Hash includes type name + all internal structure (variants, flags, etc.)

This applies to three main derive macros:

• FlatMessageEnum with #[sealed]
• FlatMessageFlags with #[sealed]
• FlatMessageVariant with #[sealed]

Sealed vs Non-Sealed Enums

Hash Calculation Difference

#![allow(unused)]
fn main() {
use flat_message::*;

// Non-sealed enum (default)
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Status {
    Active = 1,
    Inactive = 2,
}
// Hash: Hash("Status") - only name matters

// Sealed enum
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
#[sealed]
enum Protocol {
    Http = 1,
    Https = 2,
}
// Hash: Hash("Protocol" + "Http" + "Https") - includes all variants
}

Compatibility Behavior

Non-sealed enums allow adding new variants:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Color {
    Red = 1,
    Green = 2,
    Blue = 3,
}

// Version 2 - Compatible with Version 1 data
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
enum Color {
    Red = 1,
    Green = 2,
    Blue = 3,
    Yellow = 4,    // New variant - still compatible
}
}

Sealed enums break compatibility when modified:

#![allow(unused)]
fn main() {
// Version 1
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
#[sealed]
enum SecurityLevel {
    Public = 1,
    Internal = 2,
    Confidential = 3,
}

// Version 2 - NOT compatible with Version 1
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
#[sealed]
enum SecurityLevel {
    Public = 1,
    Internal = 2,
    Confidential = 3,
    TopSecret = 4,  // Adding this changes the hash completely
}
}

Sealed vs Non-Sealed Flags

Flags behave similarly to enums regarding the sealed attribute:

#![allow(unused)]
fn main() {
// Non-sealed flags (default)
#[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
#[repr(transparent)]
#[flags(Read, Write, Execute)]
pub struct Permissions(u8);
impl Permissions {
    add_flag!(Read = 1);
    add_flag!(Write = 2);
    add_flag!(Execute = 4);
}
// Hash: Hash("Permissions") - can add new flags later

// Sealed flags
#[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
#[repr(transparent)]
#[sealed]
#[flags(Admin, User, Guest)]
pub struct UserType(u8);
impl UserType {
    add_flag!(Admin = 1);
    add_flag!(User = 2);
    add_flag!(Guest = 4);
}
// Hash: Hash("UserType" + "Admin" + "User" + "Guest") - strict
}

Flag Evolution Example

#![allow(unused)]
fn main() {
// Version 1: Non-sealed flags
#[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
#[repr(transparent)]
#[flags(Debug, Info, Warning, Error)]
pub struct LogFlags(u16);
impl LogFlags {
    add_flag!(Debug = 1);
    add_flag!(Info = 2);
    add_flag!(Warning = 4);
    add_flag!(Error = 8);
}

// Version 2: Add new flag - compatible
#[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
#[repr(transparent)]
#[flags(Debug, Info, Warning, Error, Fatal)]
pub struct LogFlags(u16);
impl LogFlags {
    add_flag!(Debug = 1);
    add_flag!(Info = 2);
    add_flag!(Warning = 4);
    add_flag!(Error = 8);
    add_flag!(Fatal = 16);  // New flag - works with old data
}
}

Sealed vs Non-Sealed Variants

Variants (sum types) also support the sealed attribute:

#![allow(unused)]
fn main() {
// Non-sealed variant (default)
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
enum ApiResponse {
    Success(String),
    Error(u32),
}
// Hash: Hash("ApiResponse") - can add new variants

// Sealed variant
#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
#[sealed]
enum DatabaseCommand {
    Insert(String),
    Update(String),
    Delete(u32),
}
// Hash: Hash("DatabaseCommand" + "Insert" + "Update" + "Delete") - strict
}

Variant Evolution

#![allow(unused)]
fn main() {
// Version 1: Non-sealed variant
mod v1 {
    use flat_message::*;
    
    #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
    pub enum NetworkEvent {
        Connect(String),
        Disconnect,
    }
}

// Version 2: Add new variant - compatible
mod v2 {
    use flat_message::*;
    
    #[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
    pub enum NetworkEvent {
        Connect(String),
        Disconnect,
        Timeout(u32),  // New variant - works with v1 data
    }
}
}

When to Use Sealed Types

Use Sealed Types When:

1. Security-critical: Cryptographic algorithms, security levels
2. Stable protocols: Fixed command sets that shouldn't change
3. Data integrity: Any modification should break compatibility
4. Exact matching required: Systems must have identical type definitions

#![allow(unused)]
fn main() {
// Good candidates for sealed
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u8)]
#[sealed]
enum CryptoAlgorithm {
    Aes256 = 1,
    ChaCha20 = 2,
    // These must not change - security critical
}

#[derive(Copy, Clone, FlatMessageFlags, Eq, PartialEq, Debug)]
#[repr(transparent)]
#[sealed]
#[flags(Create, Read, Update, Delete)]
pub struct CrudOperations(u8);
// CRUD is stable and complete
}

Use Non-Sealed Types When:

1. Expected evolution: Types will grow over time
2. Forward compatibility: Older code should handle newer types
3. API evolution: Public interfaces that might expand
4. Configuration options: Settings that may increase

#![allow(unused)]
fn main() {
// Good candidates for non-sealed
#[derive(Copy, Clone, FlatMessageEnum, PartialEq, Eq, Debug)]
#[repr(u16)]
enum HttpStatus {
    Ok = 200,
    NotFound = 404,
    InternalError = 500,
    // Many more status codes might be added
}

#[derive(FlatMessageVariant, Debug, PartialEq, Eq)]
enum LogLevel {
    Debug,
    Info(String),
    Warning(String),
    Error(String),
    // Might add Critical, Trace, etc.
}
}

Changing Type

When evolving your data structures over time, you may need to change the type of a field. FlatMessage provides specific behavior and compatibility rules when field types change between versions. Understanding these rules is crucial for maintaining backward and forward compatibility.

Overview

FlatMessage identifies fields using a combination of:

• Field name
• Type information (including representation for complex types)

When you change a field's type, FlatMessage's behavior depends on whether the field can still be identified and matched between versions.

Field Identification Rules

Primitive Types

For primitive types (u8, u16, String, Option<T>, etc.), the type is part of the field's identity. Changing a primitive type means the field will not be found during deserialization.

Example:

#![allow(unused)]
fn main() {
// Version 1
struct Message {
    value: u8,  // Field identified as "value:u8"
}

// Version 2
struct Message {
    value: u16, // Field identified as "value:u16" - different from v1
}
}

Complex Types (Enums, Flags, Variants)

For complex types with repr attributes, FlatMessage uses the representation type for field identification:

Same Representation (Field Found):

#![allow(unused)]
fn main() {
// Version 1
#[derive(FlatMessageEnum)]
#[repr(u8)]
enum Color { Red, Green, Blue }

struct Message {
    #[flat_message_item(repr = u8, kind = enum)]
    color: Color,  // Field identified as "color:u8"
}

// Version 2
#[derive(FlatMessageEnum)]
#[repr(u8)]
enum Shade { Light, Dark }

struct Message {
    #[flat_message_item(repr = u8, kind = enum)]
    color: Shade,  // Field identified as "color:u8" - SAME as v1
}
}

Different Representation (Field Not Found):

#![allow(unused)]
fn main() {
// Version 1
#[derive(FlatMessageEnum)]
#[repr(u8)]
enum Color { Red, Green, Blue }

struct Message {
    #[flat_message_item(repr = u8, kind = enum)]
    color: Color,  // Field identified as "color:u8"
}

// Version 2
#[derive(FlatMessageEnum)]
#[repr(u16)]
enum Shade { Light, Dark }

struct Message {
    #[flat_message_item(repr = u16, kind = enum)]
    color: Shade,  // Field identified as "color:u16" - DIFFERENT from v1
}
}

Compatibility Behavior

The behavior when deserializing depends on two field attributes and whether the field is found:

When Field is Found (Same Name + Compatible Type)

Key Insight: When a field is found but types don't match exactly, the mandatory attribute is irrelevant because the field exists. Only validate determines the outcome.

When Field is Not Found (Different Name or Incompatible Type)

Key Insight: When a field is not found, the validate attribute is irrelevant because no validation occurs. Only mandatory determines the outcome.

Type Change Scenarios

Primitive Type Changes

All primitive type changes result in field not being found:

#![allow(unused)]
fn main() {
// v1: value: u8 → v2: value: u16
// v1: text: String → v2: text: u32  
// v1: data: Option<u8> → v2: data: Option<u16>
}

Outcome: Field not found → mandatory attribute determines success/failure

Complex Type Changes - Same Representation

#![allow(unused)]
fn main() {
// Enums with same repr
#[repr(u8)] enum Color → #[repr(u8)] enum Shade

// Flags with same repr  
#[repr(u8)] struct Permissions → #[repr(u8)] struct Rights

// Variants with same repr
#[repr(u8)] enum Status → #[repr(u8)] enum Mode
}

Outcome: Field found → validate attribute determines success/failure

Complex Type Changes - Different Representation

#![allow(unused)]
fn main() {
// Enums with different repr
#[repr(u8)] enum Color → #[repr(u16)] enum Shade

// Flags with different repr
#[repr(u8)] struct Permissions → #[repr(u16)] struct Rights

// Variants with different repr
#[repr(u8)] enum Status → #[repr(u16)] enum Mode
}

Outcome: Field not found → mandatory attribute determines success/failure

Default Values

When deserialization succeeds but uses fallback behavior, FlatMessage uses the default value for the target type:

• Primitive types: Language defaults (0 for integers, "" for strings, None for options)
• Enums: The variant marked with #[default]
• Flags: Empty flags (no flags set)
• Variants: The variant marked with #[default]

Best Practices

For Backward Compatibility

1. Avoid changing primitive types - they always break compatibility
2. Keep representation consistent for complex types when possible
3. Use mandatory = false for fields that might be removed or changed
4. Use validate = fallback to gracefully handle type mismatches

For Forward Compatibility

1. Plan representation carefully - changing repr breaks compatibility
2. Consider using Option<T> to make fields truly optional
3. Document default values clearly for fallback scenarios

Migration Strategies

1. Gradual migration: Introduce new field, deprecate old field
2. Representation preservation: Keep same repr when changing complex types
3. Fallback-friendly defaults: Ensure default values are meaningful for your application

Error Types

When type changes cause deserialization failures:

• Error::FieldIsMissing: Field not found (different types or names)
• Error::FailToDeserialize: Field found but type validation failed with strict validation

Nested Structures and Version Compatibility

When working with nested structures in FlatMessage, understanding how different struct types behave during version changes is crucial for maintaining compatibility between different versions of your application. This chapter explains the compatibility implications of using FlatMessageStruct and FlatMessagePacked types as nested structures.

Overview

FlatMessage supports two types of nested structures, each with different characteristics and compatibility behaviors:

1. FlatMessageStruct - Hash-based structures with flexible compatibility
2. FlatMessagePacked - Hash-validated structures with strict compatibility

The choice between these types significantly impacts how your data structures can evolve over time.

FlatMessageStruct - Hash-Based Structures

FlatMessageStruct uses a hash table approach for field storage and lookup, making it more flexible for version evolution.

Key Characteristics

• Field Lookup: Uses hash tables for field identification and access
• Memory Layout: Not sequential - fields can be accessed in any order
• Metadata Support: Supports Timestamp and UniqueID metadata fields (though they are ignored during serialization in nested contexts)
• Option Support: Can be wrapped in Option<T>
• Validation: Supports field-level validation attributes
• Compatibility: Can be compatible with different versions of the same struct (pending a proper usage of mandatory and validate attributes)

Basic Usage

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
struct UserProfile {
    pub name: String,
    pub age: u32,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
struct User {
    pub id: u8,
    #[flat_message_item(kind = struct, align = 4)]
    pub profile: UserProfile,
}
}

FlatMessagePacked - Hash-Validated Structures

FlatMessagePacked uses a sequential memory layout with hash validation for structure integrity.

Key Characteristics

• Memory Layout: Sequential memory layout for optimal performance
• Hash Validation: Uses structure hash for validation during deserialization
• No Metadata: No support for Timestamp or UniqueID metadata fields
• No Options: Cannot be wrapped in Option<T>
• Field Ordering: Fields are automatically reordered by alignment for optimal packing
• Compatibility: Strict - any structural change breaks compatibility

Basic Usage

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
struct Coordinates {
    pub x: f32,
    pub y: f32,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
struct Location {
    pub id: u8,
    #[flat_message_item(kind = packed, align = 4)]
    pub coords: Coordinates,
}
}

Version Compatibility Scenarios

The following scenarios demonstrate how different structural changes affect compatibility between versions. These are based on comprehensive test cases that verify the actual behavior.

Scenario 1: FlatMessageStruct - Adding Mandatory Fields with Strict Validation

Version 1:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    pub value: u32,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = struct, align = 4)]
    pub nested: NestedStruct,
}
}

Version 2:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    pub value: u32,
    pub new_field: u16, // New mandatory field added
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = struct, align = 4, validate = strict)]
    pub nested: NestedStruct,
}
}

Compatibility:

• V1 → V2: ❌ FAILS - Missing mandatory field causes deserialization failure
• V2 → V1: ✅ SUCCEEDS - Extra fields are ignored

Scenario 2: FlatMessageStruct - Adding Optional Fields

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    pub value: u32,
    #[flat_message_item(mandatory = false, validate = fallback, default = 42)]
    pub new_field: u16, // New optional field with default
}
}

Compatibility:

• V1 → V2: ✅ SUCCEEDS - Missing optional field uses default value (42)
• V2 → V1: ✅ SUCCEEDS - Extra fields are ignored

Scenario 3: FlatMessageStruct - Fallback Validation on Parent Field

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = struct, align = 4, validate = fallback)]
    pub nested: NestedStruct, // validate = fallback on the field itself
}

impl Default for NestedStruct {
    fn default() -> Self {
        Self { value: 0, new_field: 0 }
    }
}
}

Compatibility:

• V1 → V2: ✅ SUCCEEDS - When struct deserialization fails, uses Default::default()
• V2 → V1: ✅ SUCCEEDS - Extra fields are ignored

Scenario 4: FlatMessageStruct - Fallback Validation on Child Fields

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    #[flat_message_item(validate = fallback)]
    pub value: u32,
    #[flat_message_item(validate = fallback)]
    pub new_field: u16, // validate = fallback on individual fields
}
}

Compatibility:

• V1 → V2: ❌ FAILS - Field-level fallback doesn't help with missing mandatory fields
• V2 → V1: ✅ SUCCEEDS - Extra fields are ignored

Key Insight: validate = fallback on individual struct fields only helps with field-level validation issues, not with missing mandatory fields at the struct level.

Scenario 5: FlatMessageStruct - Mandatory = False on Parent Field

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = struct, align = 4, mandatory = false)]
    pub nested: NestedStruct, // mandatory = false on the field
}
}

Compatibility:

• V1 → V2: ❌ FAILS - mandatory = false doesn't help if the struct content has structural changes
• V2 → V1: ✅ SUCCEEDS - Extra fields are ignored

Key Insight: Setting mandatory = false on the parent field doesn't help when the nested struct itself has incompatible changes.

Scenario 6: FlatMessageStruct - Removing Fields

Version 1:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    pub value: u32,
    pub old_field: u16,
}
}

Version 2:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessageStruct)]
pub struct NestedStruct {
    pub value: u32, // old_field removed
}
}

Compatibility:

• V1 → V2: ✅ SUCCEEDS - Extra fields in data are ignored
• V2 → V1: ❌ FAILS - Missing mandatory field causes deserialization failure

Scenario 7: FlatMessagePacked - Adding Fields

Version 1:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
pub struct NestedStruct {
    pub value: u32,
}
}

Version 2:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
pub struct NestedStruct {
    pub value: u32,
    pub new_field: u16, // Any change breaks compatibility
}
}

Compatibility:

• V1 → V2: ❌ FAILS - Hash validation detects structural change
• V2 → V1: ❌ FAILS - Hash validation detects structural change

Key Insight: ANY structural change to a packed struct breaks compatibility due to hash validation.

Scenario 8: FlatMessagePacked - Fallback Validation on Parent Field

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = packed, align = 1, validate = fallback)]
    pub nested: NestedStruct,
}

impl Default for NestedStruct {
    fn default() -> Self {
        Self { value: 999, new_field: 888 }
    }
}
}

Compatibility:

• V1 → V2: ✅ SUCCEEDS - When packed struct validation fails, uses Default::default()
• V2 → V1: ❌ FAILS - Hash validation still detects structural change

Scenario 9: FlatMessagePacked - Mandatory = False on Parent Field

Version 2 (Modified):

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = packed, align = 1, mandatory = false)]
    pub nested: NestedStruct,
}
}

Compatibility:

• V1 → V2: ❌ FAILS - Hash validation fails before mandatory = false can take effect
• V2 → V1: ❌ FAILS - Hash validation detects structural change

Scenario 10: FlatMessagePacked - Type Changes

Version 1:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
pub struct NestedStruct {
    pub value: u32,
    pub data: u8,
}
}

Version 2:

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessagePacked)]
pub struct NestedStruct {
    pub value: u32,
    pub data: u16, // Type changed from u8 to u16
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
pub struct Test {
    pub id: u8,
    #[flat_message_item(kind = packed, align = 1, validate = fallback)]
    pub nested: NestedStruct,
}
}

Compatibility:

• V1 → V2: ✅ SUCCEEDS - Hash validation fails, falls back to default
• V2 → V1: ❌ FAILS - Hash validation detects structural change

Summary and Best Practices

FlatMessageStruct Compatibility Rules

✅ Compatible Changes:

• Adding optional fields (mandatory = false)
• Removing fields (forward compatibility only)
• Changing field order (fields are hash-based)

❌ Incompatible Changes:

• Adding mandatory fields without fallback strategies
• Removing fields that newer versions still expect

FlatMessagePacked Compatibility Rules

✅ Compatible Changes:

• None - packed structs are designed for performance, not evolution

❌ Incompatible Changes:

• Adding any fields
• Removing any fields
• Changing field types
• Changing field order (though order is automatically optimized)

Mitigation Strategies

For FlatMessageStruct

1. Use Optional Fields: Mark new fields as mandatory = false

   #![allow(unused)]
   fn main() {
   #[flat_message_item(mandatory = false, default = 42)]
   pub new_field: u16,
   }

2. Use Fallback Validation on Parent: Apply validate = fallback to the struct field

   #![allow(unused)]
   fn main() {
   #[flat_message_item(kind = struct, align = 4, validate = fallback)]
   pub nested: NestedStruct,
   }

3. Implement Default Carefully: Ensure Default implementations make sense for your domain

   #![allow(unused)]
   fn main() {
   impl Default for NestedStruct {
       fn default() -> Self {
           Self { value: 0, new_field: 100 }
       }
   }
   }

For FlatMessagePacked

1. Use Fallback Validation: Apply validate = fallback to the packed struct field

   #![allow(unused)]
   fn main() {
   #[flat_message_item(kind = packed, align = 4, validate = fallback)]
   pub packed_data: PackedStruct,
   }

2. Design for Immutability: Treat packed structs as immutable once deployed

3. Version at Container Level: Use versioning on the parent FlatMessage struct instead

When to Use Each Type

Choose FlatMessageStruct when:

• You need version evolution capabilities
• Field access patterns are random or sparse
• You have optional/metadata fields
• Schema flexibility is important

Choose FlatMessagePacked when:

• Maximum performance is critical
• Memory layout optimization is important
• The structure is stable and won't change
• You need the smallest possible serialized size

Advanced Features

While FlatMessage provides a lot of flexibility and power out of the box, there are some advanced features that you can use to get the most out of it, including:

• how some types can be interchangeable
• how to set default values
• how to ignore fields
• how to use checksums and validation

Type Interchangeability

One of FlatMessage's powerful features is the ability to use different but compatible types during serialization and deserialization. This flexibility enables efficient memory usage and performance optimization.

1. Vec vs Slice Interchangeability

You can serialize data with Vec<T> and deserialize it as &[T], or vice versa:

#![allow(unused)]
fn main() {
use flat_message::*;

// Serialization struct using Vec
#[derive(FlatMessage)]
struct DataWriter {
    numbers: Vec<u32>,
    names: Vec<String>,
}

// Deserialization struct using slices (zero-copy)
#[derive(FlatMessage)]
struct DataReader<'a> {
    numbers: &'a [u32],      // Zero-copy reference
    names: Vec<&'a str>,,    // Zero-copy reference for &str, allocation for Vec
}

fn example() -> Result<(), Error> {
    // Create data with Vec (ownership)
    let writer_data = DataWriter {
        numbers: vec![1, 2, 3, 4, 5],
        names: vec!["Alice".to_string(), "Bob".to_string()],
    };

    // Serialize
    let mut storage = Storage::default();
    writer_data.serialize_to(&mut storage, Config::default())?;

    // Deserialize as slices (zero-copy)
    let reader_data = DataReader::deserialize_from(&storage)?;
    
    println!("Numbers: {:?}", reader_data.numbers);  // [1, 2, 3, 4, 5]
    println!("Names: {:?}", reader_data.names);      // ["Alice", "Bob"]
    
    Ok(())
}
}

Performance Implications

Best Practice: Use Vec<T> for data you own/modify, &[T] for read-only access.

2. Option vs Non-Option Interchangeability

Fields can be compatible between Option<T> and T under certain conditions:

#![allow(unused)]
fn main() {
// Serialization with Option
#[derive(FlatMessage)]
struct OptionalData {
    required_field: u32,
    optional_field: Option<String>,
    optional_number: Option<i64>,
}

// Deserialization without Option
#[derive(FlatMessage)]
struct RequiredData {
    required_field: u32,
    optional_field: String,  // Must exist in data
    optional_number: i64,    // Must exist in data
}

fn option_compatibility() -> Result<(), Error> {
    // Serialize with Some values
    let opt_data = OptionalData {
        required_field: 42,
        optional_field: Some("Hello".to_string()),
        optional_number: Some(123),
    };

    let mut storage = Storage::default();
    opt_data.serialize_to(&mut storage, Config::default())?;

    // Deserialize as required fields
    let req_data = RequiredData::deserialize_from(&storage)?;
    assert_eq!(req_data.optional_field, "Hello");
    assert_eq!(req_data.optional_number, 123);

    Ok(())
}
}

Compatibility Rules

Handling None Values

When deserializing None as a required field, you get an error:

#![allow(unused)]
fn main() {
fn handle_missing_optional() -> Result<(), Error> {
    let opt_data = OptionalData {
        required_field: 42,
        optional_field: None,       // This will cause an error
        optional_number: Some(123),
    };

    let mut storage = Storage::default();
    opt_data.serialize_to(&mut storage, Config::default())?;

    // This will fail because optional_field is None
    match RequiredData::deserialize_from(&storage) {
        Err(Error::FieldIsMissing(_)) => {
            println!("Field was None, can't deserialize as required");
        }
        _ => {}
    }

    Ok(())
}
}

3. String vs &str Interchangeability

Similar rules apply to strings:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct StringWriter {
    owned_text: String,
    optional_text: Option<String>,
}

#[derive(FlatMessage)]
struct StringReader<'a> {
    owned_text: &'a str,        // String -> &str (zero-copy)
    optional_text: &'a str,     // Option<String> -> &str (if Some)
}

fn string_example() -> Result<(), Error> {
    let writer = StringWriter {
        owned_text: "Hello World".to_string(),
        optional_text: Some("Optional text".to_string()),
    };

    let mut storage = Storage::default();
    writer.serialize_to(&mut storage, Config::default())?;

    let reader = StringReader::deserialize_from(&storage)?;
    println!("Text: {}", reader.owned_text);        // "Hello World"
    println!("Optional: {}", reader.optional_text); // "Optional text"

    Ok(())
}
}

Compatibility Matrix

Default Values

There are several scenarios where a default value will be used to initialize a field during deserialization:

1. The field is not mandatory and is not present in the serialized data.
2. The field is present in the serialized data, but it has some issues trying to deserialize it and the validate attribute is set to fallback.
3. The field is skipped during serialization and it has to be defaulted during deserialization.

In these scenarios, FlatMessage will do one of the following:

1. Use the default value for the type if it is available (this implies that the type implements the Default trait).
2. If the attribute default is specified, it will use the value of the attribute.

Example:

1. Use the default value:

   #![allow(unused)]
   fn main() {
   #[derive(FlatMessage)]
   struct Test {
       #[flat_message_item(skip = true)]
       a: u32,
   }
   }

   In this case, the field a will be initialized to 0 (the default value for u32).
2. Use the value of the attribute default:

   #![allow(unused)]
   fn main() {
   #[derive(FlatMessage)]
   struct Test {
       #[flat_message_item(skip = true, default = 10)]
       a: u32,
   }
   }

   In this case, the field a will be initialized to 10.

Custom Default Values

When using the attribute default, you can specify a custom default value for the field in the following ways:

1. A constant value (e.g. default = 10 or default = MY_CONSTANT).
2. A string representation of the value (e.g. default = "10" ). In this case the value is parsed and adjusted to fit the actual type of the field.
3. A raw string representation of the value (e.g. default = r#"foo(1,2,3)"#). In this case the value is use exactly as is. This is in particular useul if you want to use exprssior or a call to a functon to initialize the value.

String representations

When using a string representation: default = "..." the following steps are checked:

1. if the type is &str the value of the default attribute is kept as it is.
2. if the type is String th value of the default attribute is converted into a String
3. if the type is NOT a string the quotes (") are removed and the actual value will be used
4. if the type is on option ( Option<T> ) then:
   • if the value of the default attribute is None then the field is set to None
   • if the value of the default attribute is Some(T) then the field is set to Some(T)
   • otherwise the value of the default attribute is converted into a Some<value>

Examples

Remark: If you need to be 100% sure that the value is converted into the correct type, you can use the raw string representation (e.g. default = r#"Some(1+2+3)"#).

Ignoring Fields

FlatMessage provides a powerful mechanism to exclude specific fields from serialization and deserialization using the ignore attribute. This feature is particularly useful for fields that contain runtime-only data, computed values, or zero-sized types that don't need to be persisted. Ignored fields have no impact on the binary representation. The serialized data will not contain any information about ignored fields.

Basic Usage

To ignore a field during serialization and deserialization, use the #[flat_message_item(ignore = true)] attribute:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
#[flat_message_options(store_name = false)]
struct User {
    id: u32,
    name: String,
    #[flat_message_item(ignore = true)]
    cached_score: u32,  // This field will be ignored
}
}

When serializing a User struct, the cached_score field will not be included in the binary data. During deserialization, this field will be set to its default value.

Alternative Syntax

FlatMessage also supports the skip attribute as an alias for ignore:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = false)]
struct Data {
    value: u8,
    #[flat_message_item(skip = true)]
    temp_data: u32,  // Equivalent to ignore = true
}
}

Both ignore = true and skip = true have identical behavior.

Default Values

Using Type's Default

When a field is ignored, it will be initialized with the type's default value during deserialization:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = false)]
struct Example {
    x: u8,
    #[flat_message_item(ignore = true)]
    y: u32,  // Will be 0 (u32's default) after deserialization
}

let data = Example { x: 1, y: 999 };
let mut storage = Storage::default();
data.serialize_to(&mut storage, Config::default()).unwrap();

let deserialized = Example::deserialize_from(&storage).unwrap();
assert_eq!(deserialized.x, 1);
assert_eq!(deserialized.y, 0);  // Default value, not 999
}

Custom Default Values

You can specify a custom default value for ignored fields using the default attribute:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = false)]
struct Config {
    version: u8,
    #[flat_message_item(ignore = true, default = 42)]
    cache_size: u32,  // Will be 42 after deserialization
}

let config = Config { version: 1, cache_size: 100 };
let mut storage = Storage::default();
config.serialize_to(&mut storage, Config::default()).unwrap();

let deserialized = Config::deserialize_from(&storage).unwrap();
assert_eq!(deserialized.version, 1);
assert_eq!(deserialized.cache_size, 42);  // Custom default value
}

Zero-Sized Types (ZST)

FlatMessage automatically ignores zero-sized types like PhantomData without requiring explicit attributes:

#![allow(unused)]
fn main() {
use std::marker::PhantomData;

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
struct Container<T> {
    data: u8,
    _phantom: PhantomData<T>,  // Automatically ignored
}
}

Zero-sized types are treated as if they have ignore = true applied automatically.

Use Cases

Runtime-Only Data

Ignore fields that are only meaningful during runtime:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Session {
    user_id: u32,
    created_at: u64,
    #[flat_message_item(ignore = true)]
    last_access_time: u64,  // Updated frequently, no need to persist
    #[flat_message_item(ignore = true)]
    is_active: bool,        // Runtime state
}
}

Computed Values

Ignore fields that can be computed from other data:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct Rectangle {
    width: f32,
    height: f32,
    #[flat_message_item(ignore = true, default = 0.0)]
    area: f32,  // Can be computed as width * height
}

impl Rectangle {
    fn new(width: f32, height: f32) -> Self {
        Self {
            width,
            height,
            area: width * height,
        }
    }
    
    fn compute_area(&mut self) {
        self.area = self.width * self.height;
    }
}
}

Temporary Buffers

Ignore fields used as temporary storage:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
struct DataProcessor {
    input: Vec<u8>,
    output: Vec<u8>,
    #[flat_message_item(ignore = true)]
    temp_buffer: Vec<u8>,  // Working space, no need to serialize
}
}

Performance Benefits

Ignoring unnecessary fields can improve performance by:

• Reducing serialized data size
• Decreasing serialization/deserialization time
• Minimizing memory usage for persistent storage
• Enabling better compression ratios

Checksum Validation

FlatMessage provides built-in data integrity features through checksums and various validation levels to ensure your data hasn't been corrupted during storage or transmission.

Checksum Basics

Enable checksums to detect data corruption:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage, Debug, PartialEq)]
#[flat_message_options(checksum = true)]
struct ImportantData {
    value: u64,
    message: String,
}

fn checksum_example() -> Result<(), Error> {
    let data = ImportantData {
        value: 12345,
        message: "Critical information".to_string(),
    };

    let mut storage = Storage::default();
    data.serialize_to(&mut storage, Config::default())?;

    // Checksum is automatically calculated and stored
    println!("Serialized with checksum: {} bytes", storage.len());

    // Checksum is automatically verified during deserialization
    let restored = ImportantData::deserialize_from(&storage)?;
    assert_eq!(data, restored);

    Ok(())
}
}

Checksum Validation Modes

The validate_checksum option controls when checksums are validated. You can set it to:

• always --> checksum should always be validated
• never --> checksum is ignored
• auto --> chcksum is checked only if present in the deserialized data

#![allow(unused)]
fn main() {
// Always validate checksums
#[derive(FlatMessage)]
#[flat_message_options(checksum = true, validate_checksum = "always")]
struct AlwaysValidated {
    data: Vec<u8>,
}

// Never validate checksums (performance optimization)
#[derive(FlatMessage)]
#[flat_message_options(checksum = true, validate_checksum = "never")]
struct NeverValidated {
    data: Vec<u8>,
}

// Auto validation (default) - validate only if checksum is present
#[derive(FlatMessage)]
#[flat_message_options(checksum = true, validate_checksum = "auto")]
struct AutoValidated {
    data: Vec<u8>,
}
}

Validation Mode Behavior

Corruption Detection

Checksums detect various types of data corruption:

#![allow(unused)]
fn main() {
fn corruption_detection_example() -> Result<(), Box<dyn std::error::Error>> {
    // Create and serialize data
    let original = ImportantData {
        value: 42,
        message: "Hello World".to_string(),
    };

    let mut storage = Storage::default();
    original.serialize_to(&mut storage, Config::default())?;

    // Simulate corruption by modifying bytes
    let mut corrupted_bytes = storage.as_slice().to_vec();
    corrupted_bytes[10] = 0xFF;  // Corrupt a byte
    let corrupted_storage = Storage::from_buffer(&corrupted_bytes);

    // Attempt to deserialize corrupted data
    match ImportantData::deserialize_from(&corrupted_storage) {
        Err(Error::InvalidChecksum((actual, expected))) => {
            println!("Corruption detected!");
            println!("Expected checksum: 0x{:08X}", expected);
            println!("Actual checksum: 0x{:08X}", actual);
        }
        Ok(_) => panic!("Should have detected corruption"),
        Err(e) => println!("Other error: {}", e),
    }

    Ok(())
}
}

Performance vs Safety Trade-offs

It is important to note that checksum validation are only performed when using deserialize_from(). If you use deserialize_from_unchecked(), the checksum validation is skipped (as the data is considered trusted).

This means that deserialize_from_unchecked() is always faster than deserialize_from(), as it skips the checksum validation and other checks, but it is unsafe as if used incorrectly, it can lead to data corruption.

Message Name Validation

FlatMessage provides a robust name validation system that allows you to control whether structure names are stored during serialization and validated during deserialization. This feature helps ensure type safety and prevents accidental deserialization of incompatible data structures.

Overview

The name validation system uses two key attributes that work together:

• store_name: Controls whether the structure name hash is stored in the serialized data
• validate_name: Controls whether the structure name is validated during deserialization

Attributes

store_name Attribute

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = true)]  // Default: true
struct MyStruct {
    value: u32,
}
}

When store_name = true:

• A 32-bit hash of the structure name is stored in the serialized data
• Adds 4 bytes to the serialized size
• Enables name validation during deserialization

When store_name = false:

• No name information is stored in the serialized data
• Saves 4 bytes of storage space
• Name validation is impossible during deserialization

validate_name Attribute

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(validate_name = true)] // Default: false
struct MyStruct {
    value: u32,
}
}

When validate_name = true:

• During deserialization, the stored name hash is compared with the target structure's name
• Throws Error::UnmatchedName if names don't match
• Throws Error::NameNotStored if no name was stored during serialization

When validate_name = false (default):

• No name validation is performed during deserialization
• Allows deserialization between different structure types (if fields are compatible)

Attribute Combinations

1. Default Behavior: store_name = true, validate_name = false

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
struct StructA {
    value: u64,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
struct StructB {
    value: u64,
}
}

Behavior:

• Name is stored during serialization (4 extra bytes)
• No validation during deserialization
• Cross-structure deserialization is allowed if fields are compatible

#![allow(unused)]
fn main() {
let a = StructA { value: 42 };
let mut storage = Storage::default();
a.serialize_to(&mut storage, Config::default()).unwrap();

// This works - same structure
let restored_a = StructA::deserialize_from(&storage).unwrap();

// This also works - different structure but compatible fields
let restored_b = StructB::deserialize_from(&storage).unwrap();
}

2. Strict Validation: store_name = true, validate_name = true

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = true, validate_name = true)]
struct StrictStruct {
    value: u64,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(validate_name = true)]
struct OtherStruct {
    value: u64,
}
}

Behavior:

• Name is stored during serialization
• Name is validated during deserialization
• Cross-structure deserialization fails with Error::UnmatchedName

#![allow(unused)]
fn main() {
let strict = StrictStruct { value: 42 };
let mut storage = Storage::default();
strict.serialize_to(&mut storage, Config::default()).unwrap();

// This works - correct structure
let restored = StrictStruct::deserialize_from(&storage).unwrap();

// This fails - different structure name
match OtherStruct::deserialize_from(&storage) {
    Err(Error::UnmatchedName) => {
        println!("Name validation prevented cross-type deserialization");
    }
    _ => panic!("Expected name validation error"),
}
}

3. Space Optimized: store_name = false, validate_name = false

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false, validate_name = false)]
struct CompactStruct {
    value: u64,
}
}

Behavior:

• No name is stored (saves 4 bytes)
• No validation is performed
• Maximum interoperability between compatible structures

#![allow(unused)]
fn main() {
let compact = CompactStruct { value: 42 };
let mut storage = Storage::default();
compact.serialize_to(&mut storage, Config::default()).unwrap();

// Works with any compatible structure
let restored = AnyCompatibleStruct::deserialize_from(&storage).unwrap();
}

4. Invalid Combination: store_name = false, validate_name = true

#![allow(unused)]
fn main() {
#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false, validate_name = true)]
struct InvalidConfig {
    value: u64,
}

#[derive(Debug, PartialEq, Eq, FlatMessage)]
#[flat_message_options(store_name = false)]
struct DataSource {
    value: u64,
}
}

Behavior:

• No name is stored during serialization
• Validation is attempted during deserialization
• Always fails with Error::NameNotStored

#![allow(unused)]
fn main() {
let source = DataSource { value: 42 };
let mut storage = Storage::default();
source.serialize_to(&mut storage, Config::default()).unwrap();

// This always fails because no name was stored
match InvalidConfig::deserialize_from(&storage) {
    Err(Error::NameNotStored) => {
        println!("Cannot validate name when none was stored");
    }
    _ => panic!("Expected NameNotStored error"),
}
}

Error Types

Error::UnmatchedName

Occurs when:

• validate_name = true on the target structure
• A name hash was stored in the serialized data
• The stored name hash doesn't match the target structure's name hash

Error::NameNotStored

Occurs when:

• validate_name = true on the target structure
• No name hash was stored in the serialized data (source had store_name = false)

Use Cases

Type Safety in APIs

Use strict validation when deserializing data from external sources:

#![allow(unused)]
fn main() {
#[derive(FlatMessage)]
#[flat_message_options(store_name = true, validate_name = true)]
struct ApiRequest {
    user_id: u64,
    action: String,
}

fn handle_request(data: &[u8]) -> Result<(), Error> {
    // This will fail if the data wasn't serialized as ApiRequest
    let request = ApiRequest::deserialize_from(data)?;
    // Process request safely...
    Ok(())
}
}

Best Practices

1. Use strict validation (store_name = true, validate_name = true) for:

   • Network protocol messages
   • API endpoints
   • Critical data structures where type safety is paramount

2. Use flexible validation (store_name = true, validate_name = false) for:

   • Data persistence with potential schema evolution
   • Internal application data structures
   • When you need some metadata but want flexibility

3. Disable name storage (store_name = false) for:

   • High-frequency data (sensor readings, telemetry)
   • Memory-constrained environments
   • When every byte counts and type safety is managed elsewhere

4. Avoid the invalid combination (store_name = false, validate_name = true):

   • This configuration will always fail during deserialization
   • The compiler doesn't prevent this, so be careful with your configuration

Runtime Introspection

You can inspect stored names using StructureInformation:

#![allow(unused)]
fn main() {
use flat_message::*;

#[derive(FlatMessage)]
#[flat_message_options(store_name = true)]
struct MyStruct {
    value: u32,
}

fn inspect_name(storage: &Storage) -> Result<(), Error> {
    let info = StructureInformation::try_from(storage)?;
    
    if let Some(name) = info.name() {
        match name {
            name!("MyStruct") => println!("Found MyStruct data"),
            name!("OtherStruct") => println!("Found OtherStruct data"),
            _ => println!("Found unknown structure: {:?}", name),
        }
    } else {
        println!("No name information stored");
    }
    
    Ok(())
}
}

The name validation system provides a flexible balance between type safety, storage efficiency, and interoperability, allowing you to choose the right trade-offs for your specific use case.

Benchmarks

In terms of benchmarks, the following metrics can be used to evaluate a serialization/deserialization process:

1. Speed - how fast is the serialization/deserialization process is compared to other serializers/deserializers
2. Size - how much space is needed to store the serialized data (in particular for a schema-less serializer, like FlatMessage, the size needed to store the metadata is relevant)

Additionally, the for FlatMessage another relevant metrics are:

1. Safe vs Unsafe deserialization (and their performance implications)
2. Zero-copy deserialization (and its performance implications)
3. FlatMessageStruct vs FlatMessagePacked for nested structures (and their performance implications in terms of speed and size)

Performance Results

The following tests were conducted against ofther serializers and deserializers crates:

• performance - different structures were serialized and deserialized and the time needed for this operation was measured
• size - the size of the serialized data was measured for different structures

Crates

The following crates were tested:

Methodology

Each test consists doing the following for a chosen structure:

• Ser Time - Serialize the structure for n times (repetitions) and measure the time needed to perform this operations
• Deser Time - Deserialize a buffer containing the serialized data for n times (repetitions) and measure the time needed to perform this operations
• Ser+Deser Time - Serialize and then deserialize the structure for n times (repetitions) and measure the time needed to perform this operations

The n parameter is usually a larger one (>1000) as usually de serialization/deserialization process is really fast and measuring it for a smaller number of times would not be representative.

Each repetition of "n" times is performed for "k" iterations and the times for each iterations are stored. From these, the median time is calculated. We prefer median time over average time as it is less sensitive to outliers.

The result for each tested structure (in terms of time) will be presended in the following way: median [min - mac]. For example: 1.5 [1.2 - 1.8] means that the median time is 1.5ms, the minimum time is 1.2ms and the maximum time is 1.8ms.

The following algorithm simulates how times are computed:

times = []
for iteration in 0..k {
    start = GetCurrentTime()
    for repetition in 0..n {
        Serialize(structure)
    }
    end = GetCurrentTime()
    times.push(end - start)
}
return (median(times), min(times), max(times))

For each structure we also compute the Data size (the minimum size required to store the data from that structure). That value is compared to the actual size of the serialized buffer. In most cases (since the serialized buffer is usually bigger than the data size) the percentage of increase is reported. The size value presented for each serialization method is presented as follows: size [+/- percentage]. For example: 355 [+69%] means that the size of the serialized buffer is 355 bytes and the data size is 209 bytes (so the percentage of increase is 69% for that method).

Remarks: It is important to highlight that some of the methods used are not schema-less (they will be marked with schema next to the name of the method). In these cases, it is possible that the actual size will be smaller than the data size (in particular if the serialization method compress some of the data)

OSes

The tests were performed on the following OSes:

1. Windows - Windows 11, 64 bit,11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz (2.80 GHz), RAM 32.0 GB
2. MacOS - MacOS 15.6.1 24G90 arm64, Apple M1 Pro, RAM 32.0 GB
3. Linux - Kubuntu 24.04.3 LTS x86_64, kernel: 6.8.0-71-generic, 11th Gen Intel(R) Core(TM) i7-11850H (16) @ 4.80GHz , RAM 64.0 GB

Overall Speed

All of the above results are averaged over all the tested structures in the following way:

• for each tested structure, we compute the speed (MB/sec) as the data size (bytes) * n (number of repetitions) / time (ms)
• this is done for each OS and then the results are averaged over all the OSes

Remarks:

• There are a lot of variation in the results - and while we did try to use a large variaty of structures, it is best to evaluate the results/structure as well and find the ones that are most appropiate to your use case.
• Protobuf results are inconclusive as they were not aveaged on the entire set of structures.

Multiple Fields

This benchmarks compares the performance of the different algorithms when serializing and deserializing a message with multiple fields. In particula for schema-less messages, this will show how well different message formats store the schema in the message. Fields have different basic types (string, u32, u64, i32, i64, f32, f64, bool, u8, i8, u16, i16).

#![allow(unused)]
fn main() {
pub struct MultipleFields {
    field_of_type_string: String,
    field_of_type_u32: u32,
    field_of_type_u64: u64,
    field_of_type_i32: i32,
    field_of_type_i64: i64,
    field_of_type_f32: f32,
    field_of_type_f64: f64,
    field_of_type_bool: bool,
    field_of_type_u8: u8,
    field_of_type_i8: i8,
    field_of_type_u16: u16,
    field_of_type_i16: i16,
    second_field_of_type_string: String,
    second_field_of_type_u32: u32,
    second_field_of_type_u64: u64,
    second_field_of_type_i32: i32,
    second_field_of_type_i64: i64,
    third_field_of_type_string: String,
    third_field_of_type_u32: u32,
    third_field_of_type_u64: u64,
    third_field_of_type_i32: i32,
    third_field_of_type_i64: i64,
    fourth_field_of_type_string: String,
    fourth_field_of_type_u32: u32,
    fourth_field_of_type_u64: u64,
    fourth_field_of_type_i32: i32,
    fourth_field_of_type_i64: i64,
}
}

Test specs

• Iterations: k = 10
• Serialization and deserialization repetitions / iteration: n = 100000
• Data size: 210 bytes
• Protobuf: Not supported (due to the fields of type i8 and i16 that are not supported by protobuf)

Results

1. Windows Execution

2. MacOs Execution

3. Linux Execution

Point

This benchmarks compares the performance of the different algorithms to serialize and deserialize a point structure. The idea is to see how well the algorithms handle the case of a small structure with 4 bytes alignment.

#![allow(unused)]
fn main() {
pub struct Point {
    x: i32,
    y: i32,
}
}

Test specs

• Iterations: k = 10
• Serialization and deserialization repetitions / iteration: n = 500000
• Data size: 8 bytes
• Protobuf: Supported

Results

1. Windows Execution

2. MacOs Execution

3. Linux Execution

Long Strings

This benchmarks compares the performance of the different algorithms when dealing with a structure that contains multiple long strings. Each of the string will be instantiated with large strings (between 100 and 2000 characters) making the total size of the structure close to 4000 bytes.

#![allow(unused)]
fn main() {
pub struct LongStringStructure {
    string_one: String,
    string_two: String,
    string_three: String,
    string_four: String,
    value_one: u32,
    value_two: u64,
}
}

Test specs

• Iterations: k = 10
• Serialization and deserialization repetitions / iteration: n = 100000
• Data size: 3919 bytes
• Protobuf: Supported

Results

1. Windows Execution

2. MacOs Execution

3. Linux Execution

Large Vectors

This benchmarks compares the performance of the different algorithms when dealing with a structure that contains multiple large vectors. The vectors will be instantiated as follows:

• ints: 2000 elements between 200 and 220
• floats: 10000 elements between -1000000.0 and 1000000.0
• uints: 25000 elements between 0 and 1000000
• doubles: 30000 elements between 0.0 and 1000000.0

#![allow(unused)]
fn main() {
pub struct LargeVectors {
    ints: Vec<i32>,
    floats: Vec<f32>,
    uints: Vec<u32>,
    doubles: Vec<f64>,
}
}

Test specs

• Iterations: k = 10
• Serialization and deserialization repetitions / iteration: n = 100
• Data size: 388008 bytes
• Protobuf: Supported

Results

1. Windows Execution

2. MacOs Execution

3. Linux Execution

Large Vectors

This benchmarks checks to see how well a structure containing different enum fields can be serialized and deserialized. The enum have different sized variants (u8, u32, i64) and are instantiated with different values.

#![allow(unused)]
fn main() {
#[repr(u8)]
enum Color {
    Red = 1,
    Green = 2,
    Blue = 3,
    Yellow = 100,
    Cyan = 101,
    Magenta = 102,
}
#[repr(u32)]
enum Math {
    A = 1,
    B = 1000,
    C = 1000000,
    D = 1000000000,
}
#[repr(i64)]
enum Negative {
    A = 1,
    B = -1000,
    C = 1000000,
    D = -1000000000,
    E = 1000000000000,
    F = -1000000000000000,
}
pub struct EnumFields {
    col: Color,
    math: Math,
    neg: Negative,
}
}

Test specs

• Iterations: k = 10
• Serialization and deserialization repetitions / iteration: n = 500000
• Data size: 13 bytes
• Protobuf: Not Supported (enums can't be used directly in protobuf via prost crate)

Results

1. Windows Execution

2. MacOs Execution

3. Linux Execution