Json Output Formatter

Filter Flow Diagram

                             _________________________
                            |      Input Frames       |
                            |     (raw JSON/YAML)     |
                            |_________________________|
                                        |
                                        v
                             _________________________
                            |    Template Loader      |
                            | (JSON / YAML / Jinja2)  |
                            |  (URL/ File / Inline)   |
                            |_________________________|
                                        |
                                        v
                          _______________________________ 
                         |  Restructure / Jinja2 Render  |
                         |_______________________________|
                                        |
                                        v
                             __________________________
                            |        Validation        |
                            |(Schema / Whitelist/ None)|    
                            |__________________________|
                                        |
                                        v
                             _________________________
                            |      Output Frames      |
                            |     (formatted JSON)    |
                            |_________________________|

Features

Template-based Output Formatting:
Supports JSON, YAML, and Jinja2 templates for output formatting.
Validation:
Supports JSON Schema and custom whitelist validation.
Flexible Template Sources:
Templates can be loaded from local files, inline strings, or remote URLs.
Extensible:
Built to be easily extended for new validation or rendering strategies.
Logging and Debugging:
Integrated logging for easier debugging and monitoring.

Directory Structure

filter-json-output-formatter/
│
├── filter_json_output_formatter/
│   ├── filter.py                # Main filter logic
│   ├── whitelist_validator.py   # Whitelist validation logic
│   └── __init__.py
│
├── tests/                       # (Presumed) Unit tests
├── docs/                        # (Presumed) Documentation
├── build-include/               # (Presumed) Build resources
├── README.md                    # Main documentation
├── Dockerfile                   # Docker support
├── Makefile                     # Build automation
├── pyproject.toml               # Python project config
├── docker-compose.yaml          # Docker Compose support
├── ... (other files)

Environment Variables & Configuration

What are the environment variables for?

Environment variables allow you to configure the filter's behavior without changing the code. They are typically set in a .env file or passed directly to the environment.

Common Variables Used by the Filter

Variable Name	Purpose
`TEMPLATE_PATH`	Path to a local template file (JSON, YAML, or Jinja2)
`TEMPLATE_URL`	URL to fetch the template from (remote JSON/YAML/Jinja2)
`TEMPLATE_INLINE`	Inline template as a JSON string
`TEMPLATE_LOADER`	How to interpret the template: `json`, `jinja`, or `yaml`
`VALIDATION_MODE`	Validation strategy: `schema`, `whitelist`, or `none`
`VALIDATION_SCHEMA`	Path, URL, or inline JSON for a JSON Schema
`VALIDATION_WHITELIST`	Path, URL, or inline JSON for a whitelist
`VALIDATION_STRICT`	If `true`, fail on validation errors; if `false`, skip invalid frames
`OUTPUT_KEY`	Key under which to store the output in the frame (default: `json`)
`DEBUG`	Enable debug logging (`true` or `false`)

How do these variables reflect in `make run`?

When you run make run, the Makefile (or Docker Compose) will start the filter and pass these environment variables to the running process.

If you have a .env file, these variables are automatically loaded and available to the process.
If you set variables in your shell before running make run, those will override the .env values.

Example .env file:

TEMPLATE_PATH=templates/template.yaml
TEMPLATE_LOADER=yaml
VALIDATION_MODE=whitelist
VALIDATION_WHITELIST=templates/whitelist.json
OUTPUT_KEY=json
DEBUG=true

How it works:

When you run make run, the filter will use the template at templates/template.yaml, interpret it as YAML, validate using the whitelist at templates/whitelist.json, and output the result under the json key.

How to configure the filter for different scenarios

You can easily switch configurations by changing the values in your .env file or by exporting variables in your shell.

Scenario 1: Use a Jinja2 template from a URL

TEMPLATE_URL=https://example.com/template.j2
TEMPLATE_LOADER=jinja
VALIDATION_MODE=none
OUTPUT_KEY=output

Example:

Input Data:

{
  'meta': {
    'id': 240,
    'src': 'vid.mp4',
    'src_fps': 29.80,
    'ts': 1746.172
  }
}

Jinja Template:

{
    "user": {
        "meta_id": {{ meta.id }},
        "meta_ts": {{ meta.ts }}
    },
    "src": "{{ meta.src }}",
    "src_fps": {{ meta.src_fps }}
}

Filter Output:

{
  "user": {
    "meta_id": 240,
    "meta_ts": 1746.172
  },
  "src": "vid.mp4",
  "src_fps": 29.80
}

Scenario 2: Use an inline JSON template and strict schema validation

TEMPLATE_INLINE="{\"user\":{\"id\":0,\"src\":\"\"}}"
TEMPLATE_LOADER=json
VALIDATION_MODE=schema
VALIDATION_SCHEMA=templates/schema.json
VALIDATION_STRICT=true
OUTPUT_KEY=outjson

Example:

Filter Input:

{
  'meta': {
    'id': 240,
    'src': 'vid.mp4',
    'src_fps': 29.80,
    'ts': 1746.172
  }
}

Draft7 Validator Json Schema:

{
  "type": "object",
  "required": ["user"],
  "properties": {
    "user": {
      "type": "object",
      "required": ["id", "src"],
      "properties": {
        "id": { "type": "integer" },
        "src": { "type": "string" }
      }
    }
  }
}

Filter Output:

{
  "outjson": {
    "user": {
      "id": 240,
      "src": "vid.mp4"
    }
  }
}

Scenario 3: Use a YAML template and whitelist validation

TEMPLATE_PATH=templates/template.yaml
TEMPLATE_LOADER=yaml
VALIDATION_MODE=whitelist
VALIDATION_WHITELIST=templates/whitelist.json

Example:

Input Data:

{
  'meta': {
    'id': 240,
    'src': 'vid.mp4',
    'src_fps': 29.80,
    'ts': 1746.172
  }
}

Yaml Template:

src: ""
src_fps: 0.0
user:
  id: 0
  ts: 0.0

Validation Whitelist Json:

{
  "user": {
    "id": "int",
    "ts": "float"
  },
  "src": "str",
  "src_fps": "float"
}

Filter Output:

{
  "user": {
    "id": 240,
    "ts": 1746.172
  },
  "src": "vid.mp4",
  "src_fps": 29.80
}

Scenario 4: Use a YAML template and embedded validation params

TEMPLATE_PATH=templates/template.yaml
TEMPLATE_LOADER=yaml

Example:

Path to Validation Schema:

src: ""
src_fps: 0.0
user:
  id: 0
  ts: 0.0
_validation:
  mode: schema
  strict: true
  schema: 'schema_path.json'

Whitelist validation as flattened yaml:

src: ""
src_fps: 0.0
user:
  id: 0
  ts: 0.0
_validation:
  mode: whitelist
  strict: true
  whitelist:
    user.id: int
    src: str

Explanation:
This setup loads a template from a local YAML file and interprets it as YAML. The output is validated using a whitelist (type-based validation) defined in templates/whitelist.json. If the output does not match the whitelist, the frame will be skipped or an error will be raised depending on the VALIDATION_STRICT setting.

How does the filter use these variables?

When the filter starts, it reads these environment variables and uses them to:

Load the correct template (from file, URL, or inline)
Choose the template loader (JSON, YAML, or Jinja2)
Set up validation (schema, whitelist or none)
Decide how to handle validation errors (Strict or Not)
Set the output key

This makes the filter highly configurable and adaptable to different data formats and validation requirements.

Summary:

Set environment variables in .env or your shell.
make run picks up these variables and configures the filter accordingly.
You can easily switch between different configurations by changing the environment variables.

Validation

Schema Validation: Uses jsonschema's Draft7Validator.
Whitelist Validation: Uses a custom WhitelistValidator class.

Template Example

JSON Template

{
  "user_profile": {
    "name": "",
    "email": "",
    "age": 0
  }
}

Jinja2 Template

{
  "user_profile": {
    "name": "{{ user.name }}",
    "email": "{{ user.email }}",
    "age": {{ user.age }}
  }
}

Validation Example

JSON Schema

{
  "type": "object",
  "properties": {
    "user_profile": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "email": {"type": "string"},
        "age": {"type": "integer"}
      },
      "required": ["name", "email", "age"]
    }
  }
}

Whitelist

{
  "user_profile": {
    "name": "str",
    "email": "str",
    "age": "int"
  }
}

Filter Flow Diagram​

Features​

Directory Structure​

Environment Variables & Configuration​

What are the environment variables for?​

Common Variables Used by the Filter​

How do these variables reflect in make run?​

How to configure the filter for different scenarios​

Scenario 1: Use a Jinja2 template from a URL​

Scenario 2: Use an inline JSON template and strict schema validation​

Scenario 3: Use a YAML template and whitelist validation​

Scenario 4: Use a YAML template and embedded validation params​

How does the filter use these variables?​

Validation​

Template Example​

JSON Template​

Jinja2 Template​

Validation Example​

JSON Schema​

Whitelist​

Filter Flow Diagram

Features

Directory Structure

Environment Variables & Configuration

What are the environment variables for?

Common Variables Used by the Filter

How do these variables reflect in `make run`?

How to configure the filter for different scenarios

Scenario 1: Use a Jinja2 template from a URL

Scenario 2: Use an inline JSON template and strict schema validation

Scenario 3: Use a YAML template and whitelist validation

Scenario 4: Use a YAML template and embedded validation params

How does the filter use these variables?

Validation

Template Example

JSON Template

Jinja2 Template

Validation Example

JSON Schema

Whitelist