Key Features

WitEngine provides a comprehensive set of features designed to make distributed computing accessible without sacrificing power or flexibility. This section gives an overview of the main capabilities — detailed explanations can be found in the referenced chapters.

Transparent Distribution

The core value proposition of WitEngine is making distributed computing feel like local computing. You write straightforward scripts, and the engine handles all distribution complexity.

~ This looks like a simple loop... ~
ImageCollection:processed = Grid.ForEach(img in images) 
    => ImageProcessor.Enhance(img);

~ ...but it actually:
   • Finds all available nodes
   • Checks which nodes can run ImageProcessor
   • Measures each node's performance
   • Divides images proportionally
   • Sends data to nodes in parallel
   • Collects and orders results
   • Handles any failures ~

The developer writes what should happen. WitEngine figures out how to distribute it efficiently.

Cross-Platform Support

WitEngine runs on diverse hardware and operating systems, enabling truly heterogeneous computing clusters.

WitEngine embraces device diversity. A single job can utilize servers, workstations, laptops, and mobile devices — each receiving work proportional to its capabilities. The architecture is designed to expand to new platforms (iOS, IoT, game consoles) as the ecosystem grows.

Plugin Architecture

WitEngine is built on a modular plugin system. All functionality — including built-in features — comes from Controllers (plugins).

What Controllers Provide

Each controller registers:

• Variables — Data types usable in scripts (e.g., Int, Matrix, Image)
• Activities — Operations on data (e.g., Int.Add, Matrix.Multiply, Image.Resize)

Built-in Controllers

Extensibility

Adding new capabilities is straightforward. Create a controller class, register your types and operations, and they're immediately available in scripts:

~ After registering an ImageProcessing controller: ~
Image:photo = Image.Load("/path/to/photo.jpg");
Image:resized = Image.Resize(photo, 1920, 1080);
Image:enhanced = Image.Enhance(resized, "auto");

Intelligent Task Allocation

As described in 1.1 What Is WitEngine, WitEngine uses pre-calculated benchmarks to distribute work intelligently. Here's what makes this powerful:

• Per-plugin benchmarks — Each plugin defines its own benchmark, so allocation is accurate for the specific operation
• Idle-time measurement — Nodes run benchmarks when not busy, so no delay when real work arrives
• Proportional batching — Work is divided so all nodes finish simultaneously, maximizing throughput
• Minimal network overhead — Large batches instead of many small requests

The allocation formula is simple: each node receives work proportional to its benchmark score relative to the total.

Two Distribution Strategies

WitEngine offers two strategies optimized for different workload patterns:

Balanced Strategy (Push Model)

Work is divided upfront based on benchmark scores. Each node receives its full batch immediately.

ProcessingOptions:opts = ProcessingOptions.Create("Balanced");
Results:r = Grid.ForEach(item in items, opts) => Process(item);

Best for: Predictable tasks, large batches, stable node performance.

Queued Strategy (Pull Model)

Nodes pull tasks from a shared queue as they complete previous work. Faster nodes automatically do more.

ProcessingOptions:opts = ProcessingOptions.Create("Queued");
Results:r = Grid.ForEach(item in items, opts) => Process(item);

Best for: Variable execution times, unpredictable workloads, fluctuating node availability.

Capability-Based Node Selection

Not every node can run every task. WitEngine automatically filters nodes based on declared requirements.

Declaring Requirements

Activities can specify hardware requirements using attributes:

[RequiresCpu(MinLogicalCores = 8)]
[RequiresMemory(MinRamMb = 16384)]
[RequiresGpu(MinVRamMb = 8192, Features = GpuFeatures.Cuda)]
public class WitActivityTrainModel : WitActivityTransform
{
    // Only runs on nodes meeting ALL requirements
}

Capabilities Tracked

The orchestrator automatically excludes incompatible nodes before allocation begins.

Type-Safe Serialization

Distributed computing requires moving data between machines. WitEngine makes this both safe and fast.

Why It Matters

Automatic Handling

You never manually serialize data. When Grid.ForEach sends work to nodes, WitEngine automatically:

1. Serializes input data using MemoryPack
2. Transmits binary payload to the node
3. Deserializes to original types on the node
4. Serializes results for the return trip
5. Deserializes results back on the orchestrator

Parallel Execution

WitEngine supports parallelism at multiple levels:

~ Local: parallel on one machine ~
Parallel.ForEach(item in items) { Process(item); }

~ Distributed: parallel across machines ~
Grid.ForEach(item in items) => ProcessBatch(item);

~ Combined: distributed batches, each processed in parallel locally ~
Grid.ForEach(batch in batches) => ParallelProcessor.ProcessBatch(batch);

Real-Time Monitoring

Track execution and respond to events as they happen.

var task = engine.ScheduleProcessing(job);

task.ProgressChanged += (s, p) => Console.WriteLine($"{p.Percentage}% complete");
task.TraceReceived += (s, msg) => Console.WriteLine($"[Trace] {msg}");

// Control execution
task.Pause();
task.Resume();
task.Cancel();

Rich Type System

WitEngine provides a comprehensive set of built-in types designed for computational workloads:

• Primitives — Int, Long, Double, Float, Decimal, Bool, String, Byte
• Date/Time — DateTime, DateTimeOffset, TimeSpan
• Collections — Typed collections for every primitive (IntCollection, StringCollection, etc.)
• Linear Algebra — Matrix, Vector, MatrixSparse, VectorSparse
• Utilities — Color, Guid, Tuple, Object, Array

The type system ensures compile-time safety (errors caught during parsing) and efficient serialization (all types are MemoryPack-compatible). Custom types can be added through the plugin system.

How Features Work Together

WitEngine's features are designed to complement each other:

This integration means:

• Plugin Architecture defines what operations exist and their requirements
• Capability Filtering ensures only suitable nodes are considered
• Intelligent Allocation divides work based on measured performance
• Type System + Serialization moves data safely and efficiently
• Cross-Platform Support enables diverse nodes to participate
• Parallel Execution maximizes throughput at every level
• Monitoring provides visibility into the entire process

The result: you write a simple Grid.ForEach, and all these systems work together automatically.

Summary

These features work together to provide a powerful yet accessible distributed computing platform.