Container Orchestration

This guide explains the multi-tenant container orchestration system in the De.IoTB platform.

Overview

Container Orchestration is a core architectural feature of De.IoTB that enables efficient multi-tenant isolation while optimizing resource utilization. The system manages the deployment, scaling, and lifecycle of containerized IoT workspaces.

Architecture Principles

De.IoTB's container orchestration follows these key principles:

Workspace Isolation - Each tenant operates in its own logical container
Resource Efficiency - Shared resources reduce overhead for smaller workloads
Scalability - Containers can be dynamically scaled based on workload
Fault Tolerance - Container failures don't affect other tenants
Performance Predictability - Dedicated resources for performance-critical workloads

Container System Architecture

┌───────────────────────────────────────────────────────────────────────────┐
│                            Orchestrator (ORC)                             │
└───────────────────────────────────────┬───────────────────────────────────┘
                                        │
                                        ▼
┌───────────────────────────────────────────────────────────────────────────┐
│                       Managed Container System (MCS)                      │
├───────────────────────────────────────────────────────────────────────────┤
│                                                                           │
│  ┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐│
│  │  Shared Container   │  │  Tenant Container 1 │  │  Tenant Container 2 ││
│  │                     │  │                     │  │                     ││
│  │  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  ││
│  │  │   Channel     │  │  │  │   Channel     │  │  │  │   Channel     │  ││
│  │  │   (MQTT)      │  │  │  │   (MQTT)      │  │  │  │   (MQTT)      │  ││
│  │  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  ││
│  │                     │  │                     │  │                     ││
│  │  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  ││
│  │  │   Database    │  │  │  │   Database    │  │  │  │   Database    │  ││
│  │  │   (MongoDB)   │  │  │  │   (MongoDB)   │  │  │  │   (MongoDB)   │  ││
│  │  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  ││
│  │                     │  │                     │  │                     ││
│  │  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  ││
│  │  │ File Storage  │  │  │  │ File Storage  │  │  │  │ File Storage  │  ││
│  │  │    (S3)       │  │  │  │    (S3)       │  │  │  │    (S3)       │  ││
│  │  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  ││
│  │                     │  │                     │  │                     ││
│  │  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  ││
│  │  │Active Rules   │  │  │  │Active Rules   │  │  │  │Active Rules   │  ││
│  │  │Engine (AR)    │  │  │  │Engine (AR)    │  │  │  │Engine (AR)    │  ││
│  │  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  ││
│  │                     │  │                     │  │                     ││
│  │  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  ││
│  │  │Event Manager  │  │  │  │Event Manager  │  │  │  │Event Manager  │  ││
│  │  │(EM)           │  │  │  │(EM)           │  │  │  │(EM)           │  ││
│  │  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  ││
│  │                     │  │                     │  │                     ││
│  └─────────────────────┘  └─────────────────────┘  └─────────────────────┘│
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘

Container Types

De.IoTB supports two container deployment modes to balance cost and performance:

Shared Containers

Shared containers allow multiple tenants to use the same resources, making them cost-effective for smaller deployments and development environments.

Features:

Common MongoDB database (with namespace isolation)
Shared MQTT broker (with topic-based isolation)
Pooled connection management
Lower operational cost
Suitable for small to medium workloads

Resource Management:

Connection pooling with reference counting
Namespace-based data isolation
Rate limiting to prevent resource contention

Standalone Containers

Standalone containers provide dedicated resources for each tenant, offering better performance, predictability, and isolation.

Features:

Dedicated MongoDB instance
Private MQTT connections
Isolated file storage
Higher performance
Required for regulated industries or high-security applications
Better scalability for large workloads

Resource Management:

Dedicated database connections
Tenant-specific MQTT client instances
Custom resource scaling

Container Components

Each container (shared or standalone) includes several components:

Channel Manager (CH)

Handles MQTT connectivity for device communication:

Manages broker connection with auto-reconnect
Implements topic subscription and message routing
Provides secure publishing capabilities
Handles connection failures and recovery

Database Manager (DB)

Manages MongoDB connectivity for data storage:

Connection pooling with automatic reconnection
Query abstraction layer
Health monitoring
Collection management

File Storage Manager (FS)

Provides S3-compatible object storage:

Secure file upload/download
Multi-region support
Security validations (path traversal prevention, extension validation)
Streaming support for large files

Active Rules Engine (AR)

Processes device messages according to defined rules:

Schema-driven message parsing
LRU caching for performance
Type transformations
Event triggering

Event Manager (EM)

Manages WebSocket connections for real-time applications:

Socket.IO namespace control
Authentication and authorization
Rate limiting
Topic subscription management

Container Lifecycle

Containers follow a well-defined lifecycle:

1. Creation & Deployment

javascript

// Container creation process
const container = new Container(connector)
await container.deploy()
await container.manage()

The deployment process includes:

Setting container status to DEPLOYING
Establishing database connection
Setting up file storage client
Connecting to MQTT broker
Updating container status to ACTIVE

2. Management

Once deployed, the container initializes management components:

Query Models (QM) - Database operations
Active Rules (AR) - Message parsing
Error Handler (EH) - Error management
Event Manager (EM) - Socket.IO control

3. Monitoring

Containers are continuously monitored:

javascript

// Health check example
const healthy = await container.isHealthy()
const stats = container.getStats()

Health metrics include:

Connection status for all resources
Error rates and last errors
Resource utilization
Container uptime

4. Scaling

Containers can scale resources based on workload:

javascript

// Resource scaling
await container.setDatabase({
  poolSize: 20,
  maxSize: 2048  // MB
})

5. Teardown

When no longer needed, containers are gracefully removed:

javascript

// Container teardown
await container.drop()

The teardown process includes:

Stopping Event Manager (disconnecting all sockets)
Clearing Active Rules cache
Disconnecting Channel with reference counting
Disconnecting Database with reference counting
Cleaning up File Storage client
Removing container from global registry

Container Configuration

Containers are configured through the connector definition:

javascript

// Example connector configuration
const connector = {
  id: "mxbh48wy",           // Unique connector ID
  workspace: {
    type: "developer",      // Workspace type
    icode: "DEV-9X00-233"   // Internal code
  },
  type: "IoT",              // Connector type
  name: "Fleet Management", // Descriptive name
  access: {
    secret: "sk_live_a3f8e9d2c1b4567890abcdef",  // Auth token
    maintainers: ["[email protected]"],           // Admin emails
    ratelimits: {
      sockets: 500,         // Max socket connections
      events: 100           // Events per second
    }
  },
  config: {
    protocol: "sio",        // Socket.IO protocol
    resources: {
      type: "standalone",   // Deployment type (shared or standalone)
      specs: {
        database: {
          url: "mongodb://custom:27017/tenant-db",
          size: 1024        // MB
        },
        ratelimit: {
          channels: 50,     // MQTT topic subscriptions
          events: 100,      // Events per second
          sockets: 500      // WebSocket connections
        }
      }
    }
  },
  enabled: true
}

Resource Management

The orchestration system manages resources efficiently with:

Reference Counting

Shared resources use reference counting to track usage:

javascript

// Example of reference counting
if (SHARED_RESOURCES.DB) {
  this.db = SHARED_RESOURCES.DB.instance
  SHARED_RESOURCES.DB.refCount++
} 
else {
  this.db = new Database('shared', config)
  SHARED_RESOURCES.DB = { instance: this.db, refCount: 1 }
}

This ensures resources are only released when no containers are using them.

Connection Pooling

Database connections are pooled for efficiency:

javascript

// Database connection pooling
const dbConfig = {
  maxPoolSize: 10,
  serverSelectionTimeoutMS: 5000,
  socketTimeoutMS: 45000
}

Rate Limiting

Prevents resource abuse through configurable rate limits:

javascript

// Rate limit configuration
const ratelimits = {
  sockets: 500,    // Max concurrent connections
  events: 100,     // Events per second per socket
  channels: 50     // MQTT channel subscriptions
}

Orchestrator

The Orchestrator (ORC) manages the container system, handling:

Job Scheduling - Periodic container management operations
Effect System - Reactive patterns for container state changes
Monitoring - Container health and performance tracking
Scaling - Dynamic container resource adjustment

Effect System Example

javascript

// Register effect listener
orchestrator.effect.register(async (data, dispose) => {
  // React to system changes
  if (data.connectorUpdated) {
    // Update container resources
    await updateContainerResources(data.connector)
  }
  
  // Self-unregister if needed
  if (shouldDispose) {
    dispose()
  }
}, 'Resource Manager', 'container_id')

Health Monitoring

De.IoTB includes comprehensive health monitoring for containers:

javascript

// Container health check
async function checkContainerHealth(containerId) {
  const container = MCS[namespace][containerId]
  
  if (!container) {
    return { status: 'not_found' }
  }
  
  const healthy = await container.isHealthy()
  const stats = container.getStats()
  
  return {
    status: healthy ? 'healthy' : 'unhealthy',
    uptime: stats.uptime,
    resources: stats.resourceUsage,
    lastError: stats.lastError
  }
}

The system tracks multiple health indicators:

Container status (ACTIVE, IDLE, ERROR, etc.)
Resource connectivity status
Error rates and messages
Performance metrics

Resource Scaling

Containers can be scaled based on workload:

javascript

// Auto-scaling configuration
await access.request({
  url: '/workspace/config/auto-scaling',
  method: 'PATCH',
  body: {
    enabled: true,
    triggers: {
      cpuUsage: 80,        // Scale at 80% CPU
      memoryUsage: 70,     // Scale at 70% memory
      connectionCount: 400, // Scale at 400 connections
      messagesPerSecond: 5000 // Scale at 5000 msg/s
    },
    cooldown: 300,         // 5 minute cooldown
    maxInstances: 5        // Maximum instances
  }
})

Security Considerations

The container system includes several security measures:

Namespace Isolation - Prevents cross-tenant data access
Authentication - Token-based access control
Resource Limits - Prevents resource exhaustion attacks
Rate Limiting - Throttles excessive connection attempts
IP Banning - Progressive ban system for abusive clients
Topic Restrictions - Limits devices to authorized topics only

Best Practices

Container Type Selection

Guidelines for choosing the appropriate container type:

Consideration	Shared	Standalone
Development environment	✅	❌
Testing/QA	✅	❌
Production (small-scale)	✅	❌
Production (large-scale)	❌	✅
Regulatory compliance	❌	✅
High performance needs	❌	✅
Cost-sensitive	✅	❌

Resource Planning

For standalone containers, calculate resource requirements:

Database: 20MB base + (5KB × # of devices) + (1KB × daily messages)
Memory: 128MB base + (512KB × # of concurrent connections)
MQTT Connections: 1 connection per 1000 devices (with proper connection sharing)

Monitoring Recommendations

Set up alerts for container health status changes
Monitor resource usage trends over time
Track error rates and message throughput
Set reasonable auto-scaling thresholds

API Reference

De.IoTB provides API endpoints for container management:

javascript

// Get container status
const containerStatus = await access.request({
  url: `/containers/${containerId}/status`,
  method: 'GET'
})

// Update container resources
await access.request({
  url: `/containers/${containerId}/resources`,
  method: 'PATCH',
  body: {
    database: {
      poolSize: 20
    },
    ratelimits: {
      events: 200,
      sockets: 800
    }
  }
})

// Restart container
await access.request({
  url: `/containers/${containerId}/restart`,
  method: 'POST'
})

// Get container metrics
const metrics = await access.request({
  url: `/containers/${containerId}/metrics`,
  method: 'GET',
  query: {
    period: '24h'  // '1h', '24h', '7d', '30d'
  }
})

System Requirements

Minimum Requirements

CPU: 2 cores
RAM: 4GB
Storage: 20GB

Production Recommendations

Scale	CPU	RAM	Storage	Network
Small	4 cores	8 GB	100 GB	1 Gbps
Medium	8 cores	16 GB	250 GB	1 Gbps
Large	16+ cores	32+ GB	500+ GB	10 Gbps

Troubleshooting

Common Issues

Container Deployment Failures

ERROR: Database connection failed

Possible causes:

Invalid MongoDB connection URI
MongoDB instance not reachable
Authentication failure

Resolution:

Check database connection parameters
Verify network connectivity
Check database user credentials

Resource Allocation Failures

ERROR: Insufficient resources to allocate container

Possible causes:

System at capacity
Resource limits reached
Configuration error

Resolution:

Scale up system resources
Review container resource allocations
Check for resource leaks

MQTT Connection Issues

ERROR: MQTT connection failed: CONNACK 5

Possible causes:

MQTT broker unreachable
Authentication failure
Connection limits reached

Resolution:

Check broker connectivity
Verify credentials
Review connection limits

Diagnostic Commands

javascript

// Get container diagnostics
const diagnostics = await access.request({
  url: `/containers/${containerId}/diagnostics`,
  method: 'GET'
})

// Check resource connections
const connections = await access.request({
  url: `/containers/${containerId}/connections`,
  method: 'GET'
})

// Generate diagnostic report
const report = await access.request({
  url: `/containers/${containerId}/report`,
  method: 'POST'
})

Container Orchestration ​

Overview ​

Architecture Principles ​

Container System Architecture ​

Container Types ​

Shared Containers ​

Standalone Containers ​

Container Components ​

Channel Manager (CH) ​

Database Manager (DB) ​

File Storage Manager (FS) ​

Active Rules Engine (AR) ​

Event Manager (EM) ​

Container Lifecycle ​

1. Creation & Deployment ​

2. Management ​

3. Monitoring ​

4. Scaling ​

5. Teardown ​

Container Configuration ​

Resource Management ​

Reference Counting ​

Connection Pooling ​

Rate Limiting ​

Orchestrator ​

Effect System Example ​

Health Monitoring ​

Resource Scaling ​

Security Considerations ​

Best Practices ​

Container Type Selection ​

Resource Planning ​

Monitoring Recommendations ​

API Reference ​

System Requirements ​

Minimum Requirements ​

Production Recommendations ​

Troubleshooting ​

Common Issues ​

Container Deployment Failures ​

Resource Allocation Failures ​

MQTT Connection Issues ​

Diagnostic Commands ​

Next Steps ​

Container Orchestration

Overview

Architecture Principles

Container System Architecture

Container Types

Shared Containers

Standalone Containers

Container Components

Channel Manager (CH)

Database Manager (DB)

File Storage Manager (FS)

Active Rules Engine (AR)

Event Manager (EM)

Container Lifecycle

1. Creation & Deployment

2. Management

3. Monitoring

4. Scaling

5. Teardown

Container Configuration

Resource Management

Reference Counting

Connection Pooling

Rate Limiting

Orchestrator

Effect System Example

Health Monitoring

Resource Scaling

Security Considerations

Best Practices

Container Type Selection

Resource Planning

Monitoring Recommendations

API Reference

System Requirements

Minimum Requirements

Production Recommendations

Troubleshooting

Common Issues

Container Deployment Failures

Resource Allocation Failures

MQTT Connection Issues

Diagnostic Commands

Next Steps