LOGAR/docs/ADMIN_GUIDE.md

LOGAR Administrator & Operations Guide

Complete Operations Manual for Fleet Hub Administration, Database Engines, PKI Licensing & AI Provisioning

---

Table of Contents

1. Executive Summary & System Architecture
2. Database Storage Engine Management
   • Local SQLite 3 (WAL Mode)
   • External PostgreSQL Enterprise Backend
   • Automated SQLite to PostgreSQL Migration
3. Telemetry Ingestion Cache & Buffer Tuning
   • Ingestion Modes: Direct Write vs In-Memory Queue Buffer
   • Tuning Parameters & Metrics
   • Manual Flush & Buffer Monitoring
4. Zero-Knowledge OpenPGP Data Encryption at Rest
   • Encryption Mechanism
   • Transparent Authorized Decryption
5. Subordinate CA & Dynamic PKI Licensing
   • Hierarchical PKI & Custom X.509 OIDs
   • Licensed Feature Capabilities (enterprise_db)
   • Importing .lic Activation Packages
   • Automated mTLS License Renewal
6. Fleet Forwarder Management & Seat Provisioning
   • Zero-Touch Linux Deployment
   • Windows Edge Client Deployment
   • Seat Lifecycle: Approve, Revoke, Delete
7. Hermes AI Agent Integration
   • Dedicated Bearer Tokens
   • 12-Hour Temporal Persistence Anomaly Stream
8. Security, Authentication & 2FA
   • Password Policies & PBKDF2 Hashing
   • Time-Based One-Time Password (TOTP 2FA)
   • Sliding Window IP Rate Limiting
9. Configuration Reference & CLI Flags
10. REST API Reference
11. Operational Runbook & Troubleshooting

---

1. Executive Summary & System Architecture

LOGAR is a centralized log aggregation, normalization, and provisioning system designed for enterprise server fleets. It connects heterogeneous edge nodes (Windows and Linux) with a central server hub over Mutual TLS 1.3 (mTLS), normalizing incoming events and presenting structured anomaly intelligence to Hermes AI.

Core Network Architecture

graph TD
    subgraph MasterAuthority [Master License Authority]
        ROOT_CA[Master Root CA]
        AUTH_API[WebSocket Licensing Hub<br/>logar.eibl.tech]
    end

    subgraph Hub [LOGAR Server Hub]
        SUB_CA[Subordinate CA & PKI Engine]
        INGEST_QUEUE[IngestQueueManager<br/>In-Memory RAM Buffer]
        STORAGE_ROUTER[Storage Engine Router]
        SQLITE[(Local SQLite 3 WAL)]
        POSTGRES[(External PostgreSQL Cluster)]
        TEMP_ENG[12h Temporal Persistence Engine]
        HERMES_API[Hermes REST & WebUI API<br/>Port 8443]
        MTLS_SOCKET[mTLS 1.3 Socket Listener<br/>Port 9443]

        MTLS_SOCKET --> INGEST_QUEUE
        INGEST_QUEUE --> STORAGE_ROUTER
        STORAGE_ROUTER --> SQLITE
        STORAGE_ROUTER --> POSTGRES
        STORAGE_ROUTER --> TEMP_ENG
        TEMP_ENG --> HERMES_API
    end

    subgraph EdgeNodes [Fleet Edge Forwarders]
        WIN_NODE[Windows Forwarder<br/>Win_Client.exe]
        LIN_NODE[Linux Forwarder<br/>Linux_Client.bin]
    end

    subgraph AI [Hermes AI Analysis Layer]
        HERMES_AGENT[Hermes AI Agent]
    end

    ROOT_CA -->|Signs Sub-CA Cert| SUB_CA
    AUTH_API <==>|Live mTLS Renewal| SUB_CA

    WIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET
    LIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET

    HERMES_API ==>|GET /api/hermes/report| HERMES_AGENT

---

2. Database Storage Engine Management

LOGAR supports dual storage backends: an embedded Local SQLite 3 (WAL Mode) engine and an External PostgreSQL Database Server for enterprise clustering.

Local SQLite 3 (WAL Mode)

• Status: Standard Default (Included out of the box, zero configuration required).
• Optimization: Operates with Write-Ahead Logging (PRAGMA journal_mode=WAL;), asynchronous normal synchronization (PRAGMA synchronous=NORMAL;), and memory-mapped I/O (PRAGMA mmap_size=268435456;).
• File Location: Configurable via db_path in server_config.json (defaults to logar_state.db).

External PostgreSQL Enterprise Backend

• Requirement: Unlocked via the enterprise_db license capability.
• Benefits: Enables horizontal database scalability, clustering, multi-node replication, and centralized enterprise backup policies.
• Connection Parameters:
  • Host / IP: Database hostname or IP address.
  • Port: Default 5432.
  • Database: Target schema/database name (default logar).
  • Username / Password: Authentication credentials.
  • SSL Mode: Supports prefer, require, verify-ca, verify-full, disable, and allow.
  • Connect Timeout: Timeout in seconds (default 5s).
  • Connection Pool Size: Maximum pooled persistent connections (default 10).

Automated SQLite to PostgreSQL Migration

Administrators can migrate all historical logs, runs, active issues, seat allocations, users, and tokens from SQLite to PostgreSQL directly in the WebUI (💾 DB Engine -> 🚀 Migrate SQLite Data to PostgreSQL) or via API (POST /api/db/migrate-to-postgres):

1. Flushes any in-memory telemetry buffers to SQLite.
2. Connects to PostgreSQL and verifies/initializes the schema.
3. Automatically transfers records table-by-table with validation.
4. Returns a detailed count of all migrated records.

---

3. Telemetry Ingestion Cache & Buffer Tuning

Ingestion Modes: Direct Write vs In-Memory Queue Buffer

Feature	⚡ Immediate Direct Write	🚀 In-Memory Queue Buffer (enterprise_db)
Ingress Latency	Low latency, immediate DB commit	Microsecond ingestion, non-blocking
Throughput	Up to 1,500 events/sec	Up to 50,000+ events/sec
Disk I/O	Continuous write cycles	Atomic batch flushing
License	Standard Community Edition	Enterprise License (enterprise_db)

Tuning Parameters & Metrics

When In-Memory Queue Buffer is enabled, administrators can tune the buffer engine:

• Flush Interval (seconds): Maximum elapsed time before buffered anomalies are written to storage (e.g. 2.0s).
• Batch Size Threshold: Number of buffered events that triggers an immediate atomic flush (e.g. 100 events).
• Max Buffer Capacity: Upper ceiling of buffered items in memory (e.g. 50,000 events) to protect system RAM under heavy load.

Manual Flush & Buffer Monitoring

• Live Monitoring: The KPI card displays live buffer size, total events ingested, total flushed, and average flush latency.
• Manual Trigger: Click 🌊 Flush Buffer Now or call POST /api/db/flush-buffer to force an immediate buffer drain.

---

4. Zero-Knowledge OpenPGP Data Encryption at Rest

Encryption Mechanism

When Data Encryption at Rest is enabled (encrypt_at_rest: true):

• All sensitive log payloads, anomaly messages, and raw traces are encrypted with the server's OpenPGP public key before writing to the database table (active_issues.message).
• Remote database administrators or third-party database hosting providers only see armored OpenPGP ciphertext (-----BEGIN PGP MESSAGE-----).

Transparent Authorized Decryption

• The central LOGAR hub instance holds the private OpenPGP key in protected memory.
• When authenticated administrators access the WebUI or when authorized Hermes AI instances query /api/hermes/report, the server transparently decrypts records on-the-fly.

---

5. Subordinate CA & Dynamic PKI Licensing

LOGAR utilizes a hierarchical Public Key Infrastructure (PKI) for cryptographic client identity and license seat enforcement.

Hierarchical PKI & Custom X.509 OIDs

The Master Authority (logar.eibl.tech) signs the hub's Subordinate CA certificate (certs/ca.crt), embedding custom ASN.1 OIDs:

• 1.3.6.1.4.1.58888.1.1 (Seat Quota): Encodes the authorized edge client seat capacity (e.g. 25 seats).
• 1.3.6.1.4.1.58888.1.2 (Master Authority URL): Identifies the upstream licensing authority.
• 1.3.6.1.4.1.58888.1.3 (Feature Capabilities): Comma-delimited list of granted features (e.g. enterprise_db,ai_heuristics).

Licensed Feature Capabilities (enterprise_db)

The 📜 Sub-CA & Licensing tab displays all active licensed features:

• enterprise_db: Authorizes external PostgreSQL database pooling, high-throughput in-memory RAM buffering, and OpenPGP data-at-rest encryption.

Importing .lic Activation Packages

1. In the WebUI, navigate to 📜 Sub-CA & Licensing and click 📥 Import License (.lic).
2. Select your .lic bundle or paste the JSON package.
3. The server immediately reloads the PKI trust chain and hot-reloads the active mTLS SSLContext without downtime.

Automated mTLS License Renewal

• Click 🔄 Sync & Renew via mTLS in the WebUI or run:

  python src/Server.py --sync-license
  

• The hub connects to logar.eibl.tech over mTLS, sends its current seat utilization, and retrieves an updated signed CA certificate.

---

6. Fleet Forwarder Management & Seat Provisioning

Zero-Touch Linux Deployment

Run the automated single-line installer on target Linux servers:

sudo ./compilation/install_linux_client.sh "http://<HUB_IP>:8443" "<ENROLLMENT_SECRET>"

• Automatically generates a 2048-bit RSA client keypair.
• Enrolls with the hub and receives a signed client certificate.
• Deploys and starts the logar-client.service systemd unit.

Windows Edge Client Deployment

1. Download and run LOGAR-Client-Setup.exe on the Windows host.
2. In the Server WebUI, click 📥 Export Client Profile and download client_config.json.
3. Place client_config.json into C:\Program Files\LOGAR\client_config.json.
4. The LOGAR_Client service automatically boots and connects via mTLS.

Seat Lifecycle: Approve, Revoke, Delete

In the 💻 Fleet Forwarders tab:

• Revoke: Temporarily blocks the node at the TLS handshake level (ssl.CERT_REQUIRED).
• Approve: Re-activates a revoked node and restores telemetry ingestion.
• Remove: Permanently deletes the client record and immediately frees the leased seat quota.

---

7. Hermes AI Agent Integration

Dedicated Bearer Tokens

1. Navigate to 🤖 Hermes AI & Tokens in the WebUI.
2. Enter a descriptive token name (e.g. Hermes-Autonomous-SecOps) and click Generate Token.
3. Copy the Bearer token for Hermes agent configuration.

12-Hour Temporal Persistence Anomaly Stream

Hermes queries the real-time anomaly stream:

curl -H "Authorization: Bearer hermes_xxxxxxxxxxxx" http://<HUB_IP>:8443/api/hermes/report

JSON Telemetry Format:

[
  {
    "fingerprint": "corp-dc:srv-db-01:postgres_wal_checkpoint_delay",
    "site": "corp-dc",
    "server": "srv-db-01",
    "signature": "postgres_wal_checkpoint_delay",
    "severity": "WARNING",
    "message": "WAL checkpoint write duration exceeded 30000ms threshold",
    "os_type": "linux",
    "first_seen": "2026-09-30 14:10:00",
    "last_seen": "2026-09-30 17:42:15",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

---

8. Security, Authentication & 2FA

Password Policies & PBKDF2 Hashing

• User passwords are protected using PBKDF2-HMAC-SHA256 with 100,000 iterations and 16-byte random salts.
• Change admin credentials via the user avatar menu -> 🔑 Change Password.

Time-Based One-Time Password (TOTP 2FA)

• Standard RFC 6238 TOTP with standard 30-second drift windows.
• Configure via avatar menu -> 🛡ï¸� Two-Factor Auth (2FA).
• Compatible with Google Authenticator, Authy, 1Password, and Bitwarden.

Sliding Window IP Rate Limiting

• Brute-Force Protection: Enforces rate limiting on /login and /api/client/enroll.
• Excess failed attempts trigger automatic IP blocking with sliding exponential backoff.

---

9. Configuration Reference & CLI Flags

server_config.json Specification

{
  "server_name": "LOGAR-Cloud-Hub",
  "tcp_host": "0.0.0.0",
  "tcp_port": 9443,
  "hermes_host": "0.0.0.0",
  "hermes_port": 8443,
  "admin_user": "admin",
  "enrollment_secret": "9a7f8e...4b",
  "max_seats": 25,
  "license_status": "active",
  "license_server_url": "https://logar.eibl.tech",
  "cert_dir": "certs",
  "tls_enabled": true,
  "db_path": "logar_state.db",
  "storage_engine": "sqlite",
  "cache_mode": "direct",
  "flush_interval": 2.0,
  "batch_size": 100,
  "max_queue": 50000,
  "encrypt_at_rest": false,
  "server_fingerprint": "6FF5BE81...",
  "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
  "private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
}

CLI Arguments

Flag	Description
--config <path>	Path to server configuration JSON file
--create-client-config	Exports a client config bundle and exits
--client-out <path>	Destination path for exported client configuration
--server-host <host>	Embedded target host for exported client configuration
--server-port <port>	Embedded target port for exported client configuration
--export-lic <path>	Exports a .lic CSR activation request package
--import-lic <path>	Imports and activates a signed .lic license package
--sync-license	Connects to master authority to renew certificate & seats

---

10. REST API Reference

Method	Endpoint	Auth	Description
GET	/health	None	Health check, server name, and TLS status
POST	/api/client/enroll	Secret	Edge forwarder mTLS enrollment & certificate issuance
GET	/api/clients	Admin	List registered forwarders, activity times, and seat quota
POST	/api/clients/approve	Admin	Reactivate / approve an enrolled forwarder seat
POST	/api/clients/revoke	Admin	Revoke an enrolled forwarder seat
POST	/api/clients/delete	Admin	Remove forwarder record and permanently free seat
GET	/api/hermes/report	Hermes/Admin	Stream verified anomalies within 12h temporal window
POST	/api/hermes/tokens/create	Admin	Generate a named Bearer token for Hermes AI
POST	/api/hermes/tokens/revoke	Admin	Revoke an existing Hermes Bearer token
GET	/api/db/stats	Admin	Telemetry buffer stats, queue metrics, and storage info
POST	/api/db/flush-buffer	Admin	Force immediate drain and write of buffered telemetry
POST	/api/db/test-connection	Admin	Test external PostgreSQL connectivity (enterprise_db)
POST	/api/db/switch-backend	Admin	Switch backend between SQLite and PostgreSQL (enterprise_db)
POST	/api/db/config	Admin	Persist DB engine, cache mode, and encryption settings
POST	/api/db/migrate-to-postgres	Admin	Migrate SQLite tables to PostgreSQL (enterprise_db)
GET	/api/license/status	Admin	Subordinate CA status, seat quota, and licensed features
POST	/api/license/import	Admin	Import .lic bundle or signed CA PEM
POST	/api/license/sync	Admin	Sync license seats with master authority over mTLS

---

11. Operational Runbook & Troubleshooting

Diagnostic Checklist

1. Verify Hub Service Status:

   # Linux
   sudo systemctl status logar-server.service
   # Windows PowerShell
   Get-Service LOGAR_Server
   

2. Check Health Endpoint:

   curl http://localhost:8443/health
   

3. Inspect Active Anomaly Telemetry:

   curl -H "Authorization: Bearer <ADMIN_OR_HERMES_TOKEN>" http://localhost:8443/api/hermes/report
   

4. Certificate Validity Check:

   openssl x509 -in certs/ca.crt -text -noout | grep -A 2 "Validity"
   

Common Issues & Remediation

• Issue: Forwarder rejected with Connection Refused or TLS Handshake Failure:
  • Ensure port 9443 is reachable through host and cloud firewalls.
  • Verify edge node clock synchronization (NTP) with the server hub.
  • Confirm the client certificate has not been revoked in the Fleet Forwarders tab.
• Issue: 403 Forbidden on PostgreSQL or Buffering settings:
  • The enterprise_db feature capability is required. Verify your license in ��“œ Sub-CA & Licensing.
• Issue: Rate limit lock on admin login:
  • Wait 60 seconds for the sliding attempt window to expire, or authenticate via an existing active session.