# LOGAR Administrator & Operations Guide
### *Complete Operations Manual for Fleet Hub Administration, Database Engines, PKI Licensing & AI Provisioning*

---

## Table of Contents

1. [Executive Summary & System Architecture](#1-executive-summary--system-architecture)
2. [Database Storage Engine Management](#2-database-storage-engine-management)
   - [Local SQLite 3 (WAL Mode)](#local-sqlite-3-wal-mode)
   - [External PostgreSQL Enterprise Backend](#external-postgresql-enterprise-backend)
   - [Automated SQLite to PostgreSQL Migration](#automated-sqlite-to-postgresql-migration)
3. [Telemetry Ingestion Cache & Buffer Tuning](#3-telemetry-ingestion-cache--buffer-tuning)
   - [Ingestion Modes: Direct Write vs In-Memory Queue Buffer](#ingestion-modes-direct-write-vs-in-memory-queue-buffer)
   - [Tuning Parameters & Metrics](#tuning-parameters--metrics)
   - [Manual Flush & Buffer Monitoring](#manual-flush--buffer-monitoring)
4. [Zero-Knowledge OpenPGP Data Encryption at Rest](#4-zero-knowledge-openpgp-data-encryption-at-rest)
   - [Encryption Mechanism](#encryption-mechanism)
   - [Transparent Authorized Decryption](#transparent-authorized-decryption)
5. [Subordinate CA & Dynamic PKI Licensing](#5-subordinate-ca--dynamic-pki-licensing)
   - [Hierarchical PKI & Custom X.509 OIDs](#hierarchical-pki--custom-x509-oids)
   - [Licensed Feature Capabilities (`enterprise_db`)](#licensed-feature-capabilities-enterprise_db)
   - [Importing `.lic` Activation Packages](#importing-lic-activation-packages)
   - [Automated mTLS License Renewal](#automated-mtls-license-renewal)
6. [Fleet Forwarder Management & Seat Provisioning](#6-fleet-forwarder-management--seat-provisioning)
   - [Zero-Touch Linux Deployment](#zero-touch-linux-deployment)
   - [Windows Edge Client Deployment](#windows-edge-client-deployment)
   - [Seat Lifecycle: Approve, Revoke, Delete](#seat-lifecycle-approve-revoke-delete)
7. [Hermes AI Agent Integration](#7-hermes-ai-agent-integration)
   - [Dedicated Bearer Tokens](#dedicated-bearer-tokens)
   - [12-Hour Temporal Persistence Anomaly Stream](#12-hour-temporal-persistence-anomaly-stream)
8. [Security, Authentication & 2FA](#8-security-authentication--2fa)
   - [Password Policies & PBKDF2 Hashing](#password-policies--pbkdf2-hashing)
   - [Time-Based One-Time Password (TOTP 2FA)](#time-based-one-time-password-totp-2fa)
   - [Sliding Window IP Rate Limiting](#sliding-window-ip-rate-limiting)
9. [Configuration Reference & CLI Flags](#9-configuration-reference--cli-flags)
10. [REST API Reference](#10-rest-api-reference)
11. [Operational Runbook & Troubleshooting](#11-operational-runbook--troubleshooting)

---

## 1. Executive Summary & System Architecture

LOGAR is a centralized log aggregation, normalization, and provisioning system designed for enterprise server fleets. It connects heterogeneous edge nodes (Windows and Linux) with a central server hub over **Mutual TLS 1.3 (mTLS)**, normalizing incoming events and presenting structured anomaly intelligence to **Hermes AI**.

### Core Network Architecture

```mermaid
graph TD
    subgraph MasterAuthority [Master License Authority]
        ROOT_CA[Master Root CA]
        AUTH_API[WebSocket Licensing Hub<br/>logar.eibl.tech]
    end

    subgraph Hub [LOGAR Server Hub]
        SUB_CA[Subordinate CA & PKI Engine]
        INGEST_QUEUE[IngestQueueManager<br/>In-Memory RAM Buffer]
        STORAGE_ROUTER[Storage Engine Router]
        SQLITE[(Local SQLite 3 WAL)]
        POSTGRES[(External PostgreSQL Cluster)]
        TEMP_ENG[12h Temporal Persistence Engine]
        HERMES_API[Hermes REST & WebUI API<br/>Port 8443]
        MTLS_SOCKET[mTLS 1.3 Socket Listener<br/>Port 9443]

        MTLS_SOCKET --> INGEST_QUEUE
        INGEST_QUEUE --> STORAGE_ROUTER
        STORAGE_ROUTER --> SQLITE
        STORAGE_ROUTER --> POSTGRES
        STORAGE_ROUTER --> TEMP_ENG
        TEMP_ENG --> HERMES_API
    end

    subgraph EdgeNodes [Fleet Edge Forwarders]
        WIN_NODE[Windows Forwarder<br/>Win_Client.exe]
        LIN_NODE[Linux Forwarder<br/>Linux_Client.bin]
    end

    subgraph AI [Hermes AI Analysis Layer]
        HERMES_AGENT[Hermes AI Agent]
    end

    ROOT_CA -->|Signs Sub-CA Cert| SUB_CA
    AUTH_API <==>|Live mTLS Renewal| SUB_CA

    WIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET
    LIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET

    HERMES_API ==>|GET /api/hermes/report| HERMES_AGENT
```

---

## 2. Database Storage Engine Management

LOGAR supports dual storage backends: an embedded **Local SQLite 3 (WAL Mode)** engine and an **External PostgreSQL Database Server** for enterprise clustering.

### Local SQLite 3 (WAL Mode)
- **Status**: Standard Default (Included out of the box, zero configuration required).
- **Optimization**: Operates with Write-Ahead Logging (`PRAGMA journal_mode=WAL;`), asynchronous normal synchronization (`PRAGMA synchronous=NORMAL;`), and memory-mapped I/O (`PRAGMA mmap_size=268435456;`).
- **File Location**: Configurable via `db_path` in `server_config.json` (defaults to `logar_state.db`).

### External PostgreSQL Enterprise Backend
- **Requirement**: Unlocked via the `enterprise_db` license capability.
- **Benefits**: Enables horizontal database scalability, clustering, multi-node replication, and centralized enterprise backup policies.
- **Connection Parameters**:
  - **Host / IP**: Database hostname or IP address.
  - **Port**: Default `5432`.
  - **Database**: Target schema/database name (default `logar`).
  - **Username / Password**: Authentication credentials.
  - **SSL Mode**: Supports `prefer`, `require`, `verify-ca`, `verify-full`, `disable`, and `allow`.
  - **Connect Timeout**: Timeout in seconds (default `5`s).
  - **Connection Pool Size**: Maximum pooled persistent connections (default `10`).

### Automated SQLite to PostgreSQL Migration
Administrators can migrate all historical logs, runs, active issues, seat allocations, users, and tokens from SQLite to PostgreSQL directly in the WebUI (`💾 DB Engine` -> `🚀 Migrate SQLite Data to PostgreSQL`) or via API (`POST /api/db/migrate-to-postgres`):
1. Flushes any in-memory telemetry buffers to SQLite.
2. Connects to PostgreSQL and verifies/initializes the schema.
3. Automatically transfers records table-by-table with validation.
4. Returns a detailed count of all migrated records.

---

## 3. Telemetry Ingestion Cache & Buffer Tuning

### Ingestion Modes: Direct Write vs In-Memory Queue Buffer

| Feature | ⚡ Immediate Direct Write | 🚀 In-Memory Queue Buffer (`enterprise_db`) |
| :--- | :--- | :--- |
| **Ingress Latency** | Low latency, immediate DB commit | Microsecond ingestion, non-blocking |
| **Throughput** | Up to 1,500 events/sec | Up to 50,000+ events/sec |
| **Disk I/O** | Continuous write cycles | Atomic batch flushing |
| **License** | Standard Community Edition | Enterprise License (`enterprise_db`) |

### Tuning Parameters & Metrics
When **In-Memory Queue Buffer** is enabled, administrators can tune the buffer engine:
- **Flush Interval (seconds)**: Maximum elapsed time before buffered anomalies are written to storage (e.g. `2.0`s).
- **Batch Size Threshold**: Number of buffered events that triggers an immediate atomic flush (e.g. `100` events).
- **Max Buffer Capacity**: Upper ceiling of buffered items in memory (e.g. `50,000` events) to protect system RAM under heavy load.

### Manual Flush & Buffer Monitoring
- **Live Monitoring**: The KPI card displays live buffer size, total events ingested, total flushed, and average flush latency.
- **Manual Trigger**: Click **`🌊 Flush Buffer Now`** or call `POST /api/db/flush-buffer` to force an immediate buffer drain.

---

## 4. Zero-Knowledge OpenPGP Data Encryption at Rest

### Encryption Mechanism
When **Data Encryption at Rest** is enabled (`encrypt_at_rest: true`):
- All sensitive log payloads, anomaly messages, and raw traces are encrypted with the server's OpenPGP public key before writing to the database table (`active_issues.message`).
- Remote database administrators or third-party database hosting providers only see armored OpenPGP ciphertext (`-----BEGIN PGP MESSAGE-----`).

### Transparent Authorized Decryption
- The central LOGAR hub instance holds the private OpenPGP key in protected memory.
- When authenticated administrators access the WebUI or when authorized Hermes AI instances query `/api/hermes/report`, the server transparently decrypts records on-the-fly.

---

## 5. Subordinate CA & Dynamic PKI Licensing

LOGAR utilizes a hierarchical Public Key Infrastructure (PKI) for cryptographic client identity and license seat enforcement.

### Hierarchical PKI & Custom X.509 OIDs
The Master Authority (`logar.eibl.tech`) signs the hub's Subordinate CA certificate (`certs/ca.crt`), embedding custom ASN.1 OIDs:
- **`1.3.6.1.4.1.58888.1.1` (Seat Quota)**: Encodes the authorized edge client seat capacity (e.g. `25` seats).
- **`1.3.6.1.4.1.58888.1.2` (Master Authority URL)**: Identifies the upstream licensing authority.
- **`1.3.6.1.4.1.58888.1.3` (Feature Capabilities)**: Comma-delimited list of granted features (e.g. `enterprise_db,ai_heuristics`).

### Licensed Feature Capabilities (`enterprise_db`)
The **📜 Sub-CA & Licensing** tab displays all active licensed features:
- **`enterprise_db`**: Authorizes external PostgreSQL database pooling, high-throughput in-memory RAM buffering, and OpenPGP data-at-rest encryption.

### Importing `.lic` Activation Packages
1. In the WebUI, navigate to **📜 Sub-CA & Licensing** and click **`📥 Import License (.lic)`**.
2. Select your `.lic` bundle or paste the JSON package.
3. The server immediately reloads the PKI trust chain and hot-reloads the active mTLS `SSLContext` without downtime.

### Automated mTLS License Renewal
- Click **`🔄 Sync & Renew via mTLS`** in the WebUI or run:
  ```bash
  python src/Server.py --sync-license
  ```
- The hub connects to `logar.eibl.tech` over mTLS, sends its current seat utilization, and retrieves an updated signed CA certificate.

---

## 6. Fleet Forwarder Management & Seat Provisioning

### Zero-Touch Linux Deployment
Run the automated single-line installer on target Linux servers:
```bash
sudo ./compilation/install_linux_client.sh "http://<HUB_IP>:8443" "<ENROLLMENT_SECRET>"
```
- Automatically generates a 2048-bit RSA client keypair.
- Enrolls with the hub and receives a signed client certificate.
- Deploys and starts the `logar-client.service` systemd unit.

### Windows Edge Client Deployment
1. Download and run `LOGAR-Client-Setup.exe` on the Windows host.
2. In the Server WebUI, click **`📥 Export Client Profile`** and download `client_config.json`.
3. Place `client_config.json` into `C:\Program Files\LOGAR\client_config.json`.
4. The `LOGAR_Client` service automatically boots and connects via mTLS.

### Seat Lifecycle: Approve, Revoke, Delete
In the **💻 Fleet Forwarders** tab:
- **Revoke**: Temporarily blocks the node at the TLS handshake level (`ssl.CERT_REQUIRED`).
- **Approve**: Re-activates a revoked node and restores telemetry ingestion.
- **Remove**: Permanently deletes the client record and immediately frees the leased seat quota.

---

## 7. Hermes AI Agent Integration

### Dedicated Bearer Tokens
1. Navigate to **🤖 Hermes AI & Tokens** in the WebUI.
2. Enter a descriptive token name (e.g. `Hermes-Autonomous-SecOps`) and click **`Generate Token`**.
3. Copy the Bearer token for Hermes agent configuration.

### 12-Hour Temporal Persistence Anomaly Stream
Hermes queries the real-time anomaly stream:
```bash
curl -H "Authorization: Bearer hermes_xxxxxxxxxxxx" http://<HUB_IP>:8443/api/hermes/report
```

#### JSON Telemetry Format:
```json
[
  {
    "fingerprint": "corp-dc:srv-db-01:postgres_wal_checkpoint_delay",
    "site": "corp-dc",
    "server": "srv-db-01",
    "signature": "postgres_wal_checkpoint_delay",
    "severity": "WARNING",
    "message": "WAL checkpoint write duration exceeded 30000ms threshold",
    "os_type": "linux",
    "first_seen": "2026-09-30 14:10:00",
    "last_seen": "2026-09-30 17:42:15",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]
```

---

## 8. Security, Authentication & 2FA

### Password Policies & PBKDF2 Hashing
- User passwords are protected using PBKDF2-HMAC-SHA256 with 100,000 iterations and 16-byte random salts.
- Change admin credentials via the user avatar menu -> **🔑 Change Password**.

### Time-Based One-Time Password (TOTP 2FA)
- Standard RFC 6238 TOTP with standard 30-second drift windows.
- Configure via avatar menu -> **🛡️ Two-Factor Auth (2FA)**.
- Compatible with Google Authenticator, Authy, 1Password, and Bitwarden.

### Sliding Window IP Rate Limiting
- **Brute-Force Protection**: Enforces rate limiting on `/login` and `/api/client/enroll`.
- Excess failed attempts trigger automatic IP blocking with sliding exponential backoff.

---

## 9. Configuration Reference & CLI Flags

### `server_config.json` Specification

```json
{
  "server_name": "LOGAR-Cloud-Hub",
  "tcp_host": "0.0.0.0",
  "tcp_port": 9443,
  "hermes_host": "0.0.0.0",
  "hermes_port": 8443,
  "admin_user": "admin",
  "enrollment_secret": "9a7f8e...4b",
  "max_seats": 25,
  "license_status": "active",
  "license_server_url": "https://logar.eibl.tech",
  "cert_dir": "certs",
  "tls_enabled": true,
  "db_path": "logar_state.db",
  "storage_engine": "sqlite",
  "cache_mode": "direct",
  "flush_interval": 2.0,
  "batch_size": 100,
  "max_queue": 50000,
  "encrypt_at_rest": false,
  "server_fingerprint": "6FF5BE81...",
  "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
  "private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
}
```

### CLI Arguments

| Flag | Description |
| :--- | :--- |
| `--config <path>` | Path to server configuration JSON file |
| `--create-client-config` | Exports a client config bundle and exits |
| `--client-out <path>` | Destination path for exported client configuration |
| `--server-host <host>` | Embedded target host for exported client configuration |
| `--server-port <port>` | Embedded target port for exported client configuration |
| `--export-lic <path>` | Exports a `.lic` CSR activation request package |
| `--import-lic <path>` | Imports and activates a signed `.lic` license package |
| `--sync-license` | Connects to master authority to renew certificate & seats |

---

## 10. REST API Reference

| Method | Endpoint | Auth | Description |
| :--- | :--- | :--- | :--- |
| `GET` | `/health` | None | Health check, server name, and TLS status |
| `POST` | `/api/client/enroll` | Secret | Edge forwarder mTLS enrollment & certificate issuance |
| `GET` | `/api/clients` | Admin | List registered forwarders, activity times, and seat quota |
| `POST` | `/api/clients/approve` | Admin | Reactivate / approve an enrolled forwarder seat |
| `POST` | `/api/clients/revoke` | Admin | Revoke an enrolled forwarder seat |
| `POST` | `/api/clients/delete` | Admin | Remove forwarder record and permanently free seat |
| `GET` | `/api/hermes/report` | Hermes/Admin | Stream verified anomalies within 12h temporal window |
| `POST` | `/api/hermes/tokens/create` | Admin | Generate a named Bearer token for Hermes AI |
| `POST` | `/api/hermes/tokens/revoke` | Admin | Revoke an existing Hermes Bearer token |
| `GET` | `/api/db/stats` | Admin | Telemetry buffer stats, queue metrics, and storage info |
| `POST` | `/api/db/flush-buffer` | Admin | Force immediate drain and write of buffered telemetry |
| `POST` | `/api/db/test-connection` | Admin | Test external PostgreSQL connectivity (`enterprise_db`) |
| `POST` | `/api/db/switch-backend` | Admin | Switch backend between SQLite and PostgreSQL (`enterprise_db`) |
| `POST` | `/api/db/config` | Admin | Persist DB engine, cache mode, and encryption settings |
| `POST` | `/api/db/migrate-to-postgres` | Admin | Migrate SQLite tables to PostgreSQL (`enterprise_db`) |
| `GET` | `/api/license/status` | Admin | Subordinate CA status, seat quota, and licensed features |
| `POST` | `/api/license/import` | Admin | Import `.lic` bundle or signed CA PEM |
| `POST` | `/api/license/sync` | Admin | Sync license seats with master authority over mTLS |

---

## 11. Operational Runbook & Troubleshooting

### Diagnostic Checklist

1. **Verify Hub Service Status**:
   ```bash
   # Linux
   sudo systemctl status logar-server.service
   # Windows PowerShell
   Get-Service LOGAR_Server
   ```
2. **Check Health Endpoint**:
   ```bash
   curl http://localhost:8443/health
   ```
3. **Inspect Active Anomaly Telemetry**:
   ```bash
   curl -H "Authorization: Bearer <ADMIN_OR_HERMES_TOKEN>" http://localhost:8443/api/hermes/report
   ```
4. **Certificate Validity Check**:
   ```bash
   openssl x509 -in certs/ca.crt -text -noout | grep -A 2 "Validity"
   ```

### Common Issues & Remediation

- **Issue: Forwarder rejected with `Connection Refused` or `TLS Handshake Failure`**:
  - Ensure port `9443` is reachable through host and cloud firewalls.
  - Verify edge node clock synchronization (NTP) with the server hub.
  - Confirm the client certificate has not been revoked in the **Fleet Forwarders** tab.
- **Issue: `403 Forbidden` on PostgreSQL or Buffering settings**:
  - The `enterprise_db` feature capability is required. Verify your license in **📜 Sub-CA & Licensing**.
- **Issue: Rate limit lock on admin login**:
  - Wait 60 seconds for the sliding attempt window to expire, or authenticate via an existing active session.