me0nline/LOGAR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity
Files
c1d4675321024963199e713eca0717b182d82211
LOGAR/wiki/ADMIN_GUIDE.md
T
me0nline c1d4675321
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m17s
Details
docs: add administrator guide, update self-tests to 51 passing, update documentation and release notes for v0.3.5
2026-09-30 17:58:35 +02:00

370 lines
17 KiB
Markdown
Raw Blame History

# LOGAR Administrator & Operations Guide
### *Complete Operations Manual for Fleet Hub Administration, Database Engines, PKI Licensing & AI Provisioning*
---
## Table of Contents
1. [Executive Summary & System Architecture](#1-executive-summary--system-architecture)
2. [Database Storage Engine Management](#2-database-storage-engine-management)
- [Local SQLite 3 (WAL Mode)](#local-sqlite-3-wal-mode)
- [External PostgreSQL Enterprise Backend](#external-postgresql-enterprise-backend)
- [Automated SQLite to PostgreSQL Migration](#automated-sqlite-to-postgresql-migration)
3. [Telemetry Ingestion Cache & Buffer Tuning](#3-telemetry-ingestion-cache--buffer-tuning)
- [Ingestion Modes: Direct Write vs In-Memory Queue Buffer](#ingestion-modes-direct-write-vs-in-memory-queue-buffer)
- [Tuning Parameters & Metrics](#tuning-parameters--metrics)
- [Manual Flush & Buffer Monitoring](#manual-flush--buffer-monitoring)
4. [Zero-Knowledge OpenPGP Data Encryption at Rest](#4-zero-knowledge-openpgp-data-encryption-at-rest)
- [Encryption Mechanism](#encryption-mechanism)
- [Transparent Authorized Decryption](#transparent-authorized-decryption)
5. [Subordinate CA & Dynamic PKI Licensing](#5-subordinate-ca--dynamic-pki-licensing)
- [Hierarchical PKI & Custom X.509 OIDs](#hierarchical-pki--custom-x509-oids)
- [Licensed Feature Capabilities (`enterprise_db`)](#licensed-feature-capabilities-enterprise_db)
- [Importing `.lic` Activation Packages](#importing-lic-activation-packages)
- [Automated mTLS License Renewal](#automated-mtls-license-renewal)
6. [Fleet Forwarder Management & Seat Provisioning](#6-fleet-forwarder-management--seat-provisioning)
- [Zero-Touch Linux Deployment](#zero-touch-linux-deployment)
- [Windows Edge Client Deployment](#windows-edge-client-deployment)
- [Seat Lifecycle: Approve, Revoke, Delete](#seat-lifecycle-approve-revoke-delete)
7. [Hermes AI Agent Integration](#7-hermes-ai-agent-integration)
- [Dedicated Bearer Tokens](#dedicated-bearer-tokens)
- [12-Hour Temporal Persistence Anomaly Stream](#12-hour-temporal-persistence-anomaly-stream)
8. [Security, Authentication & 2FA](#8-security-authentication--2fa)
- [Password Policies & PBKDF2 Hashing](#password-policies--pbkdf2-hashing)
- [Time-Based One-Time Password (TOTP 2FA)](#time-based-one-time-password-totp-2fa)
- [Sliding Window IP Rate Limiting](#sliding-window-ip-rate-limiting)
9. [Configuration Reference & CLI Flags](#9-configuration-reference--cli-flags)
10. [REST API Reference](#10-rest-api-reference)
11. [Operational Runbook & Troubleshooting](#11-operational-runbook--troubleshooting)
---
## 1. Executive Summary & System Architecture
LOGAR is a centralized log aggregation, normalization, and provisioning system designed for enterprise server fleets. It connects heterogeneous edge nodes (Windows and Linux) with a central server hub over **Mutual TLS 1.3 (mTLS)**, normalizing incoming events and presenting structured anomaly intelligence to **Hermes AI**.
### Core Network Architecture
```mermaid
graph TD
subgraph MasterAuthority [Master License Authority]
ROOT_CA[Master Root CA]
AUTH_API[WebSocket Licensing Hub<br/>logar.eibl.tech]
end
subgraph Hub [LOGAR Server Hub]
SUB_CA[Subordinate CA & PKI Engine]
INGEST_QUEUE[IngestQueueManager<br/>In-Memory RAM Buffer]
STORAGE_ROUTER[Storage Engine Router]
SQLITE[(Local SQLite 3 WAL)]
POSTGRES[(External PostgreSQL Cluster)]
TEMP_ENG[12h Temporal Persistence Engine]
HERMES_API[Hermes REST & WebUI API<br/>Port 8443]
MTLS_SOCKET[mTLS 1.3 Socket Listener<br/>Port 9443]
MTLS_SOCKET --> INGEST_QUEUE
INGEST_QUEUE --> STORAGE_ROUTER
STORAGE_ROUTER --> SQLITE
STORAGE_ROUTER --> POSTGRES
STORAGE_ROUTER --> TEMP_ENG
TEMP_ENG --> HERMES_API
end
subgraph EdgeNodes [Fleet Edge Forwarders]
WIN_NODE[Windows Forwarder<br/>Win_Client.exe]
LIN_NODE[Linux Forwarder<br/>Linux_Client.bin]
end
subgraph AI [Hermes AI Analysis Layer]
HERMES_AGENT[Hermes AI Agent]
end
ROOT_CA -->|Signs Sub-CA Cert| SUB_CA
AUTH_API <==>|Live mTLS Renewal| SUB_CA
WIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET
LIN_NODE ==>|1. Auto-Enrollment / mTLS Ingest| MTLS_SOCKET
HERMES_API ==>|GET /api/hermes/report| HERMES_AGENT
```
---
## 2. Database Storage Engine Management
LOGAR supports dual storage backends: an embedded **Local SQLite 3 (WAL Mode)** engine and an **External PostgreSQL Database Server** for enterprise clustering.
### Local SQLite 3 (WAL Mode)
- **Status**: Standard Default (Included out of the box, zero configuration required).
- **Optimization**: Operates with Write-Ahead Logging (`PRAGMA journal_mode=WAL;`), asynchronous normal synchronization (`PRAGMA synchronous=NORMAL;`), and memory-mapped I/O (`PRAGMA mmap_size=268435456;`).
- **File Location**: Configurable via `db_path` in `server_config.json` (defaults to `logar_state.db`).
### External PostgreSQL Enterprise Backend
- **Requirement**: Unlocked via the `enterprise_db` license capability.
- **Benefits**: Enables horizontal database scalability, clustering, multi-node replication, and centralized enterprise backup policies.
- **Connection Parameters**:
- **Host / IP**: Database hostname or IP address.
- **Port**: Default `5432`.
- **Database**: Target schema/database name (default `logar`).
- **Username / Password**: Authentication credentials.
- **SSL Mode**: Supports `prefer`, `require`, `verify-ca`, `verify-full`, `disable`, and `allow`.
- **Connect Timeout**: Timeout in seconds (default `5`s).
- **Connection Pool Size**: Maximum pooled persistent connections (default `10`).
### Automated SQLite to PostgreSQL Migration
Administrators can migrate all historical logs, runs, active issues, seat allocations, users, and tokens from SQLite to PostgreSQL directly in the WebUI (`💾 DB Engine` -> `🚀 Migrate SQLite Data to PostgreSQL`) or via API (`POST /api/db/migrate-to-postgres`):
1. Flushes any in-memory telemetry buffers to SQLite.
2. Connects to PostgreSQL and verifies/initializes the schema.
3. Automatically transfers records table-by-table with validation.
4. Returns a detailed count of all migrated records.
---
## 3. Telemetry Ingestion Cache & Buffer Tuning
### Ingestion Modes: Direct Write vs In-Memory Queue Buffer
| Feature | ⚡ Immediate Direct Write | 🚀 In-Memory Queue Buffer (`enterprise_db`) |
| :--- | :--- | :--- |
| **Ingress Latency** | Low latency, immediate DB commit | Microsecond ingestion, non-blocking |
| **Throughput** | Up to 1,500 events/sec | Up to 50,000+ events/sec |
| **Disk I/O** | Continuous write cycles | Atomic batch flushing |
| **License** | Standard Community Edition | Enterprise License (`enterprise_db`) |
### Tuning Parameters & Metrics
When **In-Memory Queue Buffer** is enabled, administrators can tune the buffer engine:
- **Flush Interval (seconds)**: Maximum elapsed time before buffered anomalies are written to storage (e.g. `2.0`s).
- **Batch Size Threshold**: Number of buffered events that triggers an immediate atomic flush (e.g. `100` events).
- **Max Buffer Capacity**: Upper ceiling of buffered items in memory (e.g. `50,000` events) to protect system RAM under heavy load.
### Manual Flush & Buffer Monitoring
- **Live Monitoring**: The KPI card displays live buffer size, total events ingested, total flushed, and average flush latency.
- **Manual Trigger**: Click **`🌊 Flush Buffer Now`** or call `POST /api/db/flush-buffer` to force an immediate buffer drain.
---
## 4. Zero-Knowledge OpenPGP Data Encryption at Rest
### Encryption Mechanism
When **Data Encryption at Rest** is enabled (`encrypt_at_rest: true`):
- All sensitive log payloads, anomaly messages, and raw traces are encrypted with the server's OpenPGP public key before writing to the database table (`active_issues.message`).
- Remote database administrators or third-party database hosting providers only see armored OpenPGP ciphertext (`-----BEGIN PGP MESSAGE-----`).
### Transparent Authorized Decryption
- The central LOGAR hub instance holds the private OpenPGP key in protected memory.
- When authenticated administrators access the WebUI or when authorized Hermes AI instances query `/api/hermes/report`, the server transparently decrypts records on-the-fly.
---
## 5. Subordinate CA & Dynamic PKI Licensing
LOGAR utilizes a hierarchical Public Key Infrastructure (PKI) for cryptographic client identity and license seat enforcement.
### Hierarchical PKI & Custom X.509 OIDs
The Master Authority (`logar.eibl.tech`) signs the hub's Subordinate CA certificate (`certs/ca.crt`), embedding custom ASN.1 OIDs:
- **`1.3.6.1.4.1.58888.1.1` (Seat Quota)**: Encodes the authorized edge client seat capacity (e.g. `25` seats).
- **`1.3.6.1.4.1.58888.1.2` (Master Authority URL)**: Identifies the upstream licensing authority.
- **`1.3.6.1.4.1.58888.1.3` (Feature Capabilities)**: Comma-delimited list of granted features (e.g. `enterprise_db,ai_heuristics`).
### Licensed Feature Capabilities (`enterprise_db`)
The **📜 Sub-CA & Licensing** tab displays all active licensed features:
- **`enterprise_db`**: Authorizes external PostgreSQL database pooling, high-throughput in-memory RAM buffering, and OpenPGP data-at-rest encryption.
### Importing `.lic` Activation Packages
1. In the WebUI, navigate to **📜 Sub-CA & Licensing** and click **`📥 Import License (.lic)`**.
2. Select your `.lic` bundle or paste the JSON package.
3. The server immediately reloads the PKI trust chain and hot-reloads the active mTLS `SSLContext` without downtime.
### Automated mTLS License Renewal
- Click **`🔄 Sync & Renew via mTLS`** in the WebUI or run:
```bash
python src/Server.py --sync-license
```
- The hub connects to `logar.eibl.tech` over mTLS, sends its current seat utilization, and retrieves an updated signed CA certificate.
---
## 6. Fleet Forwarder Management & Seat Provisioning
### Zero-Touch Linux Deployment
Run the automated single-line installer on target Linux servers:
```bash
sudo ./compilation/install_linux_client.sh "http://<HUB_IP>:8443" "<ENROLLMENT_SECRET>"
```
- Automatically generates a 2048-bit RSA client keypair.
- Enrolls with the hub and receives a signed client certificate.
- Deploys and starts the `logar-client.service` systemd unit.
### Windows Edge Client Deployment
1. Download and run `LOGAR-Client-Setup.exe` on the Windows host.
2. In the Server WebUI, click **`📥 Export Client Profile`** and download `client_config.json`.
3. Place `client_config.json` into `C:\Program Files\LOGAR\client_config.json`.
4. The `LOGAR_Client` service automatically boots and connects via mTLS.
### Seat Lifecycle: Approve, Revoke, Delete
In the **💻 Fleet Forwarders** tab:
- **Revoke**: Temporarily blocks the node at the TLS handshake level (`ssl.CERT_REQUIRED`).
- **Approve**: Re-activates a revoked node and restores telemetry ingestion.
- **Remove**: Permanently deletes the client record and immediately frees the leased seat quota.
---
## 7. Hermes AI Agent Integration
### Dedicated Bearer Tokens
1. Navigate to **🤖 Hermes AI & Tokens** in the WebUI.
2. Enter a descriptive token name (e.g. `Hermes-Autonomous-SecOps`) and click **`Generate Token`**.
3. Copy the Bearer token for Hermes agent configuration.
### 12-Hour Temporal Persistence Anomaly Stream
Hermes queries the real-time anomaly stream:
```bash
curl -H "Authorization: Bearer hermes_xxxxxxxxxxxx" http://<HUB_IP>:8443/api/hermes/report
```
#### JSON Telemetry Format:
```json
[
{
"fingerprint": "corp-dc:srv-db-01:postgres_wal_checkpoint_delay",
"site": "corp-dc",
"server": "srv-db-01",
"signature": "postgres_wal_checkpoint_delay",
"severity": "WARNING",
"message": "WAL checkpoint write duration exceeded 30000ms threshold",
"os_type": "linux",
"first_seen": "2026-09-30 14:10:00",
"last_seen": "2026-09-30 17:42:15",
"consecutive_runs": 4,
"evaluation_window": "12h",
"verified": true,
"status": "VERIFIED"
}
]
```
---
## 8. Security, Authentication & 2FA
### Password Policies & PBKDF2 Hashing
- User passwords are protected using PBKDF2-HMAC-SHA256 with 100,000 iterations and 16-byte random salts.
- Change admin credentials via the user avatar menu -> **🔑 Change Password**.
### Time-Based One-Time Password (TOTP 2FA)
- Standard RFC 6238 TOTP with standard 30-second drift windows.
- Configure via avatar menu -> **🛡️ Two-Factor Auth (2FA)**.
- Compatible with Google Authenticator, Authy, 1Password, and Bitwarden.
### Sliding Window IP Rate Limiting
- **Brute-Force Protection**: Enforces rate limiting on `/login` and `/api/client/enroll`.
- Excess failed attempts trigger automatic IP blocking with sliding exponential backoff.
---
## 9. Configuration Reference & CLI Flags
### `server_config.json` Specification
```json
{
"server_name": "LOGAR-Cloud-Hub",
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"admin_user": "admin",
"enrollment_secret": "9a7f8e...4b",
"max_seats": 25,
"license_status": "active",
"license_server_url": "https://logar.eibl.tech",
"cert_dir": "certs",
"tls_enabled": true,
"db_path": "logar_state.db",
"storage_engine": "sqlite",
"cache_mode": "direct",
"flush_interval": 2.0,
"batch_size": 100,
"max_queue": 50000,
"encrypt_at_rest": false,
"server_fingerprint": "6FF5BE81...",
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
}
```
### CLI Arguments
| Flag | Description |
| :--- | :--- |
| `--config <path>` | Path to server configuration JSON file |
| `--create-client-config` | Exports a client config bundle and exits |
| `--client-out <path>` | Destination path for exported client configuration |
| `--server-host <host>` | Embedded target host for exported client configuration |
| `--server-port <port>` | Embedded target port for exported client configuration |
| `--export-lic <path>` | Exports a `.lic` CSR activation request package |
| `--import-lic <path>` | Imports and activates a signed `.lic` license package |
| `--sync-license` | Connects to master authority to renew certificate & seats |
---
## 10. REST API Reference
| Method | Endpoint | Auth | Description |
| :--- | :--- | :--- | :--- |
| `GET` | `/health` | None | Health check, server name, and TLS status |
| `POST` | `/api/client/enroll` | Secret | Edge forwarder mTLS enrollment & certificate issuance |
| `GET` | `/api/clients` | Admin | List registered forwarders, activity times, and seat quota |
| `POST` | `/api/clients/approve` | Admin | Reactivate / approve an enrolled forwarder seat |
| `POST` | `/api/clients/revoke` | Admin | Revoke an enrolled forwarder seat |
| `POST` | `/api/clients/delete` | Admin | Remove forwarder record and permanently free seat |
| `GET` | `/api/hermes/report` | Hermes/Admin | Stream verified anomalies within 12h temporal window |
| `POST` | `/api/hermes/tokens/create` | Admin | Generate a named Bearer token for Hermes AI |
| `POST` | `/api/hermes/tokens/revoke` | Admin | Revoke an existing Hermes Bearer token |
| `GET` | `/api/db/stats` | Admin | Telemetry buffer stats, queue metrics, and storage info |
| `POST` | `/api/db/flush-buffer` | Admin | Force immediate drain and write of buffered telemetry |
| `POST` | `/api/db/test-connection` | Admin | Test external PostgreSQL connectivity (`enterprise_db`) |
| `POST` | `/api/db/switch-backend` | Admin | Switch backend between SQLite and PostgreSQL (`enterprise_db`) |
| `POST` | `/api/db/config` | Admin | Persist DB engine, cache mode, and encryption settings |
| `POST` | `/api/db/migrate-to-postgres` | Admin | Migrate SQLite tables to PostgreSQL (`enterprise_db`) |
| `GET` | `/api/license/status` | Admin | Subordinate CA status, seat quota, and licensed features |
| `POST` | `/api/license/import` | Admin | Import `.lic` bundle or signed CA PEM |
| `POST` | `/api/license/sync` | Admin | Sync license seats with master authority over mTLS |
---
## 11. Operational Runbook & Troubleshooting
### Diagnostic Checklist
1. **Verify Hub Service Status**:
```bash
# Linux
sudo systemctl status logar-server.service
# Windows PowerShell
Get-Service LOGAR_Server
```
2. **Check Health Endpoint**:
```bash
curl http://localhost:8443/health
```
3. **Inspect Active Anomaly Telemetry**:
```bash
curl -H "Authorization: Bearer <ADMIN_OR_HERMES_TOKEN>" http://localhost:8443/api/hermes/report
```
4. **Certificate Validity Check**:
```bash
openssl x509 -in certs/ca.crt -text -noout | grep -A 2 "Validity"
```
### Common Issues & Remediation
- **Issue: Forwarder rejected with `Connection Refused` or `TLS Handshake Failure`**:
- Ensure port `9443` is reachable through host and cloud firewalls.
- Verify edge node clock synchronization (NTP) with the server hub.
- Confirm the client certificate has not been revoked in the **Fleet Forwarders** tab.
- **Issue: `403 Forbidden` on PostgreSQL or Buffering settings**:
- The `enterprise_db` feature capability is required. Verify your license in **📜 Sub-CA & Licensing**.
- **Issue: Rate limit lock on admin login**:
- Wait 60 seconds for the sliding attempt window to expire, or authenticate via an existing active session.
Reference in New Issue View Git Blame Copy Permalink