Update README with Log-Zusammenfuehrung & Bereitstellung focus and full PKI licensing state machine
CI Test Suite / Run Component Tests & Pipeline Verification (push) Failing after 20s
CI Test Suite / Run Component Tests & Pipeline Verification (push) Failing after 20s
This commit is contained in:
+1
-1
Submodule LOGAR-LICENSING updated: b2d668af42...6f1af3ea71
@@ -1,93 +1,154 @@
|
||||
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
||||
# LOGAR: Fleet Log Aggregation & Centralized Provisioning System
|
||||
### *Sichere Log-Zusammenführung, PKI-Flottenlizenzierung & Bereitstellung für autonome KI-Analyse*
|
||||
|
||||
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, and direct anomaly pass-through enabling **Hermes AI** to perform all deep reasoning, anomaly evaluation, and autonomous decision-making.
|
||||
**LOGAR** is an enterprise-grade log aggregation, normalization, and centralized provisioning system (**"Log-Zusammenführung und Bereitstellung"**) built for heterogeneous server fleets (Windows & Linux). It pairs ultra-lightweight, zero-state edge forwarders with a secure centralized server hub that enforces mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI fleet licensing with cryptographic seat leasing, 12-hour temporal persistence tracking, and direct anomaly pass-through—delivering clean, structured, verified log telemetry directly to **Hermes AI** for autonomous deep reasoning, contextual root-cause analysis, and operational decision-making.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. [Core Philosophy](#core-philosophy)
|
||||
2. [Architecture & Data Flow](#architecture--data-flow)
|
||||
3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
|
||||
4. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning)
|
||||
5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
|
||||
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
||||
7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
|
||||
8. [Repository & Shippables Structure](#repository--shippables-structure)
|
||||
9. [Getting Started & Deployment](#getting-started--deployment)
|
||||
10. [Running Tests](#running-tests)
|
||||
11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)
|
||||
1. [Core Purpose & Philosophy (Log-Zusammenführung & Bereitstellung)](#core-purpose--philosophy)
|
||||
2. [End-to-End System Architecture](#end-to-end-system-architecture)
|
||||
3. [System State Machine & Enrollment Lifecycle](#system-state-machine--enrollment-lifecycle)
|
||||
4. [Master License Authority & PKI Fleet Licensing](#master-license-authority--pki-fleet-licensing)
|
||||
5. [mTLS 1.3 Transport Security & Forwarder Auto-Enrollment](#mtls-13-transport-security--forwarder-auto-enrollment)
|
||||
6. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning)
|
||||
7. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
|
||||
8. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
||||
9. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
|
||||
10. [Repository & Shippables Structure](#repository--shippables-structure)
|
||||
11. [Getting Started & Deployment](#getting-started--deployment)
|
||||
12. [Running Tests](#running-tests)
|
||||
13. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)
|
||||
|
||||
---
|
||||
|
||||
## Core Philosophy
|
||||
## Core Purpose & Philosophy
|
||||
|
||||
### 1. Edge Thinness & Zero State
|
||||
Site agents running on Windows and Linux act strictly as lightweight forwarders:
|
||||
- **No Local Database**: Clients maintain zero local SQLite or heavy cache files.
|
||||
- **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours.
|
||||
- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
|
||||
- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.
|
||||
### 1. Centralized Log Aggregation & Provisioning (*Log-Zusammenführung & Bereitstellung*)
|
||||
The foundational objective of LOGAR is reliable, secure collection and structured delivery of system events:
|
||||
- **Heterogeneous Fleet Aggregation**: Ingests Windows Event Logs (System, Application, Security) and Linux `systemd journalctl` events into a unified, normalized JSON data format.
|
||||
- **Centralized Provisioning**: Serves as the authoritative clearinghouse and real-time streaming endpoint (`/api/hermes/report`), providing structured logs with temporal metadata to external consumers and automated AI agents.
|
||||
- **Zero Local Edge Footprint**: Forwarders maintain zero local databases, zero cache files, and zero disk overhead. If a network disruption occurs, agents automatically re-establish mTLS streaming without state corruption.
|
||||
|
||||
### 2. Cloud-Side Temporal Persistence & Direct Hermes Reasoning
|
||||
The central Python hub handles state and temporal tracking:
|
||||
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
||||
- Candidate issues are tracked over a **12-hour temporal evaluation window** with consecutive run counts and first/last-seen timestamps.
|
||||
- **Direct Hermes Intelligence**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes. Hermes performs all multi-factor reasoning, pattern evaluation, and contextual thinking rather than relying on arbitrary hardcoded threshold rules.
|
||||
### 2. Direct Hermes Intelligence (No Heuristic Bottlenecks)
|
||||
LOGAR strictly separates **data aggregation/provisioning** from **semantic reasoning**:
|
||||
- **Removal of Arbitrary Heuristics**: The legacy 4-run throttling rule was eliminated. All ingested candidate logs (`INFO`, `WARNING`, `ERROR`, `CRITICAL`) within the active 12-hour temporal evaluation window are immediately verified and provisioned.
|
||||
- **Hermes Autonomous Thinking**: **Hermes AI** performs all multi-factor reasoning, cross-server correlation, urgency prioritization, and automated remediation decisions directly on the provisioned log stream.
|
||||
|
||||
### 3. Agentic Integration with Hermes
|
||||
Instead of engineers manually sifting through logs, **Hermes** ingests verified anomalies directly from the cloud hub (`GET /api/hermes/report`), treating errors and warnings with full temporal metadata (run counts, timestamps) as actionable system artifacts to trigger precise notifications and remediations.
|
||||
### 3. Hierarchical Dynamic PKI & Fleet Licensing
|
||||
- **Authoritative Master PKI**: Subordinate server hubs do not require manual certificate generation or static keys. The **Master License Authority** (`logar.eibl.tech`) issues cryptographically signed Sub-CA certificates with custom X.509 seat quotas (`1.3.6.1.4.1.59999.1.1`).
|
||||
- **Dynamic Seat Leasing**: Server hubs enforce local seat allocations in real-time, instantly revoking or admitting edge nodes while synchronizing with the central authority over bidirectional WebSockets.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Data Flow
|
||||
## End-to-End System Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph Edge Nodes [Lightweight Edge Forwarders]
|
||||
W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
|
||||
L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
|
||||
subgraph Authority [Master License Authority - logar.eibl.tech]
|
||||
M_CA[Master Root CA<br/>RSA-4096 / SHA-256]
|
||||
AUTH_WS[WebSocket Licensing Hub<br/>/ws/license]
|
||||
AUTH_DB[(Authority DB<br/>Tenants, Users, 2FA, Seats)]
|
||||
AUTH_UI[Authority WebUI<br/>User Mgmt & Fleet Quotas]
|
||||
AUTH_WS --- AUTH_DB
|
||||
AUTH_UI --- AUTH_DB
|
||||
end
|
||||
|
||||
subgraph Enrollment [Dynamic PKI & Licensing]
|
||||
ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
|
||||
CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
|
||||
subgraph ServerHub [LOGAR Server Hub - Central Aggregation & Provisioning]
|
||||
SUB_CA[Subordinate CA<br/>Signed with Seat OID 1.3.6.1.4.1.59999.1.1]
|
||||
SEAT_MGR[Dynamic Seat Manager<br/>Enforces Seat Quota in SQLite]
|
||||
MTLS_SRV[mTLS 1.3 Ingestion Server<br/>Port 9443 - ssl.CERT_REQUIRED]
|
||||
TEMP_ENG[12h Temporal Persistence Engine<br/>Correlates First/Last Seen & Run Counts]
|
||||
HUB_DB[(logar_state.db<br/>active_issues, clients, config)]
|
||||
PROV_API[Log Provisioning & Hermes API<br/>Port 8443 - GET /api/hermes/report]
|
||||
|
||||
SUB_CA --> SEAT_MGR
|
||||
SEAT_MGR --> MTLS_SRV
|
||||
MTLS_SRV --> TEMP_ENG
|
||||
TEMP_ENG --> HUB_DB
|
||||
HUB_DB --> PROV_API
|
||||
end
|
||||
|
||||
subgraph Transport [mTLS 1.3 Security Layer]
|
||||
MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
|
||||
AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
|
||||
subgraph EdgeNodes [Heterogeneous Edge Forwarders]
|
||||
WIN[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
|
||||
LIN[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
|
||||
end
|
||||
|
||||
subgraph Hub [LOGAR Server Hub]
|
||||
INGEST[Length-Prefixed Frame Ingestion]
|
||||
DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
|
||||
RULE{12h Window &<br/>Temporal Ingestion}
|
||||
subgraph Downstream [Hermes AI Reasoning Layer]
|
||||
HERMES[Hermes AI Engine<br/>Autonomous Analysis & Remediation]
|
||||
end
|
||||
|
||||
subgraph Downstream [Hermes Agent & Monitoring]
|
||||
API[FastAPI Reporting & Management<br/>Port 8443]
|
||||
HERMES[Hermes Agent<br/>GET /api/hermes/report]
|
||||
end
|
||||
%% PKI & Licensing Relationships
|
||||
M_CA -->|Signs Sub-CA Cert via /ws/license| SUB_CA
|
||||
AUTH_WS <==>|Bidirectional Sync & Live Seat Expansion| SUB_CA
|
||||
|
||||
%% Forwarder Enrollment & mTLS Transport
|
||||
WIN -->|1. Auto-Enrollment POST /api/client/enroll| SUB_CA
|
||||
LIN -->|1. Auto-Enrollment POST /api/client/enroll| SUB_CA
|
||||
SUB_CA -->|2. Issues Client Cert & Root Trust| WIN
|
||||
SUB_CA -->|2. Issues Client Cert & Root Trust| LIN
|
||||
|
||||
WIN ==>|3. Continuous mTLS 1.3 Stream :9443| MTLS_SRV
|
||||
LIN ==>|3. Continuous mTLS 1.3 Stream :9443| MTLS_SRV
|
||||
|
||||
W -->|Auto-Enrollment| ENR
|
||||
L -->|Auto-Enrollment| ENR
|
||||
ENR --> CA
|
||||
CA -->|ca.crt, client.crt, client.key| W
|
||||
CA -->|ca.crt, client.crt, client.key| L
|
||||
|
||||
W -->|mTLS Stream| MTLS
|
||||
L -->|mTLS Stream| MTLS
|
||||
MTLS --> AUTH
|
||||
AUTH --> INGEST
|
||||
INGEST --> RULE
|
||||
RULE --> DB
|
||||
DB --> API
|
||||
API --> HERMES
|
||||
%% Log Provisioning & Hermes Consumption
|
||||
PROV_API ==>|Centralized Log Delivery| HERMES
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## mTLS Security, Dynamic PKI & Fleet Licensing
|
||||
## System State Machine & Enrollment Lifecycle
|
||||
|
||||
### 1. Hierarchical Licensing & Forwarder Enrollment State Machine
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> AuthorityBootstrap: Operator creates Bootstrap .lic
|
||||
|
||||
state "Master License Authority (logar.eibl.tech)" as AuthState {
|
||||
AuthorityBootstrap --> PackageGenerated: Generates Communication Keypair + CSR Template + Tenant ID
|
||||
PackageGenerated --> WebSocketListening: Authority awaits Server Hub on /ws/license
|
||||
}
|
||||
|
||||
state "LOGAR Server Hub" as HubState {
|
||||
PackageGenerated --> ServerImportLic: Admin imports .lic package on Server Hub
|
||||
ServerImportLic --> ServerCSRGenerated: Server generates local Sub-CA Keypair & CSR
|
||||
ServerCSRGenerated --> WSConnecting: Connects to Authority via wss://logar.eibl.tech/ws/license
|
||||
WSConnecting --> SubCASigned: Authority signs Sub-CA with embedded Seat OID & returns Master CA
|
||||
SubCASigned --> HubReady: Hub starts mTLS Listener (Port 9443) & Provisioning API (Port 8443)
|
||||
|
||||
HubReady --> SeatLeaseRequested: Edge Forwarder sends Enrollment Request
|
||||
state SeatLeaseRequested <<choice>>
|
||||
SeatLeaseRequested --> SeatGranted: Active Seats < Max Seats
|
||||
SeatLeaseRequested --> SeatRejected: Active Seats >= Max Seats
|
||||
|
||||
SeatGranted --> ClientCertIssued: Server signs Client Cert (CN=client_id) & registers in clients table
|
||||
SeatRejected --> HubReady: Drops enrollment with HTTP 403 Seat Limit Exceeded
|
||||
|
||||
HubReady --> LiveSeatUpdate: Authority pushes quota change via WebSocket
|
||||
LiveSeatUpdate --> InFlightReload: Hub updates DB and reloads SSLContext in-flight
|
||||
InFlightReload --> HubReady
|
||||
}
|
||||
|
||||
state "Edge Forwarders (Windows / Linux)" as ClientState {
|
||||
ClientCertIssued --> ClientStoreCert: Saves client.crt, client.key, ca.crt
|
||||
ClientStoreCert --> mTLSHandshake: Initiates TLS 1.3 Handshake (Port 9443)
|
||||
mTLSHandshake --> LogStreaming: Streams candidate log frames (INFO..CRITICAL)
|
||||
LogStreaming --> HealthCheck: Proactive cert check (<30d) or connection drop
|
||||
HealthCheck --> ClientStoreCert: Auto-renews or self-heals
|
||||
}
|
||||
|
||||
state "Log Provisioning & Hermes AI" as ProvisionState {
|
||||
LogStreaming --> TemporalCorrelation: Hub ingests frame & evaluates 12h rolling window
|
||||
TemporalCorrelation --> IssuePersisted: Upserts active_issues in logar_state.db
|
||||
IssuePersisted --> HermesReport: Provisioned via GET /api/hermes/report
|
||||
HermesReport --> HermesThinking: Hermes AI executes multi-factor reasoning & alerting
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Master License Authority & PKI Fleet Licensing
|
||||
|
||||
### 1. TLS 1.3 Mutual Authentication (mTLS)
|
||||
- **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`.
|
||||
|
||||
Reference in New Issue
Block a user