diff --git a/LOGAR-LICENSING b/LOGAR-LICENSING index b2d668a..6f1af3e 160000 --- a/LOGAR-LICENSING +++ b/LOGAR-LICENSING @@ -1 +1 @@ -Subproject commit b2d668af4250c2cac62a36a0e78adc23effd7526 +Subproject commit 6f1af3ea71b2f954a3f63646f5e925de2e890e5a diff --git a/README.md b/README.md index 67605f1..83b3a64 100644 --- a/README.md +++ b/README.md @@ -1,93 +1,154 @@ -# LOGAR: Edge-Thin Log Analysis & Temporal Verification System +# LOGAR: Fleet Log Aggregation & Centralized Provisioning System +### *Sichere Log-Zusammenführung, PKI-Flottenlizenzierung & Bereitstellung für autonome KI-Analyse* -**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, and direct anomaly pass-through enabling **Hermes AI** to perform all deep reasoning, anomaly evaluation, and autonomous decision-making. +**LOGAR** is an enterprise-grade log aggregation, normalization, and centralized provisioning system (**"Log-Zusammenführung und Bereitstellung"**) built for heterogeneous server fleets (Windows & Linux). It pairs ultra-lightweight, zero-state edge forwarders with a secure centralized server hub that enforces mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI fleet licensing with cryptographic seat leasing, 12-hour temporal persistence tracking, and direct anomaly pass-through—delivering clean, structured, verified log telemetry directly to **Hermes AI** for autonomous deep reasoning, contextual root-cause analysis, and operational decision-making. --- ## Table of Contents -1. [Core Philosophy](#core-philosophy) -2. [Architecture & Data Flow](#architecture--data-flow) -3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing) -4. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning) -5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api) -6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification) -7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows) -8. [Repository & Shippables Structure](#repository--shippables-structure) -9. [Getting Started & Deployment](#getting-started--deployment) -10. [Running Tests](#running-tests) -11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions) +1. [Core Purpose & Philosophy (Log-Zusammenführung & Bereitstellung)](#core-purpose--philosophy) +2. [End-to-End System Architecture](#end-to-end-system-architecture) +3. [System State Machine & Enrollment Lifecycle](#system-state-machine--enrollment-lifecycle) +4. [Master License Authority & PKI Fleet Licensing](#master-license-authority--pki-fleet-licensing) +5. [mTLS 1.3 Transport Security & Forwarder Auto-Enrollment](#mtls-13-transport-security--forwarder-auto-enrollment) +6. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning) +7. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api) +8. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification) +9. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows) +10. [Repository & Shippables Structure](#repository--shippables-structure) +11. [Getting Started & Deployment](#getting-started--deployment) +12. [Running Tests](#running-tests) +13. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions) --- -## Core Philosophy +## Core Purpose & Philosophy -### 1. Edge Thinness & Zero State -Site agents running on Windows and Linux act strictly as lightweight forwarders: -- **No Local Database**: Clients maintain zero local SQLite or heavy cache files. -- **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours. -- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates. -- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret. +### 1. Centralized Log Aggregation & Provisioning (*Log-Zusammenführung & Bereitstellung*) +The foundational objective of LOGAR is reliable, secure collection and structured delivery of system events: +- **Heterogeneous Fleet Aggregation**: Ingests Windows Event Logs (System, Application, Security) and Linux `systemd journalctl` events into a unified, normalized JSON data format. +- **Centralized Provisioning**: Serves as the authoritative clearinghouse and real-time streaming endpoint (`/api/hermes/report`), providing structured logs with temporal metadata to external consumers and automated AI agents. +- **Zero Local Edge Footprint**: Forwarders maintain zero local databases, zero cache files, and zero disk overhead. If a network disruption occurs, agents automatically re-establish mTLS streaming without state corruption. -### 2. Cloud-Side Temporal Persistence & Direct Hermes Reasoning -The central Python hub handles state and temporal tracking: -- State tracking is managed centrally in SQLite (`logar_state.db`). -- Candidate issues are tracked over a **12-hour temporal evaluation window** with consecutive run counts and first/last-seen timestamps. -- **Direct Hermes Intelligence**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes. Hermes performs all multi-factor reasoning, pattern evaluation, and contextual thinking rather than relying on arbitrary hardcoded threshold rules. +### 2. Direct Hermes Intelligence (No Heuristic Bottlenecks) +LOGAR strictly separates **data aggregation/provisioning** from **semantic reasoning**: +- **Removal of Arbitrary Heuristics**: The legacy 4-run throttling rule was eliminated. All ingested candidate logs (`INFO`, `WARNING`, `ERROR`, `CRITICAL`) within the active 12-hour temporal evaluation window are immediately verified and provisioned. +- **Hermes Autonomous Thinking**: **Hermes AI** performs all multi-factor reasoning, cross-server correlation, urgency prioritization, and automated remediation decisions directly on the provisioned log stream. -### 3. Agentic Integration with Hermes -Instead of engineers manually sifting through logs, **Hermes** ingests verified anomalies directly from the cloud hub (`GET /api/hermes/report`), treating errors and warnings with full temporal metadata (run counts, timestamps) as actionable system artifacts to trigger precise notifications and remediations. +### 3. Hierarchical Dynamic PKI & Fleet Licensing +- **Authoritative Master PKI**: Subordinate server hubs do not require manual certificate generation or static keys. The **Master License Authority** (`logar.eibl.tech`) issues cryptographically signed Sub-CA certificates with custom X.509 seat quotas (`1.3.6.1.4.1.59999.1.1`). +- **Dynamic Seat Leasing**: Server hubs enforce local seat allocations in real-time, instantly revoking or admitting edge nodes while synchronizing with the central authority over bidirectional WebSockets. --- -## Architecture & Data Flow +## End-to-End System Architecture ```mermaid graph TB - subgraph Edge Nodes [Lightweight Edge Forwarders] - W[Win_Client.exe / Win_Client.py
Windows Event Log Ingestion] - L[Linux_Client.bin / Linux_Client.py
systemd journalctl -p warning] + subgraph Authority [Master License Authority - logar.eibl.tech] + M_CA[Master Root CA
RSA-4096 / SHA-256] + AUTH_WS[WebSocket Licensing Hub
/ws/license] + AUTH_DB[(Authority DB
Tenants, Users, 2FA, Seats)] + AUTH_UI[Authority WebUI
User Mgmt & Fleet Quotas] + AUTH_WS --- AUTH_DB + AUTH_UI --- AUTH_DB end - subgraph Enrollment [Dynamic PKI & Licensing] - ENR[POST /api/client/enroll
License Quota & Secret Validation] - CA[Internal Root CA
Signs RSA-2048 Client Cert] + subgraph ServerHub [LOGAR Server Hub - Central Aggregation & Provisioning] + SUB_CA[Subordinate CA
Signed with Seat OID 1.3.6.1.4.1.59999.1.1] + SEAT_MGR[Dynamic Seat Manager
Enforces Seat Quota in SQLite] + MTLS_SRV[mTLS 1.3 Ingestion Server
Port 9443 - ssl.CERT_REQUIRED] + TEMP_ENG[12h Temporal Persistence Engine
Correlates First/Last Seen & Run Counts] + HUB_DB[(logar_state.db
active_issues, clients, config)] + PROV_API[Log Provisioning & Hermes API
Port 8443 - GET /api/hermes/report] + + SUB_CA --> SEAT_MGR + SEAT_MGR --> MTLS_SRV + MTLS_SRV --> TEMP_ENG + TEMP_ENG --> HUB_DB + HUB_DB --> PROV_API end - subgraph Transport [mTLS 1.3 Security Layer] - MTLS[Mutual TLS 1.3 Handshake
Port 9443 - Client Cert Required] - AUTH[Extract Client CN & Fingerprint
Validate Active License in SQLite] + subgraph EdgeNodes [Heterogeneous Edge Forwarders] + WIN[Win_Client.exe / Win_Client.py
Windows Event Log Ingestion] + LIN[Linux_Client.bin / Linux_Client.py
systemd journalctl -p warning] end - subgraph Hub [LOGAR Server Hub] - INGEST[Length-Prefixed Frame Ingestion] - DB[(SQLite Persistence
active_issues, clients, license_config)] - RULE{12h Window &
Temporal Ingestion} + subgraph Downstream [Hermes AI Reasoning Layer] + HERMES[Hermes AI Engine
Autonomous Analysis & Remediation] end - subgraph Downstream [Hermes Agent & Monitoring] - API[FastAPI Reporting & Management
Port 8443] - HERMES[Hermes Agent
GET /api/hermes/report] - end + %% PKI & Licensing Relationships + M_CA -->|Signs Sub-CA Cert via /ws/license| SUB_CA + AUTH_WS <==>|Bidirectional Sync & Live Seat Expansion| SUB_CA + + %% Forwarder Enrollment & mTLS Transport + WIN -->|1. Auto-Enrollment POST /api/client/enroll| SUB_CA + LIN -->|1. Auto-Enrollment POST /api/client/enroll| SUB_CA + SUB_CA -->|2. Issues Client Cert & Root Trust| WIN + SUB_CA -->|2. Issues Client Cert & Root Trust| LIN + + WIN ==>|3. Continuous mTLS 1.3 Stream :9443| MTLS_SRV + LIN ==>|3. Continuous mTLS 1.3 Stream :9443| MTLS_SRV - W -->|Auto-Enrollment| ENR - L -->|Auto-Enrollment| ENR - ENR --> CA - CA -->|ca.crt, client.crt, client.key| W - CA -->|ca.crt, client.crt, client.key| L - - W -->|mTLS Stream| MTLS - L -->|mTLS Stream| MTLS - MTLS --> AUTH - AUTH --> INGEST - INGEST --> RULE - RULE --> DB - DB --> API - API --> HERMES + %% Log Provisioning & Hermes Consumption + PROV_API ==>|Centralized Log Delivery| HERMES ``` --- -## mTLS Security, Dynamic PKI & Fleet Licensing +## System State Machine & Enrollment Lifecycle + +### 1. Hierarchical Licensing & Forwarder Enrollment State Machine + +```mermaid +stateDiagram-v2 + [*] --> AuthorityBootstrap: Operator creates Bootstrap .lic + + state "Master License Authority (logar.eibl.tech)" as AuthState { + AuthorityBootstrap --> PackageGenerated: Generates Communication Keypair + CSR Template + Tenant ID + PackageGenerated --> WebSocketListening: Authority awaits Server Hub on /ws/license + } + + state "LOGAR Server Hub" as HubState { + PackageGenerated --> ServerImportLic: Admin imports .lic package on Server Hub + ServerImportLic --> ServerCSRGenerated: Server generates local Sub-CA Keypair & CSR + ServerCSRGenerated --> WSConnecting: Connects to Authority via wss://logar.eibl.tech/ws/license + WSConnecting --> SubCASigned: Authority signs Sub-CA with embedded Seat OID & returns Master CA + SubCASigned --> HubReady: Hub starts mTLS Listener (Port 9443) & Provisioning API (Port 8443) + + HubReady --> SeatLeaseRequested: Edge Forwarder sends Enrollment Request + state SeatLeaseRequested <> + SeatLeaseRequested --> SeatGranted: Active Seats < Max Seats + SeatLeaseRequested --> SeatRejected: Active Seats >= Max Seats + + SeatGranted --> ClientCertIssued: Server signs Client Cert (CN=client_id) & registers in clients table + SeatRejected --> HubReady: Drops enrollment with HTTP 403 Seat Limit Exceeded + + HubReady --> LiveSeatUpdate: Authority pushes quota change via WebSocket + LiveSeatUpdate --> InFlightReload: Hub updates DB and reloads SSLContext in-flight + InFlightReload --> HubReady + } + + state "Edge Forwarders (Windows / Linux)" as ClientState { + ClientCertIssued --> ClientStoreCert: Saves client.crt, client.key, ca.crt + ClientStoreCert --> mTLSHandshake: Initiates TLS 1.3 Handshake (Port 9443) + mTLSHandshake --> LogStreaming: Streams candidate log frames (INFO..CRITICAL) + LogStreaming --> HealthCheck: Proactive cert check (<30d) or connection drop + HealthCheck --> ClientStoreCert: Auto-renews or self-heals + } + + state "Log Provisioning & Hermes AI" as ProvisionState { + LogStreaming --> TemporalCorrelation: Hub ingests frame & evaluates 12h rolling window + TemporalCorrelation --> IssuePersisted: Upserts active_issues in logar_state.db + IssuePersisted --> HermesReport: Provisioned via GET /api/hermes/report + HermesReport --> HermesThinking: Hermes AI executes multi-factor reasoning & alerting + } +``` + +--- + +## Master License Authority & PKI Fleet Licensing ### 1. TLS 1.3 Mutual Authentication (mTLS) - **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`.