Initial commit: LOGAR edge-thin log analysis system with OpenPGP encryption, authenticated TCP sockets, 4-run persistence rule, Hermes reporting, and modular shippables
This commit is contained in:
@@ -0,0 +1,273 @@
|
||||
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
||||
|
||||
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to **Hermes**.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. [Core Philosophy](#core-philosophy)
|
||||
2. [Architecture & Data Flow](#architecture--data-flow)
|
||||
3. [Security & Cryptographic Model](#security--cryptographic-model)
|
||||
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
|
||||
5. [Agentic Hermes Integration](#agentic-hermes-integration)
|
||||
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
||||
7. [Repository & Shippables Structure](#repository--shippables-structure)
|
||||
8. [Getting Started & Installation](#getting-started--installation)
|
||||
9. [Running Tests](#running-tests)
|
||||
|
||||
---
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### 1. Edge Thinness & Zero State
|
||||
Site agents running on Windows and Linux act strictly as lightweight forwarders:
|
||||
- **No Local Database**: Clients maintain zero state and no local SQLite or cache files.
|
||||
- **Source-Level Noise Stripping**: Conversational, informational, and debugging log noise (`INFO`, `DEBUG`, audit entries) is dropped directly at the source.
|
||||
- **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
|
||||
- **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.
|
||||
|
||||
### 2. Cloud-Side Temporal Persistence
|
||||
The central Python/TCP hub handles the heavy lifting:
|
||||
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
||||
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
|
||||
- An issue must persist across **at least 4 consecutive runs / cycles** to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically.
|
||||
|
||||
### 3. Agentic Integration with Hermes
|
||||
Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (`GET /api/hermes/report`), treating them as verified system artifacts to trigger precise team notifications.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Data Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph Edge Nodes [Zero-State Edge Forwarders]
|
||||
W[Win_Client.py / Win_Client.exe<br/>Windows Event Log Application]
|
||||
L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p warning]
|
||||
end
|
||||
|
||||
subgraph Security Layer [Security & Framing]
|
||||
E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint]
|
||||
S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope]
|
||||
end
|
||||
|
||||
subgraph Cloud Hub [LOGAR Central Server Hub]
|
||||
TCP[Authenticated TCP Listener<br/>Port 9443]
|
||||
DEC[OpenPGP Decryption<br/>Server Private Key]
|
||||
DB[(SQLite Persistence<br/>active_issues & ingest_runs)]
|
||||
RULE{12h Window &<br/>4-Run Rule}
|
||||
end
|
||||
|
||||
subgraph Agentic Reporting [Downstream Integration]
|
||||
API[FastAPI / Uvicorn Reporting<br/>Port 8443]
|
||||
HERMES[Hermes Agent<br/>GET /api/hermes/report]
|
||||
end
|
||||
|
||||
W --> E
|
||||
L --> E
|
||||
E --> S
|
||||
S -->|TCP Stream| TCP
|
||||
TCP --> DEC
|
||||
DEC --> RULE
|
||||
RULE --> DB
|
||||
DB --> API
|
||||
API --> HERMES
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security & Cryptographic Model
|
||||
|
||||
### Pure-Python OpenPGP (RFC 4880)
|
||||
- **Zero OS Binary Dependency**: Utilizes `pgpy` and `cryptography` in pure Python. **No native GnuPG or `gpg` binary installation is required** on the server, Windows nodes, or Linux nodes.
|
||||
- **First-Run Automatic Key Generation**: On the first launch, if `server_config.json` is missing, `Server.py` automatically generates:
|
||||
- An OpenPGP RSA 2048 keypair with encryption-only usage flags.
|
||||
- An armored private key (`private_key`) and public key (`public_key`).
|
||||
- A SHA-256 public encryption fingerprint (`server_fingerprint`).
|
||||
- A cryptographically random authentication secret token (`auth_token`).
|
||||
- **Client Configuration Exporter**:
|
||||
```bash
|
||||
python Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
|
||||
```
|
||||
Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint.
|
||||
- **Socket Protocol Framing**:
|
||||
- `[4 bytes big-endian unsigned int]` : Total envelope length.
|
||||
- `[JSON Envelope]` :
|
||||
```json
|
||||
{
|
||||
"auth_token": "<SECRET_TOKEN>",
|
||||
"timestamp": "2026-09-03T...",
|
||||
"encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..."
|
||||
}
|
||||
```
|
||||
- Unauthorized clients or invalid authentication tokens are rejected immediately.
|
||||
|
||||
---
|
||||
|
||||
## Cloud-Side Temporal Persistence & 4-Run Rule
|
||||
|
||||
Incoming candidate logs are tracked in SQLite table `active_issues`:
|
||||
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
|
||||
- **12-Hour Evaluation Window**:
|
||||
- When an issue is observed, the hub compares `(now - last_seen)`.
|
||||
- If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1` with status `TRANSIENT`.
|
||||
- **4-Run Rule**:
|
||||
- For each distinct run batch, `run_count` increments.
|
||||
- Issues with `run_count < 4` are marked as `TRANSIENT` and ignored by downstream reporting.
|
||||
- When `run_count >= 4` within the active 12-hour window, the status transitions to `VERIFIED`.
|
||||
|
||||
---
|
||||
|
||||
## Agentic Hermes Integration
|
||||
|
||||
The server hub serves a REST reporting API (default port `8443`):
|
||||
|
||||
### `GET /api/hermes/report`
|
||||
Returns exclusively **verified anomalies** that have satisfied the 4-run rule within the active 12-hour evaluation window:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash",
|
||||
"site": "corp.internal",
|
||||
"server": "web-app-01.corp.internal",
|
||||
"signature": "NginxWorkerCrash",
|
||||
"severity": "ERROR",
|
||||
"message": "Worker process 4120 terminated with signal 11",
|
||||
"os_type": "linux",
|
||||
"first_seen": "2026-09-03T09:00:00+00:00",
|
||||
"last_seen": "2026-09-03T21:00:00+00:00",
|
||||
"consecutive_runs": 4,
|
||||
"evaluation_window": "12h",
|
||||
"verified": true,
|
||||
"status": "VERIFIED"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### `GET /api/hermes/all`
|
||||
Diagnostic endpoint listing all active issues (both `TRANSIENT` candidate blips and `VERIFIED` anomalies).
|
||||
|
||||
### `GET /health`
|
||||
Returns hub health, encryption fingerprint, and listener ports.
|
||||
|
||||
---
|
||||
|
||||
## Dynamic Machine & Domain Identification
|
||||
|
||||
Client configurations intentionally contain **no machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
|
||||
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
|
||||
2. **OS-Specific Domain Discovery**:
|
||||
- **Windows**: Checks Active Directory environment variable `USERDNSDOMAIN` / `USERDOMAIN`.
|
||||
- **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
|
||||
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
|
||||
4. **Fallback**: Local hostname `socket.gethostname()`.
|
||||
|
||||
The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`).
|
||||
|
||||
---
|
||||
|
||||
## Repository & Shippables Structure
|
||||
|
||||
```
|
||||
LOGAR/
|
||||
├── .gitignore # Ignore venv, caches, DBs, and private keys
|
||||
├── requirements.txt # Unified dependencies
|
||||
├── README.md # Comprehensive documentation
|
||||
├── Server.py # Central TCP server and Hermes API
|
||||
├── Win_Client.py # Windows edge forwarder
|
||||
├── Linux_Client.py # Linux edge forwarder
|
||||
├── test_pipeline.py # End-to-end integration test
|
||||
└── out/ # Standalone shippable distributions
|
||||
├── server/
|
||||
│ ├── Server.exe # Standalone Windows executable
|
||||
│ ├── Server.py # Python source
|
||||
│ ├── server_config.sample.json
|
||||
│ ├── requirements.txt
|
||||
│ ├── README.md
|
||||
│ └── test/
|
||||
│ └── test_server.py # Server unit tests
|
||||
├── win_client/
|
||||
│ ├── Win_Client.exe # Standalone Windows executable
|
||||
│ ├── Win_Client.py # Python source
|
||||
│ ├── client_config.sample.json
|
||||
│ ├── requirements.txt
|
||||
│ ├── README.md
|
||||
│ └── test/
|
||||
│ └── test_win_client.py # Windows client unit tests
|
||||
└── linux_client/
|
||||
├── Linux_Client.bin # Standalone executable binary (zipapp)
|
||||
├── build_bin.sh # PyInstaller ELF compiler script
|
||||
├── Linux_Client.py # Python source
|
||||
├── client_config.sample.json
|
||||
├── requirements.txt
|
||||
├── README.md
|
||||
└── test/
|
||||
└── test_linux_client.py# Linux client unit tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Started & Installation
|
||||
|
||||
### 1. Central Server Hub
|
||||
|
||||
1. **Install dependencies**:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
2. **Start the server** (generates `server_config.json` and keypair on first run):
|
||||
```bash
|
||||
python Server.py
|
||||
# Or run the standalone executable:
|
||||
./out/server/Server.exe
|
||||
```
|
||||
3. **Export a client configuration**:
|
||||
```bash
|
||||
python Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
|
||||
```
|
||||
|
||||
### 2. Windows Client Deployment
|
||||
|
||||
1. Copy `Win_Client.exe` (or `Win_Client.py`) and `client_config.json` to the target machine.
|
||||
2. Run manually or schedule via Task Scheduler (every 3 hours):
|
||||
```powershell
|
||||
Win_Client.exe --hours 6
|
||||
```
|
||||
|
||||
### 3. Linux Client Deployment
|
||||
|
||||
1. Copy `Linux_Client.bin` (or `Linux_Client.py`) and `client_config.json` to `/opt/logar/`.
|
||||
2. Ensure executable permissions:
|
||||
```bash
|
||||
chmod +x /opt/logar/Linux_Client.bin
|
||||
```
|
||||
3. Run via cron or systemd timer:
|
||||
```bash
|
||||
0 */3 * * * /opt/logar/Linux_Client.bin --hours 6
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running Tests
|
||||
|
||||
### 1. Component-Specific Unit Tests
|
||||
Each component in `out/` includes its own isolated test suite:
|
||||
|
||||
```bash
|
||||
# Server tests (config generation, SQLite persistence, 4-run rule)
|
||||
python out/server/test/test_server.py
|
||||
|
||||
# Windows client tests (config anonymity, machine ID, OpenPGP encryption)
|
||||
python out/win_client/test/test_win_client.py
|
||||
|
||||
# Linux client tests (config anonymity, journalctl priority filter, OpenPGP)
|
||||
python out/linux_client/test/test_linux_client.py
|
||||
```
|
||||
|
||||
### 2. End-to-End Pipeline Integration Test
|
||||
Start the server in one shell and run the pipeline test:
|
||||
```bash
|
||||
python test_pipeline.py
|
||||
```
|
||||
This tests invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API output.
|
||||
Reference in New Issue
Block a user