388 lines
17 KiB
Markdown
388 lines
17 KiB
Markdown
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
|
|
|
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
1. [Core Philosophy](#core-philosophy)
|
|
2. [Architecture & Data Flow](#architecture--data-flow)
|
|
3. [Security & Cryptographic Model](#security--cryptographic-model)
|
|
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
|
|
5. [Agentic Hermes Integration](#agentic-hermes-integration)
|
|
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
|
7. [Repository & Shippables Structure](#repository--shippables-structure)
|
|
8. [Getting Started & Installation](#getting-started--installation)
|
|
9. [Running Tests](#running-tests)
|
|
10. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)
|
|
|
|
---
|
|
|
|
## Core Philosophy
|
|
|
|
### 1. Edge Thinness & Zero State
|
|
Site agents running on Windows and Linux act strictly as lightweight forwarders:
|
|
- **No Local Database**: Clients maintain zero state and no local SQLite or cache files.
|
|
- **Source-Level Filtering**: Agents upload candidate entries spanning from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), while stripping verbose debugging noise (`DEBUG`, audit entries) and skipping any entries older than 24 hours.
|
|
- **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
|
|
- **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.
|
|
|
|
### 2. Cloud-Side Temporal Persistence & Severity Routing
|
|
The central Python/TCP hub handles the heavy lifting:
|
|
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
|
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
|
|
- **Warning Persistence (4-Run Rule)**: `WARNING` level issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering out transient blips.
|
|
- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.
|
|
|
|
### 3. Agentic Integration with Hermes
|
|
Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise team notifications.
|
|
|
|
---
|
|
|
|
## Architecture & Data Flow
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Edge Nodes [State-Tracking Edge Forwarders]
|
|
W[Win_Client.py / Win_Client.exe / Win_Client.pyz<br/>Windows Event Log Application]
|
|
L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p info]
|
|
end
|
|
|
|
subgraph Security Layer [Security & Framing]
|
|
E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint]
|
|
S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope]
|
|
end
|
|
|
|
subgraph Cloud Hub [LOGAR Central Server Hub]
|
|
TCP[Authenticated TCP Listener<br/>Port 9443]
|
|
DEC[OpenPGP Decryption<br/>Server Private Key]
|
|
DB[(SQLite Persistence<br/>active_issues & ingest_runs)]
|
|
RULE{12h Window &<br/>4-Run Rule}
|
|
end
|
|
|
|
subgraph Agentic Reporting [Downstream Integration]
|
|
API[FastAPI / Uvicorn Reporting<br/>Port 8443]
|
|
HERMES[Hermes Agent<br/>GET /api/hermes/report]
|
|
end
|
|
|
|
W --> E
|
|
L --> E
|
|
E --> S
|
|
S -->|TCP Stream| TCP
|
|
TCP --> DEC
|
|
DEC --> RULE
|
|
RULE --> DB
|
|
DB --> API
|
|
API --> HERMES
|
|
```
|
|
|
|
---
|
|
|
|
## Security & Cryptographic Model
|
|
|
|
### Pure-Python OpenPGP (RFC 4880)
|
|
- **Zero OS Binary Dependency**: Utilizes `pgpy` and `cryptography` in pure Python. **No native GnuPG or `gpg` binary installation is required** on the server, Windows nodes, or Linux nodes.
|
|
- **First-Run Automatic Key Generation**: On the first launch, if `server_config.json` is missing, `Server.py` automatically generates:
|
|
- An OpenPGP RSA 2048 keypair with encryption-only usage flags.
|
|
- An armored private key (`private_key`) and public key (`public_key`).
|
|
- A SHA-256 public encryption fingerprint (`server_fingerprint`).
|
|
- A cryptographically random authentication secret token (`auth_token`).
|
|
- **Client Configuration Exporter**:
|
|
```bash
|
|
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
|
|
```
|
|
Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint.
|
|
- **Socket Protocol Framing**:
|
|
- `[4 bytes big-endian unsigned int]` : Total envelope length.
|
|
- `[JSON Envelope]` :
|
|
```json
|
|
{
|
|
"auth_token": "<SECRET_TOKEN>",
|
|
"timestamp": "2026-09-03T...",
|
|
"encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..."
|
|
}
|
|
```
|
|
- Unauthorized clients or invalid authentication tokens are rejected immediately.
|
|
|
|
---
|
|
|
|
## Cloud-Side Temporal Persistence & 4-Run Rule
|
|
|
|
Incoming candidate logs are tracked in SQLite table `active_issues`:
|
|
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
|
|
- **12-Hour Evaluation Window**:
|
|
- When an issue is observed, the hub compares `(now - last_seen)`.
|
|
- If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1`.
|
|
- **4-Run Rule for Warnings**:
|
|
- The 4-run persistence threshold specifically applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
|
|
- For each distinct run batch, `run_count` increments.
|
|
- Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
|
|
- When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
|
|
- **Immediate Verification for Errors**:
|
|
- High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
|
|
- On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.
|
|
|
|
---
|
|
|
|
## Agentic Hermes Integration
|
|
|
|
The server hub serves a REST reporting API (default port `8443`):
|
|
|
|
### `GET /api/hermes/report`
|
|
Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour evaluation window):
|
|
|
|
```json
|
|
[
|
|
{
|
|
"fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash",
|
|
"site": "corp.internal",
|
|
"server": "web-app-01.corp.internal",
|
|
"signature": "NginxWorkerCrash",
|
|
"severity": "ERROR",
|
|
"message": "Worker process 4120 terminated with signal 11",
|
|
"os_type": "linux",
|
|
"first_seen": "2026-09-03T09:00:00+00:00",
|
|
"last_seen": "2026-09-03T21:00:00+00:00",
|
|
"consecutive_runs": 4,
|
|
"evaluation_window": "12h",
|
|
"verified": true,
|
|
"status": "VERIFIED"
|
|
}
|
|
]
|
|
```
|
|
|
|
### `GET /api/hermes/all`
|
|
Diagnostic endpoint listing all active issues (both `TRANSIENT` candidate blips and `VERIFIED` anomalies).
|
|
|
|
### `GET /health`
|
|
Returns hub health, encryption fingerprint, and listener ports.
|
|
|
|
---
|
|
|
|
## Dynamic Machine & Domain Identification
|
|
|
|
Client configurations intentionally contain **no machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
|
|
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
|
|
2. **OS-Specific Domain Discovery**:
|
|
- **Windows**: Checks Active Directory environment variable `USERDNSDOMAIN` / `USERDOMAIN`.
|
|
- **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
|
|
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
|
|
4. **Fallback**: Local hostname `socket.gethostname()`.
|
|
|
|
The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`).
|
|
|
|
---
|
|
|
|
## Repository & Shippables Structure
|
|
|
|
```
|
|
LOGAR/
|
|
├── .gitea/
|
|
│ └── workflows/
|
|
│ ├── ci.yml # Continuous Integration automated test suite (runs on every push)
|
|
│ ├── release-linux.yml # Linux release workflow (compiles Server.bin, Linux_Client.bin, checksums)
|
|
│ └── release-windows.yml # Windows release workflow (compiles Server.exe, Win_Client.exe, checksums)
|
|
├── compilation/ # Build, packaging, and release automation tools
|
|
│ ├── package_dist.py # Multi-platform standalone binary packaging script
|
|
│ ├── requirements.txt # Unified project dependencies
|
|
│ └── upload_release.py # Direct Gitea REST API release asset publisher
|
|
├── src/ # Core application source modules
|
|
│ ├── __init__.py
|
|
│ ├── Server.py # Central TCP server and Hermes API
|
|
│ ├── Win_Client.py # Windows edge forwarder
|
|
│ └── Linux_Client.py # Linux edge forwarder
|
|
├── tests/ # Automated test suites
|
|
│ ├── test_linux_client.py # Linux client unit tests
|
|
│ ├── test_pipeline.py # End-to-end integration test
|
|
│ ├── test_server.py # Server unit tests
|
|
│ └── test_win_client.py # Windows client unit tests
|
|
├── .gitignore # Ignore venv, caches, DBs, and private keys
|
|
├── README.md # Comprehensive documentation
|
|
├── RELEASE_NOTES.md # Release history and changelog
|
|
├── server_config.sample.json # Central server sample configuration
|
|
└── out/ # Standalone deployment documentation & sample configs
|
|
├── linux_server/
|
|
│ ├── README.md # Linux systemd service installation & hub guide
|
|
│ └── server_config.sample.json # Reference server configuration
|
|
├── win_server/
|
|
│ ├── README.md # Windows service (NSSM/Task Scheduler) installation guide
|
|
│ └── server_config.sample.json # Reference server configuration
|
|
├── linux_client/
|
|
│ ├── README.md # Linux service & timer installation guide
|
|
│ └── client_config.sample.json # Reference client configuration
|
|
└── win_client/
|
|
├── README.md # Windows service installation & configuration guide
|
|
└── client_config.sample.json # Reference client configuration
|
|
```
|
|
|
|
---
|
|
|
|
## Getting Started & Installation
|
|
|
|
### 1. Central Server Hub
|
|
|
|
1. **Install dependencies**:
|
|
```bash
|
|
pip install -r compilation/requirements.txt
|
|
```
|
|
2. **Start the server** (generates `server_config.json` and keypair on first run):
|
|
```bash
|
|
python src/Server.py
|
|
```
|
|
3. **Export a client configuration**:
|
|
```bash
|
|
python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
|
|
```
|
|
|
|
### 2. Windows Client Deployment
|
|
|
|
#### Option A: Precompiled Standalone Executable (Recommended)
|
|
1. Download `Win_Client.exe` (or `Win_Client.pyz`) from the repository releases.
|
|
2. Place `client_config.json` (exported from the server) in the same directory.
|
|
3. Run manually or schedule via Task Scheduler (every 3 hours):
|
|
```powershell
|
|
.\Win_Client.exe --hours 24
|
|
```
|
|
|
|
#### Option B: Python Source Execution
|
|
1. Copy `Win_Client.py`, `requirements.txt`, and `client_config.json` to the target machine.
|
|
2. Install client dependencies:
|
|
```powershell
|
|
python -m pip install -r requirements.txt
|
|
```
|
|
3. Run manually or schedule via Task Scheduler:
|
|
```powershell
|
|
python Win_Client.py --hours 24
|
|
```
|
|
|
|
### 3. Linux Client Deployment
|
|
|
|
#### Option A: Precompiled Standalone Binary (Recommended)
|
|
1. Download `Linux_Client.bin` from the repository releases.
|
|
2. Place `Linux_Client.bin` and `client_config.json` into `/opt/logar/` and make it executable:
|
|
```bash
|
|
chmod +x /opt/logar/Linux_Client.bin
|
|
```
|
|
3. Run via cron or systemd timer:
|
|
```bash
|
|
0 */3 * * * /opt/logar/Linux_Client.bin --hours 24
|
|
```
|
|
|
|
#### Option B: Python Source Execution
|
|
1. Copy `Linux_Client.py`, `requirements.txt`, and `client_config.json` to `/opt/logar/`.
|
|
2. Install client dependencies:
|
|
```bash
|
|
python3 -m pip install -r requirements.txt
|
|
```
|
|
3. (Optional) Run `out/linux_client/build_bin.sh` to compile a standalone ELF binary locally if desired.
|
|
4. Run via cron or systemd timer:
|
|
```bash
|
|
0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 24
|
|
```
|
|
|
|
---
|
|
|
|
## Running Tests
|
|
|
|
### 1. Component-Specific Unit Tests
|
|
The test suite is located in `tests/` and exercises all components:
|
|
|
|
```bash
|
|
# Run all unit tests
|
|
python -m unittest discover -s tests
|
|
|
|
# Or run component tests individually:
|
|
python -m unittest tests/test_server.py
|
|
python -m unittest tests/test_win_client.py
|
|
python -m unittest tests/test_linux_client.py
|
|
```
|
|
|
|
### 2. End-to-End Pipeline Integration Test
|
|
The pipeline test exercises invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API report output.
|
|
|
|
1. **Start the server** in Shell 1 (creates `server_config.json` on first run):
|
|
```bash
|
|
python src/Server.py
|
|
```
|
|
2. **Export client configuration** in Shell 2 (required for testing):
|
|
```bash
|
|
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
|
|
```
|
|
3. **Execute the integration test** in Shell 2:
|
|
```bash
|
|
python tests/test_pipeline.py
|
|
```
|
|
|
|
## Continuous Integration via Gitea Actions
|
|
|
|
Continuous integration is automated via [`.gitea/workflows/ci.yml`](.gitea/workflows/ci.yml) and triggers automatically on **every push** and pull request:
|
|
1. **Syntax Compilation**: Validates all Python scripts (`src/Server.py`, `src/Win_Client.py`, `src/Linux_Client.py`, `compilation/package_dist.py`, `compilation/upload_release.py`, `tests/test_pipeline.py`, and test suites).
|
|
2. **Component Unit Tests**: Discovers and runs all unit tests in `tests/` (`test_server.py`, `test_win_client.py`, `test_linux_client.py`).
|
|
3. **End-to-End Pipeline Verification**: Automatically spins up the LOGAR server hub, generates test configs, runs `tests/test_pipeline.py` (testing socket authentication, 4-run rule persistence, Hermes API report, and client integrations), and shuts down the test instance.
|
|
|
|
---
|
|
|
|
## Automated Releases via Gitea Actions
|
|
|
|
Release builds are automated via two dedicated Gitea Actions workflows running concurrently on native platform runners:
|
|
- [`.gitea/workflows/release-linux.yml`](.gitea/workflows/release-linux.yml) (`ubuntu-latest`)
|
|
- [`.gitea/workflows/release-windows.yml`](.gitea/workflows/release-windows.yml) (`windows-latest`)
|
|
|
|
### Publishing a Release
|
|
Whenever you want to release a new version with compiled standalone binaries:
|
|
```bash
|
|
git tag v1.0.1
|
|
git push origin v1.0.1
|
|
```
|
|
*(You can also trigger builds manually via the Gitea UI using the **Run workflow** button (`workflow_dispatch`) on either workflow).*
|
|
|
|
### Automated Multi-Platform Compilation:
|
|
1. **Linux Runner** (`release-linux.yml`):
|
|
- Compiles native Linux ELF executables: `Linux_Client.bin` and `Server.bin`.
|
|
- Generates dedicated SHA-256 checksum files:
|
|
- `linux_client_sha256sum` (verification for `Linux_Client.bin`)
|
|
- `linux_agent_sha256sum` (alias for client/agent integrations)
|
|
- `linux_server_sha256sum` (verification for `Server.bin`)
|
|
- `SHA256SUMS_linux.txt` (summary manifest)
|
|
- Attaches all Linux assets to the Gitea release.
|
|
|
|
2. **Windows Runner** (`release-windows.yml`):
|
|
- Compiles native Windows PE executables: `Win_Client.exe` and `Server.exe`.
|
|
- Generates dedicated SHA-256 checksum files:
|
|
- `win_client_sha256sum` (verification for `Win_Client.exe`)
|
|
- `win_agent_sha256sum` (alias for client/agent integrations)
|
|
- `win_server_sha256sum` (verification for `Server.exe`)
|
|
- `SHA256SUMS_windows.txt` (summary manifest)
|
|
- Attaches all Windows assets to the Gitea release.
|
|
|
|
3. **Concurrent Publishing & Conflict Handling**:
|
|
`upload_release.py` includes automatic retry and conflict resolution so concurrent Windows and Linux runners attach their respective assets to the release without collision.
|
|
|
|
### Verifying Checksums
|
|
- On Linux:
|
|
```bash
|
|
sha256sum -c linux_client_sha256sum
|
|
# or
|
|
sha256sum -c linux_server_sha256sum
|
|
```
|
|
- On Windows (PowerShell):
|
|
```powershell
|
|
Get-FileHash .\Win_Client.exe -Algorithm SHA256
|
|
Get-Content .\win_client_sha256sum
|
|
```
|
|
|
|
### Local Packaging & Manual Upload
|
|
You can also compile and package binaries locally anytime:
|
|
```bash
|
|
# Windows
|
|
py -3.12 compilation/package_dist.py --target windows
|
|
|
|
# Linux
|
|
python3 compilation/package_dist.py --target linux
|
|
```
|
|
To upload local builds directly to Gitea:
|
|
```powershell
|
|
python compilation/upload_release.py --tag v1.0.1 --token <YOUR_GITEA_TOKEN>
|
|
```
|
|
*(Environment variables `GITEA_TOKEN`, `GITEA_SERVER_URL`, `GITEA_REPOSITORY`, and `GITEA_REF_NAME` are also supported automatically).*
|