Implement edge filtering, state tracking, clean out/ directory, and add Gitea CI workflow
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m40s
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m40s
This commit is contained in:
+112
-25
@@ -1,50 +1,111 @@
|
||||
# LOGAR Linux Edge Forwarder
|
||||
|
||||
Lightweight edge log forwarder for Linux servers running systemd.
|
||||
Standalone compiled binary distribution for Linux edge servers running systemd.
|
||||
|
||||
## Features
|
||||
- **Zero Local State**: No local SQLite database or state tracking on the edge server.
|
||||
- **Edge Noise Stripping**: Strips conversational/informational noise (`INFO`, `DEBUG`) directly at the source via `journalctl -p warning`.
|
||||
- **End-to-End OpenPGP Encryption**: Encrypts logs using the server's public key; decrypted exclusively on the cloud hub.
|
||||
- **Authenticated TCP Socket**: Direct, low-overhead TCP streaming with token authentication.
|
||||
- **No GPG Binary Required**: Pure-Python implementation (`pgpy` + `cryptography`).
|
||||
---
|
||||
|
||||
## Overview
|
||||
`Linux_Client.bin` is a self-contained, pre-compiled executable that queries `systemd-journald` via `journalctl`, filters logs directly at the source, encrypts the payload using OpenPGP, and streams candidate events over an authenticated TCP socket to the central LOGAR hub.
|
||||
|
||||
### Key Capabilities
|
||||
- **Pre-compiled & Dependency-Free**: Ships as a standalone executable binary (`Linux_Client.bin`). No Python environment, pip packages, or GnuPG binaries are required on the host.
|
||||
- **Source-Level Filtering**: Retains events spanning `INFO`, `WARNING`, and `ERROR` (`journalctl -p info`). Drops debug noise (priority 7) and skips events older than 24 hours.
|
||||
- **State Tracking & Deduplication**: Maintains persistent client state in `client_state.json` (tracking systemd journalctl cursors and microsecond timestamps) so every log record is forwarded exactly once without duplicates.
|
||||
- **Fail-Safe State Commit**: State is committed only when the server returns a verified `success` response. In the event of a network outage, state remains unchanged and unsent events are retried automatically on the next run.
|
||||
- **End-to-End Encryption**: Encrypts payloads using the server's OpenPGP public key before transmission.
|
||||
|
||||
---
|
||||
|
||||
## 1. Generating & Deploying the Configuration File
|
||||
|
||||
### Step 1: Generate `client_config.json` on the Server
|
||||
Run the following command on your central LOGAR server to export a client bundle tailored for your environment:
|
||||
|
||||
## Installation
|
||||
```bash
|
||||
python3 -m pip install -r requirements.txt
|
||||
python Server.py --create-client-config --server-host <SERVER_IP_OR_DNS> --server-port 9443 --client-out client_config.json
|
||||
```
|
||||
|
||||
## Configuration
|
||||
Place `client_config.json` generated by the server (`Server.py --create-client-config`) in the same directory as `Linux_Client.py`.
|
||||
- Replace `<SERVER_IP_OR_DNS>` with the reachable IP address or FQDN of your central LOGAR server hub.
|
||||
- Default TCP port is `9443`.
|
||||
|
||||
## Running the Forwarder
|
||||
```bash
|
||||
python3 Linux_Client.py --hours 6
|
||||
### Step 2: Configuration Structure
|
||||
The generated `client_config.json` contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"server_host": "192.168.1.100",
|
||||
"server_port": 9443,
|
||||
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
|
||||
"server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
|
||||
"auth_token": "a1b2c3d4e5f6..."
|
||||
}
|
||||
```
|
||||
|
||||
## Cron / Systemd Timer Deployment
|
||||
### Option A: Cron Job (Every 3 hours)
|
||||
> [!NOTE]
|
||||
> A reference example is provided in `client_config.sample.json`. The configuration file contains **no host-specific names or site names** to ensure client anonymity and easy redistribution.
|
||||
|
||||
### Step 3: Copy to Edge Node
|
||||
Place `Linux_Client.bin` and `client_config.json` into the target directory (recommended: `/opt/logar/`):
|
||||
|
||||
```bash
|
||||
0 */3 * * * cd /opt/logar && /usr/bin/python3 Linux_Client.py --hours 6 >> /var/log/logar_client.log 2>&1
|
||||
sudo mkdir -p /opt/logar
|
||||
sudo cp Linux_Client.bin client_config.json /opt/logar/
|
||||
sudo chmod +x /opt/logar/Linux_Client.bin
|
||||
```
|
||||
|
||||
### Option B: Systemd Service & Timer
|
||||
1. Create `/etc/systemd/system/logar-forwarder.service`:
|
||||
---
|
||||
|
||||
## 2. Running Manually
|
||||
|
||||
Test the forwarder interactively:
|
||||
|
||||
```bash
|
||||
cd /opt/logar
|
||||
./Linux_Client.bin --hours 24
|
||||
```
|
||||
|
||||
### Command-Line Arguments
|
||||
| Argument | Default | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `--config` | `client_config.json` | Path to client configuration file |
|
||||
| `--hours` | `24` | Lookback window in hours for journal logs |
|
||||
| `--state-file` | `client_state.json` | Path to persistent state file |
|
||||
| `--no-state` | `False` | Disable state tracking and send all events matching lookback window |
|
||||
|
||||
---
|
||||
|
||||
## 3. Installing as a Systemd Service & Timer (Recommended)
|
||||
|
||||
Running `Linux_Client.bin` via a systemd timer ensures reliable periodic execution, automatic restart, and native log integration with `journalctl`.
|
||||
|
||||
### Step 1: Create the Systemd Service Unit
|
||||
Create `/etc/systemd/system/logar-forwarder.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=LOGAR Edge Forwarder
|
||||
After=network.target
|
||||
Description=LOGAR Edge Log Forwarder
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/opt/logar
|
||||
ExecStart=/usr/bin/python3 /opt/logar/Linux_Client.py --hours 6
|
||||
ExecStart=/opt/logar/Linux_Client.bin --hours 24
|
||||
User=root
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
2. Create `/etc/systemd/system/logar-forwarder.timer`:
|
||||
### Step 2: Create the Systemd Timer Unit
|
||||
Create `/etc/systemd/system/logar-forwarder.timer` to execute the forwarder every 3 hours (with a 5-minute initial delay upon boot):
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Run LOGAR Edge Forwarder every 3 hours
|
||||
Description=Run LOGAR Edge Forwarder periodically
|
||||
Requires=logar-forwarder.service
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
@@ -55,8 +116,34 @@ Persistent=true
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
3. Enable and start:
|
||||
### Step 3: Enable and Start the Timer
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now logar-forwarder.timer
|
||||
```
|
||||
|
||||
### Step 4: Verify Timer & Service Status
|
||||
```bash
|
||||
# Check timer schedule
|
||||
sudo systemctl list-timers --all | grep logar
|
||||
|
||||
# Trigger an immediate manual execution
|
||||
sudo systemctl start logar-forwarder.service
|
||||
|
||||
# View execution logs
|
||||
sudo journalctl -u logar-forwarder.service -n 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Alternative: Cron Job Deployment
|
||||
|
||||
If systemd timers are not preferred, configure a periodic cron job running every 3 hours:
|
||||
|
||||
```bash
|
||||
# Open root crontab
|
||||
sudo crontab -e
|
||||
|
||||
# Add the following entry:
|
||||
0 */3 * * * cd /opt/logar && ./Linux_Client.bin --hours 24 >> /var/log/logar_forwarder.log 2>&1
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user