eosswedenorg/thalos-docs
1
0
Fork 0
mirror of https://github.com/eosswedenorg/thalos-docs synced 2026-10-14 03:46:44 +02:00
Code Releases Activity

docs: align Thalos docs with current config, CLI, and API

Browse source
This commit is contained in:
Henrik Hautakoski 2026-08-31 14:51:54 +02:00
parent 2d90392d97
commit 3bb8bb5c26
11 changed files with 516 additions and 450 deletions
Download patch file Download diff file Expand all files Collapse all files

16
docs/api/clients.md
View file

@ -1,8 +1,8 @@
# Clients
List of officially supported client libraries
| Name | Language |
| -------------------------------------------------------------------------- | -------- |
| [thalos-go](https://github.com/eosswedenorg/thalos/tree/master/api) | Golang |
| [thalos-nodejs](https://www.npmjs.com/package/@eosswedenorg/thalos-client) | Nodejs |
# Clients
List of officially supported client libraries.
| Name | Language |
| -------------------------------------------------------------------------- | -------- |
| [thalos-go](https://github.com/eosswedenorg/thalos/tree/master/api) | Go |
| [@eosswedenorg/thalos-client](https://www.npmjs.com/package/@eosswedenorg/thalos-client) | Node.js |

35
docs/api/examples.md
View file

@ -1,3 +1,7 @@
# Examples
These examples assume Thalos uses `message_codec: json` (default).
If you run with `message_codec: msgpack`, use a MessagePack decoder in your client.
## Go
@ -18,15 +22,15 @@ import (
)
func main() {
// Create redis client
// Create Redis client
rdb := redis.NewClient(&redis.Options{})
sub := api_redis.NewSubscriber(context.Background(), rdb, api_redis.Namespace{
Prefix: "ship",
ChainID: "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4", // Wax mainnet.
ChainID: "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4", // WAX mainnet
})
// Create client
// Create client
codec, err := message.GetCodec("json")
if err != nil {
fmt.Println("Failed to get json codec")
@ -35,7 +39,7 @@ func main() {
client := api.NewClient(sub, codec.Decoder)
// Subscribe to some channels.
// Subscribe to some channels.
err = client.Subscribe(
api.TransactionChannel,
api.ActionChannel{Contract: "eosio"}.Channel(),
@ -49,18 +53,18 @@ func main() {
return
}
// Wait for interrupt in a go routine and close the client.
// Wait for interrupt and close the client.
go func() {
sig := make(chan os.Signal)
sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt)
<-sig
fmt.Println("Got interrupt")
client.Close()
_ = client.Close()
}()
// Read messages
// Read messages
for t := range client.Channel() {
switch msg := t.(type) {
case error:
@ -69,7 +73,10 @@ func main() {
fmt.Println("Transaction", msg.BlockNum, msg.ID)
fmt.Println(msg)
fmt.Println("---")
case message.HeartBeat :
case message.ActionTrace:
fmt.Println("Action", msg.Contract, msg.Name)
fmt.Println("---")
case message.HeartBeat:
fmt.Println("Heartbeat")
fmt.Println(msg)
fmt.Println("---")
@ -78,9 +85,9 @@ func main() {
}
```
## Nodejs
## Node.js
install the thalos client library:
Install the thalos client library:
`npm install --save @eosswedenorg/thalos-client`
@ -129,9 +136,9 @@ Transfer From/To: ponyslaystat -> wombatmaster Assets: [ '1099880264627' ]
## Python
This example will listen for new atomicasset transfers and print them to standard output.
This example listens for new `atomicassets::logtransfer` actions and prints them to standard output.
You can specify multiple channels to listen to by adding them to the `redis_channels` list.
Before you start this script, make sure you have the redis-server running.
Before you start this script, make sure Redis is running.
::: info NOTE
You need to have the redis-py library installed for this to work
@ -189,4 +196,4 @@ stdbuf -oL redis-cli --raw SUBSCRIBE ${CHANNEL} | while read RCMD; do
echo "Transfer From/To:" $(echo $MSG | jq -r ".data.from, \"->\", .data.to, \"Assets:\", .data.asset_ids, .data.memo")
fi
done
```
```

244
docs/api/messages.md
View file

@ -1,107 +1,137 @@
# Messages
This document describes the different messages that are sent
## Encoding
All messages are encoded in `json` format
## Types
### HeartBeat
Heartbeat messages are posted to the heartbeat channel periodically.
| Field | Datatype | Description |
| -------------------------- | -------- | ------------------------------------------- |
| blocknum | `int` | Current block number |
| head_blocknum | `int` | Head block number |
| last_irreversible_blocknum | `int` | block number of the last irreversible block |
### Transaction
### ActionTrace
| Field | Datatype | Description |
| -------------- | --------------------------------------- | ----------------------------------------------------------------- |
| tx_id | `string` | Transaction ID |
| blocknum | `int` | Block number where this action trace (and transaction) belongs to |
| blocktimestamp | `time` | Block timestamp |
| receipt | [`ActionReceipt`](#actionreceipt) | Action receipt |
| receiver | `string` | Receiver account |
| first_receiver | `bool` | True if receiver is the first account to get notified |
| contract | `string` | Contract account |
| action | `string` | What action was executed on the contract |
| data | `any` | Contract specific data (decoded using the contracts abi) |
| authorization | [`PermissionLevel[]`](#permissionlevel) | Authorization |
### ActionReceipt
| Field | Datatype | Description |
| --------------- | ----------------------------------------------- | ------------------ |
| receiver | `string` | Actor account name |
| act_digest | `string` | Action digest |
| global_sequence | `int` | Global sequence |
| recv_sequence | `int` | Receive sequence |
| auth_sequence | [`AccountAuthSequence[]`](#accountauthsequence) | Auth sequence |
| code_sequence | `int` | Code sequence |
| abi_sequence | `int` | ABI sequence |
### PermissionLevel
| Field | Datatype | Description |
| ---------- | -------- | -------------------------------- |
| actor | `string` | Actor account name |
| permission | `string` | Permission (for example: active) |
### AccountAuthSequence
| Field | Datatype | Description |
| -------- | -------- | ------------ |
| account | `string` | Account name |
| sequence | `int` | Sequence |
### RollbackMessage
| Field | Datatype | Description |
| --------- | -------- | ------------------------------------ |
| new_block | `int` | The current block number |
| old_block | `int` | Last block number that was received. |
### TableDelta
| Field | Datatype | Description |
| -------------- | ----------------------------------- | ----------------- |
| blocknum | `int` | Block number |
| blocktimestamp | `time` | Block timestamp |
| name | `string` | Table name |
| rows | [`TableDeltaRow[]`](#tabledeltarow) | Rows in the delta |
### TableDeltaRow
| Field | Datatype | Description |
| -------- | ---------------------------------------------------------- | --------------------------------- |
| present | `bool` | - |
| data | [`TableDeltaRowContract`](#tabledeltarowcontract) \| `any` | Decoded data |
| raw_data | `base64_string` | Raw data in base64 encoded string |
### TableDeltaRowContract
These messages are only relevant when listening to [Table deltas](redis-channels#table-delta) channel when name is `contract_row`.
| Field | Datatyp | Description |
| ----------- | ------------------------ | ---------------------------------------------------------- |
| table | `string` | table name |
| scope | `string` | table scope |
| primary_key | `string` \| `int` | Primary ID for the row. |
| code | `string` | Account name where that has the contract code |
| payer | `string` | Account name that payed for the ram that this row consumes |
| value | `map` \| `base64_string` | Actual data in the table |
Thalos can decode contract row data in `vaule` using the contracts abi.
If Thalos could not decode the data then `value` will contain a `base64_string` of the raw value instead.
# Messages
This document describes the message payloads published by Thalos.
## Encoding
Thalos supports two codecs (configured via `message_codec` / `--codec`):
- `json` (default)
- `msgpack`
When using JSON:
- `[]byte` fields are base64-encoded strings
- timestamps are UTC and serialized by the JSON codec
## Message types
### HeartBeat
Published on `heartbeat` channel periodically.
| Field | Type | Description |
| -------------------------- | -------- | ------------------------------------------- |
| `blocknum` | `uint32` | Current block number |
| `head_blocknum` | `uint32` | Current chain head block number |
| `last_irreversible_blocknum` | `uint32` | Last irreversible block number |
### TransactionTrace
Published on `transactions` channel (unless disabled).
| Field | Type | Description |
| ---------------- | --------------------------------- | ----------- |
| `id` | `string` | Transaction id |
| `blocknum` | `uint32` | Block number |
| `blocktimestamp` | `time` | Block timestamp |
| `status` | `string` | Transaction status |
| `cpu_usage_us` | `uint32` | CPU usage (microseconds) |
| `net_usage_words` | `uint32` | NET usage words |
| `elapsed` | `int64` | Execution time |
| `net_usage` | `uint64` | NET usage |
| `scheduled` | `bool` | Scheduled transaction flag |
| `action_traces` | [`ActionTrace[]`](#actiontrace) | Action traces for this transaction |
| `except` | `string` | Exception text (if any) |
| `error` | `uint64` | Error code |
### ActionTrace
Published on `actions` channels (unless disabled), and also nested in `TransactionTrace.action_traces`.
| Field | Type | Description |
| ---------------- | ------------------------------------------ | ----------- |
| `tx_id` | `string` | Parent transaction id |
| `blocknum` | `uint32` | Block number |
| `blocktimestamp` | `time` | Block timestamp |
| `receipt` | [`ActionReceipt`](#actionreceipt) / `null` | Action receipt |
| `name` | `string` | Action name |
| `contract` | `string` | Contract account |
| `receiver` | `string` | Receiver account |
| `first_receiver` | `bool` | True if receiver is first receiver |
| `data` | `any` | ABI-decoded action data when available |
| `authorization` | [`PermissionLevel[]`](#permissionlevel) | Authorization list |
| `except` | `string` | Exception text (if any) |
| `error` | `uint64` | Error code |
| `return` | `bytes` | Return value bytes |
> Note: the field is `name` (not `action`).
### ActionReceipt
| Field | Type | Description |
| ----------------- | ------------------------------------------------- | ----------- |
| `receiver` | `string` | Receiver account |
| `act_digest` | `string` | Action digest |
| `global_sequence` | `uint64` | Global sequence |
| `recv_sequence` | `uint64` | Receiver sequence |
| `auth_sequence` | [`AccountAuthSequence[]`](#accountauthsequence) | Auth sequence list |
| `code_sequence` | `uint32` | Code sequence |
| `abi_sequence` | `uint32` | ABI sequence |
### PermissionLevel
| Field | Type | Description |
| ------------ | -------- | ----------- |
| `actor` | `string` | Actor account |
| `permission` | `string` | Permission name |
### AccountAuthSequence
| Field | Type | Description |
| ---------- | -------- | ----------- |
| `account` | `string` | Account name |
| `sequence` | `uint64` | Sequence |
### RollbackMessage
Published on `rollback` channel on microfork rollback detection.
| Field | Type | Description |
| ----------- | -------- | ----------- |
| `old_block` | `uint32` | Previous current block |
| `new_block` | `uint32` | New current block |
### TableDelta
Published on `tabledeltas` channels (unless disabled).
| Field | Type | Description |
| ---------------- | ----------------------------------- | ----------- |
| `blocknum` | `uint32` | Block number |
| `blocktimestamp` | `time` | Block timestamp |
| `name` | `string` | SHIP delta name |
| `rows` | [`TableDeltaRow[]`](#tabledeltarow) | Rows in the delta |
### TableDeltaRow
| Field | Type | Description |
| ---------- | --------- | ----------- |
| `present` | `bool` | Row present/removed flag |
| `data` | `object` | Parsed row data when available |
| `raw_data` | `bytes` | Raw row bytes |
### TableDeltaRowContract (`name = contract_row`)
For `contract_row` deltas, `data` typically contains:
| Field | Type | Description |
| ------------- | ------------------------ | ----------- |
| `table` | `string` | Table name |
| `scope` | `string` | Table scope |
| `primary_key` | `string \| int` | Primary key |
| `code` | `string` | Contract account |
| `payer` | `string` | RAM payer |
| `value` | `object \| base64_string` | Decoded row value when ABI is available; otherwise raw bytes |
If ABI decode fails, Thalos keeps raw data and may leave decoded fields partial.

138
docs/api/redis-channels.md
View file

@ -1,65 +1,73 @@
# Redis channels
This document describes the redis channels used by thalos to deliver messages.
## Namespace
First. all channels have a namespace attached to them. this is done to prevent other application to clash with the keys.
The namespace have the following format: `<prefix>::<chain_id>`
* `prefix` is per default `ship` but can be configured to be something else.
* `chain_id` is the chain's id and is used to separate transactions if multiple chains are setup in the same redis database.
## Transactions
All transactions are posted to the following channel:
`<namespace>::transactions`
## Actions
there is 4 types of channels for actions.
The channel where all actions are posted is:
`<namespace>::actions`
Channel where only specific actions are posted:
`<namespace>::actions/name/<action>`
Channel where only actions on a specific `<contract>` is posted:
`<namespace>::actions/contract/<contract>`
Channel where only `<action>` on a specific `<contract>` is posted:
`<namespace>::actions/contract/<contract>/name/<action>`
## Rollback
Rollback mesages are posted to this channel.
`<namespace>::rollback`
## Table delta
Table deltas are posted to the following channels
All deltas:
`<namespace>::tabledeltas`
Only deltas for a specific type.
`<namespace>::tabledeltas/name/<name>`
`<name>` can be one of
* `account_metadata`
* `contract_table`
* `contract_row`
* `contract_index64`
* `resource_usage`
* `resource_limits_state`
# Redis channels
This document describes the Redis Pub/Sub channels used by Thalos.
## Channel key format
All published channels are namespaced to avoid collisions:
`<prefix>::<chain_id>::<channel-path>`
- `prefix`: defaults to `ship` (config: `redis.prefix`)
- `chain_id`: chain id from API by default, or overridden with `ship.chain`
- `channel-path`: one of the paths below
If no chain id is available, Thalos uses an all-zero fallback chain id internally.
## Channel paths
### Heartbeat
`<namespace>::heartbeat`
Published periodically (every 10 blocks) so subscribers can detect liveness.
### Transactions
`<namespace>::transactions`
Contains `TransactionTrace` messages.
### Actions
Thalos publishes each action trace to up to 4 channels:
- All actions:
`<namespace>::actions`
- By action name:
`<namespace>::actions/name/<action>`
- By contract:
`<namespace>::actions/contract/<contract>`
- By contract + action:
`<namespace>::actions/contract/<contract>/name/<action>`
### Rollback
`<namespace>::rollback`
Contains `RollbackMessage` messages when microfork rollbacks are detected.
### Table deltas
- All table deltas:
`<namespace>::tabledeltas`
- Table deltas by SHIP delta name:
`<namespace>::tabledeltas/name/<name>`
`<name>` is the table delta name received from SHIP (for example `contract_row`,
`contract_table`, `resource_usage`, `resource_limits_state`).
## Notes
- `transactions`, `actions`, and `tabledeltas` output can be enabled/disabled via config/flags.
- If `ship.table_delta_whitelist` is configured, only matching `contract_row` deltas are published.

209
docs/configuration.md
View file

@ -1,61 +1,92 @@
# Configuration
The configuration file is located at `config.yml` in the installed directory and contains
an example configuration with extensive documentation.
Below are the essential fields that you need to modify.
You can adjust the settings according to your preferences.
```yaml
name: MyShipReader
api: "http://api.example.com:8888"
ship:
url: "ws://ship.example.com:8080"
```
There is also the option to configure Thalos via command line flags.
See [this](running-the-server#cli-flags) section for more inforation
### General
`name` ([`string`](#string)) - Name of the talos node.
`api` ([`string`](#string)) - Nodeos API Endpoint.
`message_codec` ([`string`](#string)) default: `json`
What codec thalos should use when pushing out messages to redis.
### Ship
`ship.url` ([`string`](#string)) - Url to the nodeos node.
# Configuration
The default configuration file is `./config.yml`.
- For bundled/manual installs this is typically in your install directory.
- For Debian packages this is usually `/etc/thalos/config.yml`.
CLI flags override configuration values when provided.
```yaml
name: "my-thalos-node"
api: "http://127.0.0.1:8888"
message_codec: "json"
ship:
url: "ws://127.0.0.1:8080"
redis:
addr: "127.0.0.1:6379"
```
There is also the option to configure Thalos via command line flags.
See [CLI Flags](/docs/running-the-server#cli-flags).
### General
`name` ([`string`](#string)) - Name of the Thalos node.
`api` ([`string`](#string)) - Nodeos API Endpoint.
`message_codec` ([`string`](#string)) default: `json`
What codec Thalos should use when publishing messages to Redis.
### ABI Cache
`abi_cache.api_timeout` ([`duration_string`](#duration-string)) default: `1s`
Timeout used when fetching ABI definitions from the API.
### Ship
`ship.url` ([`string`](#string)) - URL to the nodeos SHIP endpoint.
`ship.chain` ([`string`](#string))
name of the chain, if not defined thalos will use the chain id reported from `api`.
This value (or chain_id if undefined) is prepended to channel names, to allow for multiple thalos instances
to run
Name of the chain namespace. If not defined, Thalos uses the chain id reported by `api`.
This value (or the chain id) is prepended to channel names to allow multiple Thalos instances
to run against the same Redis database.
`ship.irreversible_only` ([`boolean`](#boolean))
If `true`, the ship node will only send transactions once they are considered irreversible.
If `false` the transactions are posted as soon as possible.
`ship.table_deltas` ([`boolean`](#boolean))
If set to `true` thalos will publish table deltas updates to clients.
`ship.start_block_num` ([`integer`](#integer))
Thalos will start streaming blocks starting from this one. if undefined, the currect block reported by `api` is used.
`ship.irreversible_only` ([`boolean`](#boolean))
If `true`, the ship node will only send transactions once they are considered irreversible.
If `false` the transactions are posted as soon as possible.
`ship.max_messages_in_flight` ([`integer`](#integer)) default: `10`
Maximum number of SHIP messages in flight before acknowledgement.
`ship.table_deltas` ([`boolean`](#boolean))
If set to `true`, Thalos publishes table delta updates to clients.
`ship.table_delta_whitelist` ([`ContractList`](#contractlist))
Optional whitelist for table deltas (`contract:table`).
Use `*` as wildcard for table names.
`ship.transactions` ([`boolean`](#boolean)) default: `true`
If `false`, Thalos skips transaction trace messages.
`ship.actions` ([`boolean`](#boolean)) default: `true`
If `false`, Thalos skips action trace messages.
`ship.start_block_num` ([`integer`](#integer))
Thalos will start streaming blocks starting from this one.
If undefined, Thalos will use cached state when available, otherwise API head/LIB depending on mode.
`ship.end_block_num` ([`integer`](#integer))
Thalos will stop streaming when the block number defined by this value will be reached. if undefined thalos will never stop.
Thalos will stop streaming when this block number is reached.
If undefined, Thalos will keep streaming.
`ship.blacklist` ([`ContractList`](#contractlist))
List of contract,actions pairs that Thalos will not process if encountered.
List of `contract:action` pairs that Thalos will skip.
`ship.blacklist_is_whitelist` ([`boolean`](#boolean))
@ -71,49 +102,49 @@ Thalos will treat `ship.blacklist` as a whitelist.
`redis.db` ([`integer`](#integer)) - Database index to use
`redis.prefix` ([`string`](#string)) - Key prefix, this will be prepended to all channels that thalos is using to avoid name collision
`redis.prefix` ([`string`](#string)) - Key prefix prepended to all Thalos channels to avoid name collisions.
### Cache
These settings control the cache used by Thalos to cache abi definitions and internal state.
`cache.storage` ([`string`](#string) default: `redis`) - Cache storage to use.
Avaible values are: `memory`, `redis`
`cache.options` (`map`) - Options for the cache storage, see section for each cache storage.
`cache.storage` ([`string`](#string) default: `redis`) - Cache storage to use.
Available values: `memory`, `redis`
`cache.options` (`map`) - Options for the cache storage, see section for each cache storage.
#### Memory
No configuration.
#### Redis
`cache.options.stats` ([`boolean`](#string) default: `false`) - True if statistics should be collected
`cache.options.size` ([`number`](#string) default: `1000`) - How many items to store in process local memory for faster lookup of popular items
`cache.options.ttl` ([`number`](#string) default: `10`) - How long (in minutes) each item should be kept in process local memory before being discared (and has to be fetched from redis again.)
### Logging
This block configures how thalos will log information.
`log.filename` ([`string`](#string)) - Filename (without extension)
`log.directory` ([`string`](#string)) - Directory where to store log files.
`log.timeformat` ([`dateformat`](#dateformat)) - Format to rename log files when rotating
`log.max_filesize` ([`integer`](#integer)) - Rotate when the file reaches this size.
`log.max_time` ([`duration_string`](#duration-string)) - Rotate when the file is this old.
#### Redis
`cache.options.stats` ([`boolean`](#boolean) default: `false`) - True if statistics should be collected
`cache.options.size` ([`integer`](#integer) default: `1000`) - How many items to store in process local memory for faster lookup of popular items
`cache.options.ttl` ([`integer`](#integer) default: `10`) - How long (in minutes) each item should be kept in process local memory before being discarded (and has to be fetched from redis again.)
### Logging
This block configures how Thalos logs information.
`log.filename` ([`string`](#string)) - Filename (without extension)
`log.directory` ([`string`](#string)) - Directory where to store log files.
`log.file_timestamp_format` ([`dateformat`](#dateformat)) - Format to rename log files when rotating
`log.maxfilesize` ([`string`](#string)) - Rotate when the file reaches this size (for example `10mb`, `200mb`).
`log.maxtime` ([`duration_string`](#duration-string)) - Rotate when the file is this old.
### Telegram notifications
These settings control the telegram notification mechanism built into thalos.
if undefined thalos will simple skip this.
If they are defined, thalos will send messages when important events happen.
These settings control the Telegram notification mechanism built into Thalos.
If undefined, Thalos skips Telegram notifications.
If defined, Thalos sends messages when important events happen.
`telegram.id` ([`string`](#string)) - bot id
@ -136,10 +167,11 @@ Boolean value (`true` or `false`)
### dateformat
The dateformat type is a `string` but describes a date format in go.
The dateformat type is a `string` but describes a date format in Go.
Go uses a different approach then other languages. usually you can format a date like `2023-03-22` like
`YYYY-MM-DD`. Go however uses a **reference time** when parsing time. that is just a single point in time.
Go uses a different approach than many other languages.
Usually you might format a date like `2023-03-22` as `YYYY-MM-DD`.
Go uses a **reference time** when parsing time:
Mon Jan 2 15:04:05 MST 2006
@ -151,7 +183,7 @@ Go uses a different approach then other languages. usually you can format a date
| `Jan` | Three letter word of month | Oct, Jun |
| `January` | Full name of month | April, September |
| `2` | Day in month without leading zero | 1 to 31 |
| `02`,`_2` | Day in moth with leading zero | 01 to 31 |
| `02`,`_2` | Day in month with leading zero | 01 to 31 |
| `002`,`__2` | Day in year | 002 |
| `Mon` | Three letter word for day in week | Tue, Fri |
| `Monday` | Full name of day in week | Friday, Saturday |
@ -178,9 +210,12 @@ Valid time units are `"ns"`, `"us"` (or `"µs"`), `"ms"`, `"s"`, `"m"`, `"h"`.
### ContractList
A array of contract and actions pairs.
Keys (contracts) are of type [`string`](#string) and values are an array of [`string`](#string) (one or more actions).
An array/map of `contract:value` pairs.
Keys (contracts) are of type [`string`](#string) and values are an array of [`string`](#string).
For `ship.blacklist` the values are action names.
For `ship.table_delta_whitelist` the values are table names.
Action array can hold a special string `*` that matches any action.

4
docs/installation/debian.md
View file

@ -10,7 +10,7 @@ The following documentation assumes that you have already set up a `redis` serve
## 1. Installing the package
### Using Sw/edens apt repository
### Using EOSSweden apt repository
First, obtain the key.
@ -52,4 +52,4 @@ See [Configuration](/docs/configuration) for more information
## 3. Starting the Server
There are serveral ways to run the server. the most common on debian is via [systemd](/docs/running-the-server#with-systemd) or [manually](/docs/running-the-server#with-systemd)
There are several ways to run the server. The most common on Debian is via [systemd](/docs/running-the-server#with-systemd) or [manually](/docs/running-the-server#starting-manually).

14
docs/installation/docker.md
View file

@ -4,7 +4,7 @@
# Docker
Do you want to run Thalos in a container? No problem as Thalos provides official docker images
Do you want to run Thalos in a container? No problem as Thalos provides official Docker images
to get you going in no time.
The images can be found [here](https://github.com/eosswedenorg/thalos/pkgs/container/thalos)
@ -19,12 +19,12 @@ Check that it works by spawning the container and display the version:
```sh
$ docker run ghcr.io/eosswedenorg/thalos:latest --version
thalos-server v1.1.1
thalos-server v<version>
```
## Using docker compose
## Using Docker Compose
Use `docker-compose` to spin up a SHIP node, Redis and Thalos all at once.
Use Docker Compose to spin up a SHIP node, Redis, and Thalos all at once.
```yaml
services:
@ -33,11 +33,11 @@ services:
depends_on:
- redis
- ship
# pass arguments to thalos.
# Pass arguments to thalos-server.
command: [
"--redis-addr=redis:6379",
"--url=http://ship:80",
"--ship-url=http://ship:8080"
"--url=http://ship:8888",
"--ship-url=ws://ship:8080"
]
# or mount a config file
#volumes:

12
docs/installation/index.md
View file

@ -7,7 +7,7 @@ next:
# Installation
There are several ways to install thalos, via package manager, downloading a pre-built binary or building directly from source.
There are several ways to install Thalos: package manager, pre-built binaries, or building from source.
## Package Managers
@ -31,7 +31,7 @@ Using curl command above, the files are extracted into the `thalos` subdirectory
### Compiling from source
You will need golang version `1.26` or later to compile the source.
You will need Go `1.26` or later to compile the source.
#### Compile using make
@ -39,11 +39,11 @@ You will need golang version `1.26` or later to compile the source.
make
```
or using go directly if you dont have make installed.
Or using `go` directly if you don't have `make` installed:
```shell
go build -o build/thalos-server cmd/thalos/main.go
```
```shell
go build -o build/thalos-server ./cmd/thalos/
```
#### Install

139
docs/redis/security/acl.md
View file

@ -1,116 +1,91 @@
# ACL
By default, Redis does not impose any restrictions on the commands that clients can execute. This lack of
restrictions raises security concerns, especially when dealing with clients that may not be fully trusted.
However, there is no need to worry as Redis offers a solution in the form of [Access Control Lists (ACL)](https://en.wikipedia.org/wiki/Access-control_list).
Redis ACLs let you control **which commands** and **which channels/keys** each user can access.
For Thalos, this is the main way to safely separate:
ACL in Redis enables the establishment of limited access for connections to the server, specifying which
commands they are allowed to execute. In Thalos, this feature is crucial for maintaining security, ensuring
that clients can only access specific channels within Thalos.
- subscriber access (read from Thalos channels)
- server access (publish messages + state/cache reads/writes)
The special account called `default` serves as the default account for unauthorized users, provided it is
configured with a password. Connections can authenticate against this account without specifying a username.
Thalos utilizes this account as the default user account.
## Recommended user model
Additionally, it is advisable to restrict the Thalos server account as an added precaution against any
unauthorized actions it may inadvertently perform, although such occurrences are highly unlikely.
For a simple setup, use two users:
The ACL in thalos is simple and uses 2 accounts:
- `default` (or `thalos-client`) for subscribers
- `thalos` for thalos-server
* `default` account (user account, used by clients. only allowed to read from thalos specific channels)
* `thalos-server` account (is allowed to publish to channels and also write to it's cache)
there is also the `admin` account that is use for redis mangement. has access to everything.
::: danger IMPORTANT
Make sure you replace the passwords with a secure ones.
It is recommended to use [ACL GENPASS](https://redis.io/commands/acl-genpass) to generate strong passwords.
::: warning IMPORTANT
Use strong passwords. `ACL GENPASS` is recommended:
<https://redis.io/docs/latest/commands/acl-genpass/>
:::
`redis.conf`
## Example ACL rules
```
user default on >client_password resetchannels &ship::* +@connection +subscribe
user admin on >admin_password ~* &* +@all
user thalos on >server_password resetchannels &ship::* ~thalos::* +@connection +@read +@write +publish
Update `ship::*` if you use a different `redis.prefix`.
```conf
user default on >client_password resetkeys resetchannels &ship::* -@all +@connection +subscribe +psubscribe +unsubscribe +punsubscribe
user thalos on >server_password resetkeys resetchannels ~ship::* &ship::* -@all +@connection +get +set +publish
```
## External file
### Why both `~...` and `&...`?
It is possible to use external config files to define users in redis.
- `~pattern` controls key access
- `&pattern` controls Pub/Sub channel access
Just place the configuration above in an external file for example: `/etc/redis/users.acl` and then add this in `/etc/redis/redis.conf`
They are separate controls in Redis ACL.
```
## External ACL file
Store ACL rules in a dedicated file, then reference it from `redis.conf`:
```conf
aclfile /etc/redis/users.acl
```
## Thalos tools
## Thalos tooling
There is also a tool to create the config lines for you.
```
$ thalos-tools redis-acl
Passwords
default: NgCseRB8QcxdHwcwmg43OOjxZW0YbngN
thalos: kmJIi1kP5bp2MHQeSj925otWnxTvfRe5
thalos-client: WBGP9mXuUIIv11KhLgbuaTcl1NnRmhWn
# Created by thalos-tools on Mon Feb 12 15:56:26 CET 2024
user default on #3c0745d926021292a521267d803b9c20f3570bee30697e401920151d2593d1e4 ~* &* +@all
user thalos on #cf2b3e36521b0e67cb50fcc1f931f577384bb84ad7d0492d00e07beb4660ff1d resetchannels ~ship::* &ship::* -@all +get +publish +set
user thalos-client on #06d88f1e91d8bc82a5074223fd380006395b80a7b41dcd1318be1dd288282e4a resetchannels &ship::* -@all +subscribe
```
## Test the accounts permissions
It is a good idea to test if the configuration works.
Thalos includes an ACL helper:
```sh
$ redis-cli --pass mypassword
127.0.0.1:6379> SUBSCRIBE some_channel
Reading messages... (press Ctrl-C to quit)
(error) NOPERM this user has no permissions to access one of the channels used as arguments
127.0.0.1:6379> SET random_key value
(error) NOPERM this user has no permissions to run the 'set' command or its subcommand
127.0.0.1:6379> SUBSCRIBE ship::1234
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "ship::1234"
3) (integer) 1
thalos-tools redis-acl
```
You can customize generated users/prefix with flags such as:
- `--server`
- `--client`
- `--prefix`
- `--cleartext`
Review generated rules before applying them in production.
## Verify permissions
Test that subscriber credentials can only access allowed channels:
```sh
redis-cli --user default --pass client_password
```
Then try:
- `SUBSCRIBE ship::1234::actions` ✅
- `SUBSCRIBE some_other_prefix::1234::actions` ❌ (`NOPERM` expected)
- `SET random_key value` ❌ (`NOPERM` expected)
## Thalos config
After you have setup ACL, make sure to update your `config.yml` with the account and password.
After ACL setup, configure thalos-server with the server user:
```yaml
redis:
user: thalos
password: p4ssw0rd
```
## Other ACL Configurations
### No user password
While not recommended, it is possible to have the default (user) account without password. that way the user does not need to authenticate (but still has limited access). just remove the password from `user default` line:
```
user default on resetchannels &ship::* +@connection +subscribe
```
### Thalos users with a different account
It is also possible to provide a different account for users
```
user thalos-client on >client_password resetchannels &ship::* +@connection +subscribe
password: server_password
```
## Useful links
* [Config File Example](https://redis.io/docs/management/config-file)
* [Official ACL Documentation](https://redis.io/docs/management/security/acl)
- [Redis ACL documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/)
- [ACL SETUSER command](https://redis.io/docs/latest/commands/acl-setuser/)
- [ACL CAT command](https://redis.io/docs/latest/commands/acl-cat/)
- [Redis Pub/Sub](https://redis.io/docs/latest/develop/pubsub/)

66
docs/redis/security/index.md
View file

@ -1,51 +1,49 @@
# Securing Redis
# Securing redis
This guide focuses on Redis deployments where other systems or users can connect
to the same Redis server as Thalos.
This documentation primarily focuses on setups where Redis is exposed to the internet or an internal
network where there is not complete control over the clients.
For example, you may want to grant access to your Thalos instance to a friend.
While trusting your friend is reasonable, it is essential to consider potential future scenarios where
trust may no longer exist or their server could be compromised.
## Baseline recommendations
If you intend to run Thalos for internal use only, such as having internal applications that are relying on a blockchain stream,
it is perfectly acceptable to skip these steps if you have complete control over all involved
servers and do not expose the instance over a public IP.
1. Do not expose Redis directly to the public internet.
2. Restrict network access (bind + firewall/security groups).
3. Use ACL users with least privilege (see [ACL](/docs/redis/security/acl)).
4. Use TLS when traffic crosses untrusted networks.
## Isolating redis
## Isolating Redis
To ensure security, it is highly recommended to run Thalos on a dedicated Redis instance, ideally within a
container or virtual machine.
This isolation helps prevent data leaks in case of misconfigured Redis ACLs or unauthorized access due to
leaked/guessed admin password.
Running Thalos on a dedicated Redis instance (or dedicated DB/service) reduces
blast radius if credentials leak or ACL rules are misconfigured.
Additionally, it safeguards against potential misconfigurations, such as other applications mistakenly
writing sensitive data to Redis channels that can be accessed by Thalos clients.
It also lowers the risk of other applications accidentally publishing sensitive
data to channels your subscribers can read.
In summary, isolating Thalos in its own Redis instance provides an extra layer of safety.
## Network and protected mode
## Network
Use `bind` in `redis.conf` to restrict interfaces:
The `bind` directive in `redis.conf` is used to tell redis what network interfaces it should bind to.
Make sure to update this with the interfaces you intend to use.
```conf
bind 127.0.0.1 ::1 # loopback only
bind 192.168.1.100 10.0.0.1 # specific interfaces
```
::: danger IMPORTANT
Although it is recommended to limit Redis to the localhost interface for security purposes, with proper
firewall and ACL configurations, it can be safely exposed to additional interfaces. Carefully evaluate the necessity of
external access before making the change in the config file.
::: warning
`protected-mode` is a safety fallback, not a full security model.
For production, still enforce strict network controls and ACLs.
:::
```
bind 192.168.1.100 10.0.0.1 # listens on two specific IPv4 addresses
bind 127.0.0.1 ::1 # listens on loopback IPv4 and IPv6
bind * -::* # like the default, all available interfaces
```
## Firewall
Make sure you setup your firewall rules correctly. Only allowing the IP addresses you trust to access the Redis port.
This is out of scope of this documentation. Consult your operating system or router manuals.
Allow Redis port access only from trusted hosts/subnets.
Firewall setup is environment-specific, so refer to your platform docs.
## TLS
Redis supports native TLS (Redis 6+).
When using TLS-only deployments, disable plaintext (`port 0`) and use `tls-port`.
## Useful links
* [Official Security Documentation](https://redis.io/docs/management/security)
* [Config File Example](https://redis.io/docs/management/config-file)
- [Redis security overview](https://redis.io/docs/latest/operate/oss_and_stack/management/security/)
- [Redis ACL documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/)
- [Redis TLS documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/encryption/)

89
docs/running-the-server.md
View file

@ -24,7 +24,7 @@ If desired, Thalos can also be started manually for quick configuration testing.
./bin/thalos-server
```
or if you want to specify another config file then the default
Or, if you want to specify a different config file than the default:
```shell
./bin/thalos-server --config /path/to/thalos.yml
@ -38,51 +38,64 @@ sudo systemctl start thalos-server
```
After executing these commands, the server should be up and running.
You can check the logs at `/var/log/thalos.log` (unless specified otherwise in the configuration), or by running `sudo systemctl status thalos-server`.
You can inspect logs with `sudo journalctl -u thalos-server -f`,
or check the directory configured in `log.directory` (commonly `/var/log/thalos` on Debian packages).
> TIP: if you want to modify the arguments passed to thalos-server when starting
via systemd, look in the `/etc/sysconfig/thalos-server` file
## CLI Flags
Here is the output of `thalos-server -h` showing the available cli flags:
Run `thalos-server -h` to see the current flag list for your installed version.
```bash
thalos-server v1.1.4
The current source supports these flags:
Usage:
thalos-server [flags]
### Core
Flags:
--abi-cache-api-timeout duration Duration before the api call times out when the ABI cache requests an abi. (default 1s)
--blacklist strings Define a list of 'contract:action' pairs that will be blacklisted (thalos will not process those actions)
--blacklist-is-whitelist Thalos will treat the blacklist as a whitelist
--cache string What cache driver to use (default "redis")
--chain string ChainID used in channel namespace, can be any string (default from api)
--codec string Codec used to send messages (default "json")
-c, --config string Config file to read (default "./config.yml")
--end-block uint32 Stop streaming when this block is reached (default none)
-h, --help help for thalos-server
--irreversible-only Only stream irreversible blocks from ship
-L, --level string Log level to use (default "info")
-l, --log string Path to log file (default: print to stdout/stderr)
--log-file-timestamp string Timestamp format to use when rotating log files (default "2006-01-02_150405")
--log-max-filesize string Max filesize for logfile to rotate (default "10mb")
--log-max-time duration Max time for logfile to rotate (default 24h0m0s)
--max-msg-in-flight int Maximum messages that can be sent from SHIP without acknowledgement (default 10)
-n, --no-state-cache Force the application to take start block from config/api
-p, --pid string Where to write process id
--redis-addr string host:port to redis server (default "127.0.0.1:6379")
--redis-db int Redis database
--redis-password string Redis password
--redis-prefix string Redis channel prefix (default "ship")
--redis-user string Redis username
--ship-url string Url to ship node (default "ws://127.0.0.1:8080")
--start-block uint32 Start to stream from this block (default config value, cache, head from api)
--telegram-channel int Telegram channel to send notifications to
--telegram-id string Id of telegram bot
-u, --url string Url to antelope api
-v, --version version for thalos-server
```
- `-c, --config string` - Config file to read (default `./config.yml`)
- `-u, --url string` - Antelope API URL
- `--codec string` - Message codec (default `json`)
- `-L, --level string` - Log level (default `info`)
- `-p, --pid string` - Write process id to file
- `-n, --no-state-cache` - Ignore cached state and use config/API start logic
- `-h, --help` - Show help
- `-v, --version` - Show version
### Redis / Cache / Notifications
- `--redis-addr string` - Redis `host:port` (default `127.0.0.1:6379`)
- `--redis-user string` - Redis username
- `--redis-password string` - Redis password
- `--redis-db int` - Redis database index
- `--redis-prefix string` - Redis channel prefix (default `ship`)
- `--cache string` - Cache driver (default `redis`)
- `--abi-cache-api-timeout duration` - ABI fetch timeout (default `1s`)
- `--telegram-id string` - Telegram bot id
- `--telegram-channel int` - Telegram channel id
### SHIP / Stream control
- `--ship-url string` - SHIP websocket URL (default `ws://127.0.0.1:8080`)
- `--start-block uint32` - Start stream from this block
- `--end-block uint32` - Stop stream at this block
- `--irreversible-only` - Stream irreversible blocks only
- `--max-msg-in-flight int` - Max outstanding SHIP messages (default `10`)
- `--chain string` - Chain namespace value (defaults to chain id from API)
### Filtering / Published message types
- `--blacklist strings` - Blacklist `contract:action` pairs
- `--blacklist-is-whitelist` - Treat blacklist as whitelist
- `--table-deltas` - Enable table delta processing (default `true`)
- `--table-delta-whitelist strings` - Whitelist table deltas as `contract:table` pairs (supports `*` wildcard)
- `--transactions` - Enable transaction trace messages (default `true`)
- `--actions` - Enable action trace messages (default `true`)
### Log rotation (when `--log` is used)
- `-l, --log string` - Log file path (default: stdout/stderr)
- `--log-max-filesize string` - Rotate file at size (default `10mb`)
- `--log-max-time duration` - Rotate file at age (default `24h`)
- `--log-file-timestamp string` - Rotation filename timestamp format (default `2006-01-02_150405`)
> Note: setting `--start-block` automatically enables `--no-state-cache`.