eosswedenorg/thalos-docs
1
0
Fork 0
mirror of https://github.com/eosswedenorg/thalos-docs synced 2026-10-14 03:46:44 +02:00
Code Releases Activity

docs: align Thalos docs with current config, CLI, and API

Browse source
This commit is contained in:
Henrik Hautakoski 2026-08-31 14:51:54 +02:00
parent 2d90392d97
commit 3bb8bb5c26
11 changed files with 516 additions and 450 deletions
Download patch file Download diff file Expand all files Collapse all files

16
docs/api/clients.md
View file

@ -1,8 +1,8 @@
# Clients # Clients
List of officially supported client libraries List of officially supported client libraries.
| Name | Language | | Name | Language |
| -------------------------------------------------------------------------- | -------- | | -------------------------------------------------------------------------- | -------- |
| [thalos-go](https://github.com/eosswedenorg/thalos/tree/master/api) | Golang | | [thalos-go](https://github.com/eosswedenorg/thalos/tree/master/api) | Go |
| [thalos-nodejs](https://www.npmjs.com/package/@eosswedenorg/thalos-client) | Nodejs | | [@eosswedenorg/thalos-client](https://www.npmjs.com/package/@eosswedenorg/thalos-client) | Node.js |

35
docs/api/examples.md
View file

@ -1,3 +1,7 @@
# Examples
These examples assume Thalos uses `message_codec: json` (default).
If you run with `message_codec: msgpack`, use a MessagePack decoder in your client.
## Go ## Go
@ -18,15 +22,15 @@ import (
) )
func main() { func main() {
// Create redis client // Create Redis client
rdb := redis.NewClient(&redis.Options{}) rdb := redis.NewClient(&redis.Options{})
sub := api_redis.NewSubscriber(context.Background(), rdb, api_redis.Namespace{ sub := api_redis.NewSubscriber(context.Background(), rdb, api_redis.Namespace{
Prefix: "ship", Prefix: "ship",
ChainID: "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4", // Wax mainnet. ChainID: "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4", // WAX mainnet
}) })
// Create client // Create client
codec, err := message.GetCodec("json") codec, err := message.GetCodec("json")
if err != nil { if err != nil {
fmt.Println("Failed to get json codec") fmt.Println("Failed to get json codec")
@ -35,7 +39,7 @@ func main() {
client := api.NewClient(sub, codec.Decoder) client := api.NewClient(sub, codec.Decoder)
// Subscribe to some channels. // Subscribe to some channels.
err = client.Subscribe( err = client.Subscribe(
api.TransactionChannel, api.TransactionChannel,
api.ActionChannel{Contract: "eosio"}.Channel(), api.ActionChannel{Contract: "eosio"}.Channel(),
@ -49,18 +53,18 @@ func main() {
return return
} }
// Wait for interrupt in a go routine and close the client. // Wait for interrupt and close the client.
go func() { go func() {
sig := make(chan os.Signal) sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt) signal.Notify(sig, os.Interrupt)
<-sig <-sig
fmt.Println("Got interrupt") fmt.Println("Got interrupt")
client.Close() _ = client.Close()
}() }()
// Read messages // Read messages
for t := range client.Channel() { for t := range client.Channel() {
switch msg := t.(type) { switch msg := t.(type) {
case error: case error:
@ -69,7 +73,10 @@ func main() {
fmt.Println("Transaction", msg.BlockNum, msg.ID) fmt.Println("Transaction", msg.BlockNum, msg.ID)
fmt.Println(msg) fmt.Println(msg)
fmt.Println("---") fmt.Println("---")
case message.HeartBeat : case message.ActionTrace:
fmt.Println("Action", msg.Contract, msg.Name)
fmt.Println("---")
case message.HeartBeat:
fmt.Println("Heartbeat") fmt.Println("Heartbeat")
fmt.Println(msg) fmt.Println(msg)
fmt.Println("---") fmt.Println("---")
@ -78,9 +85,9 @@ func main() {
} }
``` ```
## Nodejs ## Node.js
install the thalos client library: Install the thalos client library:
`npm install --save @eosswedenorg/thalos-client` `npm install --save @eosswedenorg/thalos-client`
@ -129,9 +136,9 @@ Transfer From/To: ponyslaystat -> wombatmaster Assets: [ '1099880264627' ]
## Python ## Python
This example will listen for new atomicasset transfers and print them to standard output. This example listens for new `atomicassets::logtransfer` actions and prints them to standard output.
You can specify multiple channels to listen to by adding them to the `redis_channels` list. You can specify multiple channels to listen to by adding them to the `redis_channels` list.
Before you start this script, make sure you have the redis-server running. Before you start this script, make sure Redis is running.
::: info NOTE ::: info NOTE
You need to have the redis-py library installed for this to work You need to have the redis-py library installed for this to work
@ -189,4 +196,4 @@ stdbuf -oL redis-cli --raw SUBSCRIBE ${CHANNEL} | while read RCMD; do
echo "Transfer From/To:" $(echo $MSG | jq -r ".data.from, \"->\", .data.to, \"Assets:\", .data.asset_ids, .data.memo") echo "Transfer From/To:" $(echo $MSG | jq -r ".data.from, \"->\", .data.to, \"Assets:\", .data.asset_ids, .data.memo")
fi fi
done done
``` ```

244
docs/api/messages.md
View file

@ -1,107 +1,137 @@
# Messages # Messages
This document describes the different messages that are sent This document describes the message payloads published by Thalos.
## Encoding ## Encoding
All messages are encoded in `json` format Thalos supports two codecs (configured via `message_codec` / `--codec`):
## Types - `json` (default)
- `msgpack`
### HeartBeat
When using JSON:
Heartbeat messages are posted to the heartbeat channel periodically.
- `[]byte` fields are base64-encoded strings
| Field | Datatype | Description | - timestamps are UTC and serialized by the JSON codec
| -------------------------- | -------- | ------------------------------------------- |
| blocknum | `int` | Current block number | ## Message types
| head_blocknum | `int` | Head block number |
| last_irreversible_blocknum | `int` | block number of the last irreversible block | ### HeartBeat
### Transaction Published on `heartbeat` channel periodically.
| Field | Type | Description |
### ActionTrace | -------------------------- | -------- | ------------------------------------------- |
| `blocknum` | `uint32` | Current block number |
| Field | Datatype | Description | | `head_blocknum` | `uint32` | Current chain head block number |
| -------------- | --------------------------------------- | ----------------------------------------------------------------- | | `last_irreversible_blocknum` | `uint32` | Last irreversible block number |
| tx_id | `string` | Transaction ID |
| blocknum | `int` | Block number where this action trace (and transaction) belongs to | ### TransactionTrace
| blocktimestamp | `time` | Block timestamp |
| receipt | [`ActionReceipt`](#actionreceipt) | Action receipt | Published on `transactions` channel (unless disabled).
| receiver | `string` | Receiver account |
| first_receiver | `bool` | True if receiver is the first account to get notified | | Field | Type | Description |
| contract | `string` | Contract account | | ---------------- | --------------------------------- | ----------- |
| action | `string` | What action was executed on the contract | | `id` | `string` | Transaction id |
| data | `any` | Contract specific data (decoded using the contracts abi) | | `blocknum` | `uint32` | Block number |
| authorization | [`PermissionLevel[]`](#permissionlevel) | Authorization | | `blocktimestamp` | `time` | Block timestamp |
| `status` | `string` | Transaction status |
### ActionReceipt | `cpu_usage_us` | `uint32` | CPU usage (microseconds) |
| `net_usage_words` | `uint32` | NET usage words |
| Field | Datatype | Description | | `elapsed` | `int64` | Execution time |
| --------------- | ----------------------------------------------- | ------------------ | | `net_usage` | `uint64` | NET usage |
| receiver | `string` | Actor account name | | `scheduled` | `bool` | Scheduled transaction flag |
| act_digest | `string` | Action digest | | `action_traces` | [`ActionTrace[]`](#actiontrace) | Action traces for this transaction |
| global_sequence | `int` | Global sequence | | `except` | `string` | Exception text (if any) |
| recv_sequence | `int` | Receive sequence | | `error` | `uint64` | Error code |
| auth_sequence | [`AccountAuthSequence[]`](#accountauthsequence) | Auth sequence |
| code_sequence | `int` | Code sequence | ### ActionTrace
| abi_sequence | `int` | ABI sequence |
Published on `actions` channels (unless disabled), and also nested in `TransactionTrace.action_traces`.
### PermissionLevel
| Field | Type | Description |
| Field | Datatype | Description | | ---------------- | ------------------------------------------ | ----------- |
| ---------- | -------- | -------------------------------- | | `tx_id` | `string` | Parent transaction id |
| actor | `string` | Actor account name | | `blocknum` | `uint32` | Block number |
| permission | `string` | Permission (for example: active) | | `blocktimestamp` | `time` | Block timestamp |
| `receipt` | [`ActionReceipt`](#actionreceipt) / `null` | Action receipt |
### AccountAuthSequence | `name` | `string` | Action name |
| `contract` | `string` | Contract account |
| Field | Datatype | Description | | `receiver` | `string` | Receiver account |
| -------- | -------- | ------------ | | `first_receiver` | `bool` | True if receiver is first receiver |
| account | `string` | Account name | | `data` | `any` | ABI-decoded action data when available |
| sequence | `int` | Sequence | | `authorization` | [`PermissionLevel[]`](#permissionlevel) | Authorization list |
| `except` | `string` | Exception text (if any) |
### RollbackMessage | `error` | `uint64` | Error code |
| `return` | `bytes` | Return value bytes |
| Field | Datatype | Description |
| --------- | -------- | ------------------------------------ | > Note: the field is `name` (not `action`).
| new_block | `int` | The current block number |
| old_block | `int` | Last block number that was received. | ### ActionReceipt
### TableDelta | Field | Type | Description |
| ----------------- | ------------------------------------------------- | ----------- |
| Field | Datatype | Description | | `receiver` | `string` | Receiver account |
| -------------- | ----------------------------------- | ----------------- | | `act_digest` | `string` | Action digest |
| blocknum | `int` | Block number | | `global_sequence` | `uint64` | Global sequence |
| blocktimestamp | `time` | Block timestamp | | `recv_sequence` | `uint64` | Receiver sequence |
| name | `string` | Table name | | `auth_sequence` | [`AccountAuthSequence[]`](#accountauthsequence) | Auth sequence list |
| rows | [`TableDeltaRow[]`](#tabledeltarow) | Rows in the delta | | `code_sequence` | `uint32` | Code sequence |
| `abi_sequence` | `uint32` | ABI sequence |
### TableDeltaRow ### PermissionLevel
| Field | Datatype | Description | | Field | Type | Description |
| -------- | ---------------------------------------------------------- | --------------------------------- | | ------------ | -------- | ----------- |
| present | `bool` | - | | `actor` | `string` | Actor account |
| data | [`TableDeltaRowContract`](#tabledeltarowcontract) \| `any` | Decoded data | | `permission` | `string` | Permission name |
| raw_data | `base64_string` | Raw data in base64 encoded string |
### AccountAuthSequence
### TableDeltaRowContract
| Field | Type | Description |
These messages are only relevant when listening to [Table deltas](redis-channels#table-delta) channel when name is `contract_row`. | ---------- | -------- | ----------- |
| `account` | `string` | Account name |
| `sequence` | `uint64` | Sequence |
| Field | Datatyp | Description | ### RollbackMessage
| ----------- | ------------------------ | ---------------------------------------------------------- |
| table | `string` | table name | Published on `rollback` channel on microfork rollback detection.
| scope | `string` | table scope |
| primary_key | `string` \| `int` | Primary ID for the row. | | Field | Type | Description |
| code | `string` | Account name where that has the contract code | | ----------- | -------- | ----------- |
| payer | `string` | Account name that payed for the ram that this row consumes | | `old_block` | `uint32` | Previous current block |
| value | `map` \| `base64_string` | Actual data in the table | | `new_block` | `uint32` | New current block |
### TableDelta
Thalos can decode contract row data in `vaule` using the contracts abi.
If Thalos could not decode the data then `value` will contain a `base64_string` of the raw value instead. Published on `tabledeltas` channels (unless disabled).
| Field | Type | Description |
| ---------------- | ----------------------------------- | ----------- |
| `blocknum` | `uint32` | Block number |
| `blocktimestamp` | `time` | Block timestamp |
| `name` | `string` | SHIP delta name |
| `rows` | [`TableDeltaRow[]`](#tabledeltarow) | Rows in the delta |
### TableDeltaRow
| Field | Type | Description |
| ---------- | --------- | ----------- |
| `present` | `bool` | Row present/removed flag |
| `data` | `object` | Parsed row data when available |
| `raw_data` | `bytes` | Raw row bytes |
### TableDeltaRowContract (`name = contract_row`)
For `contract_row` deltas, `data` typically contains:
| Field | Type | Description |
| ------------- | ------------------------ | ----------- |
| `table` | `string` | Table name |
| `scope` | `string` | Table scope |
| `primary_key` | `string \| int` | Primary key |
| `code` | `string` | Contract account |
| `payer` | `string` | RAM payer |
| `value` | `object \| base64_string` | Decoded row value when ABI is available; otherwise raw bytes |
If ABI decode fails, Thalos keeps raw data and may leave decoded fields partial.

138
docs/api/redis-channels.md
View file

@ -1,65 +1,73 @@
# Redis channels # Redis channels
This document describes the redis channels used by thalos to deliver messages. This document describes the Redis Pub/Sub channels used by Thalos.
## Namespace ## Channel key format
First. all channels have a namespace attached to them. this is done to prevent other application to clash with the keys. All published channels are namespaced to avoid collisions:
The namespace have the following format: `<prefix>::<chain_id>` `<prefix>::<chain_id>::<channel-path>`
* `prefix` is per default `ship` but can be configured to be something else. - `prefix`: defaults to `ship` (config: `redis.prefix`)
* `chain_id` is the chain's id and is used to separate transactions if multiple chains are setup in the same redis database. - `chain_id`: chain id from API by default, or overridden with `ship.chain`
- `channel-path`: one of the paths below
## Transactions
If no chain id is available, Thalos uses an all-zero fallback chain id internally.
All transactions are posted to the following channel:
## Channel paths
`<namespace>::transactions`
### Heartbeat
## Actions
`<namespace>::heartbeat`
there is 4 types of channels for actions.
Published periodically (every 10 blocks) so subscribers can detect liveness.
The channel where all actions are posted is:
### Transactions
`<namespace>::actions`
`<namespace>::transactions`
Channel where only specific actions are posted:
Contains `TransactionTrace` messages.
`<namespace>::actions/name/<action>`
### Actions
Channel where only actions on a specific `<contract>` is posted:
Thalos publishes each action trace to up to 4 channels:
`<namespace>::actions/contract/<contract>`
- All actions:
Channel where only `<action>` on a specific `<contract>` is posted:
`<namespace>::actions`
`<namespace>::actions/contract/<contract>/name/<action>`
- By action name:
## Rollback
`<namespace>::actions/name/<action>`
Rollback mesages are posted to this channel.
- By contract:
`<namespace>::rollback`
`<namespace>::actions/contract/<contract>`
## Table delta
- By contract + action:
Table deltas are posted to the following channels
`<namespace>::actions/contract/<contract>/name/<action>`
All deltas:
### Rollback
`<namespace>::tabledeltas`
`<namespace>::rollback`
Only deltas for a specific type.
Contains `RollbackMessage` messages when microfork rollbacks are detected.
`<namespace>::tabledeltas/name/<name>`
### Table deltas
`<name>` can be one of
- All table deltas:
* `account_metadata`
* `contract_table` `<namespace>::tabledeltas`
* `contract_row`
* `contract_index64` - Table deltas by SHIP delta name:
* `resource_usage`
* `resource_limits_state` `<namespace>::tabledeltas/name/<name>`
`<name>` is the table delta name received from SHIP (for example `contract_row`,
`contract_table`, `resource_usage`, `resource_limits_state`).
## Notes
- `transactions`, `actions`, and `tabledeltas` output can be enabled/disabled via config/flags.
- If `ship.table_delta_whitelist` is configured, only matching `contract_row` deltas are published.

209
docs/configuration.md
View file

@ -1,61 +1,92 @@
# Configuration # Configuration
The configuration file is located at `config.yml` in the installed directory and contains The default configuration file is `./config.yml`.
an example configuration with extensive documentation.
Below are the essential fields that you need to modify. - For bundled/manual installs this is typically in your install directory.
You can adjust the settings according to your preferences. - For Debian packages this is usually `/etc/thalos/config.yml`.
```yaml CLI flags override configuration values when provided.
name: MyShipReader
api: "http://api.example.com:8888" ```yaml
name: "my-thalos-node"
ship: api: "http://127.0.0.1:8888"
url: "ws://ship.example.com:8080" message_codec: "json"
```
ship:
There is also the option to configure Thalos via command line flags. url: "ws://127.0.0.1:8080"
See [this](running-the-server#cli-flags) section for more inforation
redis:
### General addr: "127.0.0.1:6379"
```
`name` ([`string`](#string)) - Name of the talos node.
There is also the option to configure Thalos via command line flags.
`api` ([`string`](#string)) - Nodeos API Endpoint. See [CLI Flags](/docs/running-the-server#cli-flags).
`message_codec` ([`string`](#string)) default: `json` ### General
What codec thalos should use when pushing out messages to redis. `name` ([`string`](#string)) - Name of the Thalos node.
### Ship `api` ([`string`](#string)) - Nodeos API Endpoint.
`ship.url` ([`string`](#string)) - Url to the nodeos node. `message_codec` ([`string`](#string)) default: `json`
What codec Thalos should use when publishing messages to Redis.
### ABI Cache
`abi_cache.api_timeout` ([`duration_string`](#duration-string)) default: `1s`
Timeout used when fetching ABI definitions from the API.
### Ship
`ship.url` ([`string`](#string)) - URL to the nodeos SHIP endpoint.
`ship.chain` ([`string`](#string)) `ship.chain` ([`string`](#string))
name of the chain, if not defined thalos will use the chain id reported from `api`. Name of the chain namespace. If not defined, Thalos uses the chain id reported by `api`.
This value (or chain_id if undefined) is prepended to channel names, to allow for multiple thalos instances This value (or the chain id) is prepended to channel names to allow multiple Thalos instances
to run to run against the same Redis database.
`ship.irreversible_only` ([`boolean`](#boolean)) `ship.irreversible_only` ([`boolean`](#boolean))
If `true`, the ship node will only send transactions once they are considered irreversible. If `true`, the ship node will only send transactions once they are considered irreversible.
If `false` the transactions are posted as soon as possible. If `false` the transactions are posted as soon as possible.
`ship.table_deltas` ([`boolean`](#boolean)) `ship.max_messages_in_flight` ([`integer`](#integer)) default: `10`
If set to `true` thalos will publish table deltas updates to clients. Maximum number of SHIP messages in flight before acknowledgement.
`ship.start_block_num` ([`integer`](#integer)) `ship.table_deltas` ([`boolean`](#boolean))
Thalos will start streaming blocks starting from this one. if undefined, the currect block reported by `api` is used. If set to `true`, Thalos publishes table delta updates to clients.
`ship.table_delta_whitelist` ([`ContractList`](#contractlist))
Optional whitelist for table deltas (`contract:table`).
Use `*` as wildcard for table names.
`ship.transactions` ([`boolean`](#boolean)) default: `true`
If `false`, Thalos skips transaction trace messages.
`ship.actions` ([`boolean`](#boolean)) default: `true`
If `false`, Thalos skips action trace messages.
`ship.start_block_num` ([`integer`](#integer))
Thalos will start streaming blocks starting from this one.
If undefined, Thalos will use cached state when available, otherwise API head/LIB depending on mode.
`ship.end_block_num` ([`integer`](#integer)) `ship.end_block_num` ([`integer`](#integer))
Thalos will stop streaming when the block number defined by this value will be reached. if undefined thalos will never stop. Thalos will stop streaming when this block number is reached.
If undefined, Thalos will keep streaming.
`ship.blacklist` ([`ContractList`](#contractlist)) `ship.blacklist` ([`ContractList`](#contractlist))
List of contract,actions pairs that Thalos will not process if encountered. List of `contract:action` pairs that Thalos will skip.
`ship.blacklist_is_whitelist` ([`boolean`](#boolean)) `ship.blacklist_is_whitelist` ([`boolean`](#boolean))
@ -71,49 +102,49 @@ Thalos will treat `ship.blacklist` as a whitelist.
`redis.db` ([`integer`](#integer)) - Database index to use `redis.db` ([`integer`](#integer)) - Database index to use
`redis.prefix` ([`string`](#string)) - Key prefix, this will be prepended to all channels that thalos is using to avoid name collision `redis.prefix` ([`string`](#string)) - Key prefix prepended to all Thalos channels to avoid name collisions.
### Cache ### Cache
These settings control the cache used by Thalos to cache abi definitions and internal state. These settings control the cache used by Thalos to cache abi definitions and internal state.
`cache.storage` ([`string`](#string) default: `redis`) - Cache storage to use. `cache.storage` ([`string`](#string) default: `redis`) - Cache storage to use.
Avaible values are: `memory`, `redis` Available values: `memory`, `redis`
`cache.options` (`map`) - Options for the cache storage, see section for each cache storage. `cache.options` (`map`) - Options for the cache storage, see section for each cache storage.
#### Memory #### Memory
No configuration. No configuration.
#### Redis #### Redis
`cache.options.stats` ([`boolean`](#string) default: `false`) - True if statistics should be collected `cache.options.stats` ([`boolean`](#boolean) default: `false`) - True if statistics should be collected
`cache.options.size` ([`number`](#string) default: `1000`) - How many items to store in process local memory for faster lookup of popular items `cache.options.size` ([`integer`](#integer) default: `1000`) - How many items to store in process local memory for faster lookup of popular items
`cache.options.ttl` ([`number`](#string) default: `10`) - How long (in minutes) each item should be kept in process local memory before being discared (and has to be fetched from redis again.) `cache.options.ttl` ([`integer`](#integer) default: `10`) - How long (in minutes) each item should be kept in process local memory before being discarded (and has to be fetched from redis again.)
### Logging ### Logging
This block configures how thalos will log information. This block configures how Thalos logs information.
`log.filename` ([`string`](#string)) - Filename (without extension) `log.filename` ([`string`](#string)) - Filename (without extension)
`log.directory` ([`string`](#string)) - Directory where to store log files. `log.directory` ([`string`](#string)) - Directory where to store log files.
`log.timeformat` ([`dateformat`](#dateformat)) - Format to rename log files when rotating `log.file_timestamp_format` ([`dateformat`](#dateformat)) - Format to rename log files when rotating
`log.max_filesize` ([`integer`](#integer)) - Rotate when the file reaches this size. `log.maxfilesize` ([`string`](#string)) - Rotate when the file reaches this size (for example `10mb`, `200mb`).
`log.max_time` ([`duration_string`](#duration-string)) - Rotate when the file is this old. `log.maxtime` ([`duration_string`](#duration-string)) - Rotate when the file is this old.
### Telegram notifications ### Telegram notifications
These settings control the telegram notification mechanism built into thalos. These settings control the Telegram notification mechanism built into Thalos.
if undefined thalos will simple skip this. If undefined, Thalos skips Telegram notifications.
If they are defined, thalos will send messages when important events happen. If defined, Thalos sends messages when important events happen.
`telegram.id` ([`string`](#string)) - bot id `telegram.id` ([`string`](#string)) - bot id
@ -136,10 +167,11 @@ Boolean value (`true` or `false`)
### dateformat ### dateformat
The dateformat type is a `string` but describes a date format in go. The dateformat type is a `string` but describes a date format in Go.
Go uses a different approach then other languages. usually you can format a date like `2023-03-22` like Go uses a different approach than many other languages.
`YYYY-MM-DD`. Go however uses a **reference time** when parsing time. that is just a single point in time. Usually you might format a date like `2023-03-22` as `YYYY-MM-DD`.
Go uses a **reference time** when parsing time:
Mon Jan 2 15:04:05 MST 2006 Mon Jan 2 15:04:05 MST 2006
@ -151,7 +183,7 @@ Go uses a different approach then other languages. usually you can format a date
| `Jan` | Three letter word of month | Oct, Jun | | `Jan` | Three letter word of month | Oct, Jun |
| `January` | Full name of month | April, September | | `January` | Full name of month | April, September |
| `2` | Day in month without leading zero | 1 to 31 | | `2` | Day in month without leading zero | 1 to 31 |
| `02`,`_2` | Day in moth with leading zero | 01 to 31 | | `02`,`_2` | Day in month with leading zero | 01 to 31 |
| `002`,`__2` | Day in year | 002 | | `002`,`__2` | Day in year | 002 |
| `Mon` | Three letter word for day in week | Tue, Fri | | `Mon` | Three letter word for day in week | Tue, Fri |
| `Monday` | Full name of day in week | Friday, Saturday | | `Monday` | Full name of day in week | Friday, Saturday |
@ -178,9 +210,12 @@ Valid time units are `"ns"`, `"us"` (or `"µs"`), `"ms"`, `"s"`, `"m"`, `"h"`.
### ContractList ### ContractList
A array of contract and actions pairs. An array/map of `contract:value` pairs.
Keys (contracts) are of type [`string`](#string) and values are an array of [`string`](#string) (one or more actions). Keys (contracts) are of type [`string`](#string) and values are an array of [`string`](#string).
For `ship.blacklist` the values are action names.
For `ship.table_delta_whitelist` the values are table names.
Action array can hold a special string `*` that matches any action. Action array can hold a special string `*` that matches any action.

4
docs/installation/debian.md
View file

@ -10,7 +10,7 @@ The following documentation assumes that you have already set up a `redis` serve
## 1. Installing the package ## 1. Installing the package
### Using Sw/edens apt repository ### Using EOSSweden apt repository
First, obtain the key. First, obtain the key.
@ -52,4 +52,4 @@ See [Configuration](/docs/configuration) for more information
## 3. Starting the Server ## 3. Starting the Server
There are serveral ways to run the server. the most common on debian is via [systemd](/docs/running-the-server#with-systemd) or [manually](/docs/running-the-server#with-systemd) There are several ways to run the server. The most common on Debian is via [systemd](/docs/running-the-server#with-systemd) or [manually](/docs/running-the-server#starting-manually).

14
docs/installation/docker.md
View file

@ -4,7 +4,7 @@
# Docker # Docker
Do you want to run Thalos in a container? No problem as Thalos provides official docker images Do you want to run Thalos in a container? No problem as Thalos provides official Docker images
to get you going in no time. to get you going in no time.
The images can be found [here](https://github.com/eosswedenorg/thalos/pkgs/container/thalos) The images can be found [here](https://github.com/eosswedenorg/thalos/pkgs/container/thalos)
@ -19,12 +19,12 @@ Check that it works by spawning the container and display the version:
```sh ```sh
$ docker run ghcr.io/eosswedenorg/thalos:latest --version $ docker run ghcr.io/eosswedenorg/thalos:latest --version
thalos-server v1.1.1 thalos-server v<version>
``` ```
## Using docker compose ## Using Docker Compose
Use `docker-compose` to spin up a SHIP node, Redis and Thalos all at once. Use Docker Compose to spin up a SHIP node, Redis, and Thalos all at once.
```yaml ```yaml
services: services:
@ -33,11 +33,11 @@ services:
depends_on: depends_on:
- redis - redis
- ship - ship
# pass arguments to thalos. # Pass arguments to thalos-server.
command: [ command: [
"--redis-addr=redis:6379", "--redis-addr=redis:6379",
"--url=http://ship:80", "--url=http://ship:8888",
"--ship-url=http://ship:8080" "--ship-url=ws://ship:8080"
] ]
# or mount a config file # or mount a config file
#volumes: #volumes:

12
docs/installation/index.md
View file

@ -7,7 +7,7 @@ next:
# Installation # Installation
There are several ways to install thalos, via package manager, downloading a pre-built binary or building directly from source. There are several ways to install Thalos: package manager, pre-built binaries, or building from source.
## Package Managers ## Package Managers
@ -31,7 +31,7 @@ Using curl command above, the files are extracted into the `thalos` subdirectory
### Compiling from source ### Compiling from source
You will need golang version `1.26` or later to compile the source. You will need Go `1.26` or later to compile the source.
#### Compile using make #### Compile using make
@ -39,11 +39,11 @@ You will need golang version `1.26` or later to compile the source.
make make
``` ```
or using go directly if you dont have make installed. Or using `go` directly if you don't have `make` installed:
```shell ```shell
go build -o build/thalos-server cmd/thalos/main.go go build -o build/thalos-server ./cmd/thalos/
``` ```
#### Install #### Install

139
docs/redis/security/acl.md
View file

@ -1,116 +1,91 @@
# ACL # ACL
By default, Redis does not impose any restrictions on the commands that clients can execute. This lack of Redis ACLs let you control **which commands** and **which channels/keys** each user can access.
restrictions raises security concerns, especially when dealing with clients that may not be fully trusted. For Thalos, this is the main way to safely separate:
However, there is no need to worry as Redis offers a solution in the form of [Access Control Lists (ACL)](https://en.wikipedia.org/wiki/Access-control_list).
ACL in Redis enables the establishment of limited access for connections to the server, specifying which - subscriber access (read from Thalos channels)
commands they are allowed to execute. In Thalos, this feature is crucial for maintaining security, ensuring - server access (publish messages + state/cache reads/writes)
that clients can only access specific channels within Thalos.
The special account called `default` serves as the default account for unauthorized users, provided it is ## Recommended user model
configured with a password. Connections can authenticate against this account without specifying a username.
Thalos utilizes this account as the default user account.
Additionally, it is advisable to restrict the Thalos server account as an added precaution against any For a simple setup, use two users:
unauthorized actions it may inadvertently perform, although such occurrences are highly unlikely.
The ACL in thalos is simple and uses 2 accounts: - `default` (or `thalos-client`) for subscribers
- `thalos` for thalos-server
* `default` account (user account, used by clients. only allowed to read from thalos specific channels) ::: warning IMPORTANT
* `thalos-server` account (is allowed to publish to channels and also write to it's cache) Use strong passwords. `ACL GENPASS` is recommended:
<https://redis.io/docs/latest/commands/acl-genpass/>
there is also the `admin` account that is use for redis mangement. has access to everything.
::: danger IMPORTANT
Make sure you replace the passwords with a secure ones.
It is recommended to use [ACL GENPASS](https://redis.io/commands/acl-genpass) to generate strong passwords.
::: :::
`redis.conf` ## Example ACL rules
``` Update `ship::*` if you use a different `redis.prefix`.
user default on >client_password resetchannels &ship::* +@connection +subscribe
user admin on >admin_password ~* &* +@all
user thalos on >server_password resetchannels &ship::* ~thalos::* +@connection +@read +@write +publish
```conf
user default on >client_password resetkeys resetchannels &ship::* -@all +@connection +subscribe +psubscribe +unsubscribe +punsubscribe
user thalos on >server_password resetkeys resetchannels ~ship::* &ship::* -@all +@connection +get +set +publish
``` ```
## External file ### Why both `~...` and `&...`?
It is possible to use external config files to define users in redis. - `~pattern` controls key access
- `&pattern` controls Pub/Sub channel access
Just place the configuration above in an external file for example: `/etc/redis/users.acl` and then add this in `/etc/redis/redis.conf` They are separate controls in Redis ACL.
``` ## External ACL file
Store ACL rules in a dedicated file, then reference it from `redis.conf`:
```conf
aclfile /etc/redis/users.acl aclfile /etc/redis/users.acl
``` ```
## Thalos tools ## Thalos tooling
There is also a tool to create the config lines for you. Thalos includes an ACL helper:
```
$ thalos-tools redis-acl
Passwords
default: NgCseRB8QcxdHwcwmg43OOjxZW0YbngN
thalos: kmJIi1kP5bp2MHQeSj925otWnxTvfRe5
thalos-client: WBGP9mXuUIIv11KhLgbuaTcl1NnRmhWn
# Created by thalos-tools on Mon Feb 12 15:56:26 CET 2024
user default on #3c0745d926021292a521267d803b9c20f3570bee30697e401920151d2593d1e4 ~* &* +@all
user thalos on #cf2b3e36521b0e67cb50fcc1f931f577384bb84ad7d0492d00e07beb4660ff1d resetchannels ~ship::* &ship::* -@all +get +publish +set
user thalos-client on #06d88f1e91d8bc82a5074223fd380006395b80a7b41dcd1318be1dd288282e4a resetchannels &ship::* -@all +subscribe
```
## Test the accounts permissions
It is a good idea to test if the configuration works.
```sh ```sh
$ redis-cli --pass mypassword thalos-tools redis-acl
127.0.0.1:6379> SUBSCRIBE some_channel
Reading messages... (press Ctrl-C to quit)
(error) NOPERM this user has no permissions to access one of the channels used as arguments
127.0.0.1:6379> SET random_key value
(error) NOPERM this user has no permissions to run the 'set' command or its subcommand
127.0.0.1:6379> SUBSCRIBE ship::1234
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "ship::1234"
3) (integer) 1
``` ```
You can customize generated users/prefix with flags such as:
- `--server`
- `--client`
- `--prefix`
- `--cleartext`
Review generated rules before applying them in production.
## Verify permissions
Test that subscriber credentials can only access allowed channels:
```sh
redis-cli --user default --pass client_password
```
Then try:
- `SUBSCRIBE ship::1234::actions` ✅
- `SUBSCRIBE some_other_prefix::1234::actions` ❌ (`NOPERM` expected)
- `SET random_key value` ❌ (`NOPERM` expected)
## Thalos config ## Thalos config
After you have setup ACL, make sure to update your `config.yml` with the account and password. After ACL setup, configure thalos-server with the server user:
```yaml ```yaml
redis: redis:
user: thalos user: thalos
password: p4ssw0rd password: server_password
```
## Other ACL Configurations
### No user password
While not recommended, it is possible to have the default (user) account without password. that way the user does not need to authenticate (but still has limited access). just remove the password from `user default` line:
```
user default on resetchannels &ship::* +@connection +subscribe
```
### Thalos users with a different account
It is also possible to provide a different account for users
```
user thalos-client on >client_password resetchannels &ship::* +@connection +subscribe
``` ```
## Useful links ## Useful links
* [Config File Example](https://redis.io/docs/management/config-file) - [Redis ACL documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/)
* [Official ACL Documentation](https://redis.io/docs/management/security/acl) - [ACL SETUSER command](https://redis.io/docs/latest/commands/acl-setuser/)
- [ACL CAT command](https://redis.io/docs/latest/commands/acl-cat/)
- [Redis Pub/Sub](https://redis.io/docs/latest/develop/pubsub/)

66
docs/redis/security/index.md
View file

@ -1,51 +1,49 @@
# Securing Redis
# Securing redis This guide focuses on Redis deployments where other systems or users can connect
to the same Redis server as Thalos.
This documentation primarily focuses on setups where Redis is exposed to the internet or an internal ## Baseline recommendations
network where there is not complete control over the clients.
For example, you may want to grant access to your Thalos instance to a friend.
While trusting your friend is reasonable, it is essential to consider potential future scenarios where
trust may no longer exist or their server could be compromised.
If you intend to run Thalos for internal use only, such as having internal applications that are relying on a blockchain stream, 1. Do not expose Redis directly to the public internet.
it is perfectly acceptable to skip these steps if you have complete control over all involved 2. Restrict network access (bind + firewall/security groups).
servers and do not expose the instance over a public IP. 3. Use ACL users with least privilege (see [ACL](/docs/redis/security/acl)).
4. Use TLS when traffic crosses untrusted networks.
## Isolating redis ## Isolating Redis
To ensure security, it is highly recommended to run Thalos on a dedicated Redis instance, ideally within a Running Thalos on a dedicated Redis instance (or dedicated DB/service) reduces
container or virtual machine. blast radius if credentials leak or ACL rules are misconfigured.
This isolation helps prevent data leaks in case of misconfigured Redis ACLs or unauthorized access due to
leaked/guessed admin password.
Additionally, it safeguards against potential misconfigurations, such as other applications mistakenly It also lowers the risk of other applications accidentally publishing sensitive
writing sensitive data to Redis channels that can be accessed by Thalos clients. data to channels your subscribers can read.
In summary, isolating Thalos in its own Redis instance provides an extra layer of safety. ## Network and protected mode
## Network Use `bind` in `redis.conf` to restrict interfaces:
The `bind` directive in `redis.conf` is used to tell redis what network interfaces it should bind to. ```conf
Make sure to update this with the interfaces you intend to use. bind 127.0.0.1 ::1 # loopback only
bind 192.168.1.100 10.0.0.1 # specific interfaces
```
::: danger IMPORTANT ::: warning
Although it is recommended to limit Redis to the localhost interface for security purposes, with proper `protected-mode` is a safety fallback, not a full security model.
firewall and ACL configurations, it can be safely exposed to additional interfaces. Carefully evaluate the necessity of For production, still enforce strict network controls and ACLs.
external access before making the change in the config file.
::: :::
```
bind 192.168.1.100 10.0.0.1 # listens on two specific IPv4 addresses
bind 127.0.0.1 ::1 # listens on loopback IPv4 and IPv6
bind * -::* # like the default, all available interfaces
```
## Firewall ## Firewall
Make sure you setup your firewall rules correctly. Only allowing the IP addresses you trust to access the Redis port. Allow Redis port access only from trusted hosts/subnets.
This is out of scope of this documentation. Consult your operating system or router manuals. Firewall setup is environment-specific, so refer to your platform docs.
## TLS
Redis supports native TLS (Redis 6+).
When using TLS-only deployments, disable plaintext (`port 0`) and use `tls-port`.
## Useful links ## Useful links
* [Official Security Documentation](https://redis.io/docs/management/security) - [Redis security overview](https://redis.io/docs/latest/operate/oss_and_stack/management/security/)
* [Config File Example](https://redis.io/docs/management/config-file) - [Redis ACL documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/)
- [Redis TLS documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/security/encryption/)

89
docs/running-the-server.md
View file

@ -24,7 +24,7 @@ If desired, Thalos can also be started manually for quick configuration testing.
./bin/thalos-server ./bin/thalos-server
``` ```
or if you want to specify another config file then the default Or, if you want to specify a different config file than the default:
```shell ```shell
./bin/thalos-server --config /path/to/thalos.yml ./bin/thalos-server --config /path/to/thalos.yml
@ -38,51 +38,64 @@ sudo systemctl start thalos-server
``` ```
After executing these commands, the server should be up and running. After executing these commands, the server should be up and running.
You can check the logs at `/var/log/thalos.log` (unless specified otherwise in the configuration), or by running `sudo systemctl status thalos-server`. You can inspect logs with `sudo journalctl -u thalos-server -f`,
or check the directory configured in `log.directory` (commonly `/var/log/thalos` on Debian packages).
> TIP: if you want to modify the arguments passed to thalos-server when starting > TIP: if you want to modify the arguments passed to thalos-server when starting
via systemd, look in the `/etc/sysconfig/thalos-server` file via systemd, look in the `/etc/sysconfig/thalos-server` file
## CLI Flags ## CLI Flags
Here is the output of `thalos-server -h` showing the available cli flags: Run `thalos-server -h` to see the current flag list for your installed version.
```bash The current source supports these flags:
thalos-server v1.1.4
Usage: ### Core
thalos-server [flags]
Flags: - `-c, --config string` - Config file to read (default `./config.yml`)
--abi-cache-api-timeout duration Duration before the api call times out when the ABI cache requests an abi. (default 1s) - `-u, --url string` - Antelope API URL
--blacklist strings Define a list of 'contract:action' pairs that will be blacklisted (thalos will not process those actions) - `--codec string` - Message codec (default `json`)
--blacklist-is-whitelist Thalos will treat the blacklist as a whitelist - `-L, --level string` - Log level (default `info`)
--cache string What cache driver to use (default "redis") - `-p, --pid string` - Write process id to file
--chain string ChainID used in channel namespace, can be any string (default from api) - `-n, --no-state-cache` - Ignore cached state and use config/API start logic
--codec string Codec used to send messages (default "json") - `-h, --help` - Show help
-c, --config string Config file to read (default "./config.yml") - `-v, --version` - Show version
--end-block uint32 Stop streaming when this block is reached (default none)
-h, --help help for thalos-server
--irreversible-only Only stream irreversible blocks from ship
-L, --level string Log level to use (default "info")
-l, --log string Path to log file (default: print to stdout/stderr)
--log-file-timestamp string Timestamp format to use when rotating log files (default "2006-01-02_150405")
--log-max-filesize string Max filesize for logfile to rotate (default "10mb")
--log-max-time duration Max time for logfile to rotate (default 24h0m0s)
--max-msg-in-flight int Maximum messages that can be sent from SHIP without acknowledgement (default 10)
-n, --no-state-cache Force the application to take start block from config/api
-p, --pid string Where to write process id
--redis-addr string host:port to redis server (default "127.0.0.1:6379")
--redis-db int Redis database
--redis-password string Redis password
--redis-prefix string Redis channel prefix (default "ship")
--redis-user string Redis username
--ship-url string Url to ship node (default "ws://127.0.0.1:8080")
--start-block uint32 Start to stream from this block (default config value, cache, head from api)
--telegram-channel int Telegram channel to send notifications to
--telegram-id string Id of telegram bot
-u, --url string Url to antelope api
-v, --version version for thalos-server
```
### Redis / Cache / Notifications
- `--redis-addr string` - Redis `host:port` (default `127.0.0.1:6379`)
- `--redis-user string` - Redis username
- `--redis-password string` - Redis password
- `--redis-db int` - Redis database index
- `--redis-prefix string` - Redis channel prefix (default `ship`)
- `--cache string` - Cache driver (default `redis`)
- `--abi-cache-api-timeout duration` - ABI fetch timeout (default `1s`)
- `--telegram-id string` - Telegram bot id
- `--telegram-channel int` - Telegram channel id
### SHIP / Stream control
- `--ship-url string` - SHIP websocket URL (default `ws://127.0.0.1:8080`)
- `--start-block uint32` - Start stream from this block
- `--end-block uint32` - Stop stream at this block
- `--irreversible-only` - Stream irreversible blocks only
- `--max-msg-in-flight int` - Max outstanding SHIP messages (default `10`)
- `--chain string` - Chain namespace value (defaults to chain id from API)
### Filtering / Published message types
- `--blacklist strings` - Blacklist `contract:action` pairs
- `--blacklist-is-whitelist` - Treat blacklist as whitelist
- `--table-deltas` - Enable table delta processing (default `true`)
- `--table-delta-whitelist strings` - Whitelist table deltas as `contract:table` pairs (supports `*` wildcard)
- `--transactions` - Enable transaction trace messages (default `true`)
- `--actions` - Enable action trace messages (default `true`)
### Log rotation (when `--log` is used)
- `-l, --log string` - Log file path (default: stdout/stderr)
- `--log-max-filesize string` - Rotate file at size (default `10mb`)
- `--log-max-time duration` - Rotate file at age (default `24h`)
- `--log-file-timestamp string` - Rotation filename timestamp format (default `2006-01-02_150405`)
> Note: setting `--start-block` automatically enables `--no-state-cache`.