From 3b0a963879b63568bf3d8cf050f20c69411ba9c6 Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Sun, 25 Aug 2024 17:33:14 +0200 Subject: [PATCH 1/6] docs/configuration.md: document cache --- docs/configuration.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 240ac11..27f0cf9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -63,6 +63,28 @@ Thalos will treat `ship.blacklist` as a whitelist. `redis.prefix` ([`string`](#string)) - Key prefix, this will be prepended to all channels that thalos is using to avoid name collision +### Cache + +These settings control the cache used by Thalos to cache abi definitions and internal state. + +`cache.storage` ([`string`](#string) default: `redis`) - Cache storage to use. + +Avaible values are: `memory`, `redis` + +`cache.options` (`map`) - Options for the cache storage, see section for each cache storage. + +#### Memory + +No configuration. + +#### Redis + +`cache.options.stats` ([`boolean`](#string) default: `false`) - True if statistics should be collected + +`cache.options.size` ([`number`](#string) default: `1000`) - How many items to store in process local memory for faster lookup of popular items + +`cache.options.ttl` ([`number`](#string) default: `10`) - How long (in minutes) each item should be kept in process local memory before being discared (and has to be fetched from redis again.) + ### Logging This block configures how thalos will log information. From 18a1b3ed89b18fe31c7cb768fe186390f4c81be1 Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Sun, 25 Aug 2024 18:28:48 +0200 Subject: [PATCH 2/6] docs/api/messages.md: typo --- docs/api/messages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/messages.md b/docs/api/messages.md index 0fce04e..3b9724b 100644 --- a/docs/api/messages.md +++ b/docs/api/messages.md @@ -48,7 +48,7 @@ Heartbeat messages are posted to the heartbeat channel periodically. | code_sequence | int | Code sequence | | abi_sequence | int | ABI sequence | -### PermisssionLevel +### PermissionLevel | Field | Datatype | Description | | ---------- | -------- | -------------------------------- | From a642e49df4af0ffc33449d565eda66b706673e99 Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Sun, 25 Aug 2024 18:36:10 +0200 Subject: [PATCH 3/6] docs/api/messages.md: stylefix and link datatypes to other datatypes --- docs/api/messages.md | 82 ++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/docs/api/messages.md b/docs/api/messages.md index 3b9724b..4b8ad44 100644 --- a/docs/api/messages.md +++ b/docs/api/messages.md @@ -14,75 +14,75 @@ Heartbeat messages are posted to the heartbeat channel periodically. | Field | Datatype | Description | | -------------------------- | -------- | ------------------------------------------- | -| blocknum | int | Current block number | -| head_blocknum | int | Head block number | -| last_irreversible_blocknum | int | block number of the last irreversible block | +| blocknum | `int` | Current block number | +| head_blocknum | `int` | Head block number | +| last_irreversible_blocknum | `int` | block number of the last irreversible block | ### Transaction ### ActionTrace -| Field | Datatype | Description | -| -------------- | ----------------- | ----------------------------------------------------------------- | -| tx_id | string | Transaction ID | -| blocknum | int | Block number where this action trace (and transaction) belongs to | -| blocktimestamp | time | Block timestamp | -| receipt | ActionReceipt | Action receipt | -| receiver | string | Receiver account | -| first_receiver | bool | True if receiver is the first account to get notified | -| contract | string | Contract account | -| action | string | What action was executed on the contract | -| data | any | Contract specific data (decoded using the contracts abi) | -| authorization | PermissionLevel[] | Authorization | +| Field | Datatype | Description | +| -------------- | --------------------------------------- | ----------------------------------------------------------------- | +| tx_id | `string` | Transaction ID | +| blocknum | `int` | Block number where this action trace (and transaction) belongs to | +| blocktimestamp | `time` | Block timestamp | +| receipt | [`ActionReceipt`](#actionreceipt) | Action receipt | +| receiver | `string` | Receiver account | +| first_receiver | `bool` | True if receiver is the first account to get notified | +| contract | `string` | Contract account | +| action | `string` | What action was executed on the contract | +| data | `any` | Contract specific data (decoded using the contracts abi) | +| authorization | [`PermissionLevel[]`](#permissionlevel) | Authorization | ### ActionReceipt -| Field | Datatype | Description | -| --------------- | --------------------- | ------------------ | -| receiver | string | Actor account name | -| act_digest | string | Action digest | -| global_sequence | int | Global sequence | -| recv_sequence | int | Receive sequence | -| auth_sequence | AccountAuthSequence[] | Auth sequence | -| code_sequence | int | Code sequence | -| abi_sequence | int | ABI sequence | +| Field | Datatype | Description | +| --------------- | ----------------------------------------------- | ------------------ | +| receiver | `string` | Actor account name | +| act_digest | `string` | Action digest | +| global_sequence | `int` | Global sequence | +| recv_sequence | `int` | Receive sequence | +| auth_sequence | [`AccountAuthSequence[]`](#accountauthsequence) | Auth sequence | +| code_sequence | `int` | Code sequence | +| abi_sequence | `int` | ABI sequence | ### PermissionLevel | Field | Datatype | Description | | ---------- | -------- | -------------------------------- | -| actor | string | Actor account name | -| permission | string | Permission (for example: active) | +| actor | `string` | Actor account name | +| permission | `string` | Permission (for example: active) | ### AccountAuthSequence | Field | Datatype | Description | | -------- | -------- | ------------ | -| account | string | Account name | -| sequence | int | Sequence | +| account | `string` | Account name | +| sequence | `int` | Sequence | ### RollbackMessage | Field | Datatype | Description | | --------- | -------- | ------------------------------------ | -| new_block | int | The current block number | -| old_block | int | Last block number that was received. | +| new_block | `int` | The current block number | +| old_block | `int` | Last block number that was received. | ### TableDelta -| Field | Datatype | Description | -| -------------- | --------------- | ----------------- | -| blocknum | int | Block number | -| blocktimestamp | time | Block timestamp | -| name | string | Table name | -| rows | TableDeltaRow[] | Rows in the delta | +| Field | Datatype | Description | +| -------------- | ----------------------------------- | ----------------- | +| blocknum | `int` | Block number | +| blocktimestamp | `time` | Block timestamp | +| name | `string` | Table name | +| rows | [`TableDeltaRow[]`](#tabledeltarow) | Rows in the delta | ### TableDeltaRow -| Field | Datatype | Description | -| -------- | ------------- | --------------------------------- | -| present | bool | - | -| data | any | Decoded data | -| raw_data | base64_string | Raw data in base64 encoded string | +| Field | Datatype | Description | +| -------- | --------------- | --------------------------------- | +| present | `bool` | - | +| data | `any` | Decoded data | +| raw_data | `base64_string` | Raw data in base64 encoded string | From 781da3b6a8e2f45948d04e0457d853c5fa8c0d50 Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Sun, 25 Aug 2024 18:47:23 +0200 Subject: [PATCH 4/6] docs/api/messages.md: document TableDeltaRowContract --- docs/api/messages.md | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/api/messages.md b/docs/api/messages.md index 4b8ad44..72c982a 100644 --- a/docs/api/messages.md +++ b/docs/api/messages.md @@ -81,8 +81,27 @@ Heartbeat messages are posted to the heartbeat channel periodically. ### TableDeltaRow -| Field | Datatype | Description | -| -------- | --------------- | --------------------------------- | -| present | `bool` | - | -| data | `any` | Decoded data | -| raw_data | `base64_string` | Raw data in base64 encoded string | +| Field | Datatype | Description | +| -------- | ---------------------------------------------------------- | --------------------------------- | +| present | `bool` | - | +| data | [`TableDeltaRowContract`](#tabledeltarowcontract) \| `any` | Decoded data | +| raw_data | `base64_string` | Raw data in base64 encoded string | + +### TableDeltaRowContract + +These messages are only relevant when listening to [Table deltas](redis-channels#table-delta) channel when name is `contract_row`. + + + +| Field | Datatyp | Description | +| ----------- | ------------------------ | ---------------------------------------------------------- | +| table | `string` | table name | +| scope | `string` | table scope | +| primary_key | `string` \| `int` | Primary ID for the row. | +| code | `string` | Account name where that has the contract code | +| payer | `string` | Account name that payed for the ram that this row consumes | +| value | `map` \| `base64_string` | Actual data in the table | + + +Thalos can decode contract row data in `vaule` using the contracts abi. +If Thalos could not decode the data then `value` will contain a `base64_string` of the raw value instead. From d564a0b3ee2570a5d99923cbc996d3641c4f20cd Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Thu, 29 Aug 2024 14:53:35 +0200 Subject: [PATCH 5/6] minor style fixes --- docs/architecture/benchmark.md | 12 ++++++++---- docs/architecture/index.md | 7 +++++-- docs/installation/index.md | 4 ++-- docs/redis/security/acl.md | 5 +++-- docs/redis/security/index.md | 14 ++++++++++---- 5 files changed, 28 insertions(+), 14 deletions(-) diff --git a/docs/architecture/benchmark.md b/docs/architecture/benchmark.md index e0a28be..0b93b01 100644 --- a/docs/architecture/benchmark.md +++ b/docs/architecture/benchmark.md @@ -31,7 +31,7 @@ Consequently, it is important to consider that the performance on these hardware ![Live Data msg per sec](./live.png) ::: details -\* Redis discards messages for clients because they cannot be processed in time. resulting in dataloss. +\* Redis discards messages for clients because they cannot be processed in time. Resulting in data loss. ::: @@ -43,7 +43,8 @@ Raw data: These results are quite interesting. High and mid-end hardware are about equal even for 100 clients. And the low-end hardware is not so far of. capable of handling atleast 10 clients. -Note that one block contains alot of transactions and actions resulting in alot more then one redis message per block. therefore even if there is just 2 blocks per second. There is alot more redis messages sent out. +Note that one block contains a lot of transactions and actions resulting in a lot more then one Redis message per block. +Therefore even if there is just 2 blocks per second. There is a lot more Redis messages sent out. ## Replay @@ -66,6 +67,9 @@ it is just as 10xlow, not usable. ## Conclusion By looking at the live data graphs they process roughly the same amount of messages per second. That is because the bottleneck is the blockchain -itself. If we look at the replay data, redis clients can't keep up as number of clients increases. however, it is still pretty fast. consider that all 10x clients performs roughly equal. +itself. If we look at the replay data, redis clients can't keep up as number of clients increases. +However, it is still pretty fast. Consider that all 10x clients performs roughly equal. -Also the fact that the benchmark tests fetchees **all** messages. That is pretty unrealistic as applications should in 99% of the cases only care about actions on a subset of contracts. Only case where you would want all actions is if you are building some sort of blockexplorer. \ No newline at end of file +Also the fact that the benchmark tests fetches **all** messages. That is pretty unrealistic as applications +should in 99% of the cases only care about actions on a subset of contracts. +Only case where you would want all actions is if you are building some sort of blockexplorer. diff --git a/docs/architecture/index.md b/docs/architecture/index.md index ff4b6eb..382afa1 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -37,7 +37,10 @@ However, these advantages can also be considered drawbacks in certain situations - **Immediate message consumption**: Pub/Sub requires subscribers to consume messages immediately upon publication. If a client is unable to read a message promptly, it is discarded. -In contrast, Streams in Redis provide built-in message persistence. Messages are stored as entries in the stream and can be consumed at any time, even if they were published before the subscriber connected. +In contrast, Streams in Redis provide built-in message persistence. +Messages are stored as entries in the stream and can be consumed at any time, even if they were published before the subscriber connected. This feature addresses the drawbacks of immediate message consumption and message loss associated with Pub/Sub. -Currently, Streams are not implemented in Thalos. However, there are plans to consider implementing Streams in the future, based on the demand for such functionality. It's important to note that Streams introduce additional complexity to the client implementation. \ No newline at end of file +Currently, Streams are not implemented in Thalos. +However, there are plans to consider implementing Streams in the future, based on the demand for such functionality. +It's important to note that Streams introduce additional complexity to the client implementation. diff --git a/docs/installation/index.md b/docs/installation/index.md index 75c5c0c..6f5e8b6 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -19,7 +19,7 @@ There are several ways to install thalos, via package manager, downloading a pre You can get the latest archive package [here](https://github.com/eosswedenorg/thalos/releases/latest) -Simply download using your webbrowser or via curl: +Simply download using your web browser or via curl: ```shell curl -Ls https://github.com/eosswedenorg/thalos/releases/download//thalos-server--linux-amd64.tar.gz | tar -z --one-top-level=thalos -xvf - @@ -51,4 +51,4 @@ After building the binary you can install it along with basic config file and st ```shell ./install.sh /path/to/your/directory/of/choice -``` \ No newline at end of file +``` diff --git a/docs/redis/security/acl.md b/docs/redis/security/acl.md index 88b44be..a54a6d7 100644 --- a/docs/redis/security/acl.md +++ b/docs/redis/security/acl.md @@ -12,7 +12,8 @@ The special account called `default` serves as the default account for unauthori configured with a password. Connections can authenticate against this account without specifying a username. Thalos utilizes this account as the default user account. -Additionally, it is advisable to restrict the Thalos server account as an added precaution against any unauthorized actions it may inadvertently perform, although such occurrences are highly unlikely. +Additionally, it is advisable to restrict the Thalos server account as an added precaution against any +unauthorized actions it may inadvertently perform, although such occurrences are highly unlikely. The ACL in thalos is simple and uses 2 accounts: @@ -112,4 +113,4 @@ user thalos-client on >client_password resetchannels &ship::* +@connection +subs ## Useful links * [Config File Example](https://redis.io/docs/management/config-file) -* [Official ACL Documentation](https://redis.io/docs/management/security/acl) \ No newline at end of file +* [Official ACL Documentation](https://redis.io/docs/management/security/acl) diff --git a/docs/redis/security/index.md b/docs/redis/security/index.md index cb64fa6..1040a71 100644 --- a/docs/redis/security/index.md +++ b/docs/redis/security/index.md @@ -1,9 +1,15 @@ # Securing redis -This documentation primarily focuses on setups where Redis is exposed to the internet or an internal network where there is not complete control over the clients. For example, you may want to grant access to your Thalos instance to a friend. While trusting your friend is reasonable, it is essential to consider potential future scenarios where trust may no longer exist or their server could be compromised. +This documentation primarily focuses on setups where Redis is exposed to the internet or an internal +network where there is not complete control over the clients. +For example, you may want to grant access to your Thalos instance to a friend. +While trusting your friend is reasonable, it is essential to consider potential future scenarios where +trust may no longer exist or their server could be compromised. -If you intend to run Thalos for internal use only, such as having internal applications that are relying on a blockchain stream, it is perfectly acceptable to skip these steps if you have complete control over all involved servers and do not expose the instance over a public IP. +If you intend to run Thalos for internal use only, such as having internal applications that are relying on a blockchain stream, +it is perfectly acceptable to skip these steps if you have complete control over all involved +servers and do not expose the instance over a public IP. ## Isolating redis @@ -35,8 +41,8 @@ bind * -::* # like the default, all available interfaces ## Firewall -Make sure you setup your firewall rules correctly. only allowing the ip's you trust to access the redis port. -This is out of scope of this documentation. consult your operating system or router manuals. +Make sure you setup your firewall rules correctly. Only allowing the IP addresses you trust to access the Redis port. +This is out of scope of this documentation. Consult your operating system or router manuals. ## Useful links From 9acb57ffaa48b6900ff983bf824b5c981af9a05f Mon Sep 17 00:00:00 2001 From: Henrik Hautakoski Date: Thu, 29 Aug 2024 14:53:51 +0200 Subject: [PATCH 6/6] docs/installation/index.md: golang v1.21 or later is now required --- docs/installation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation/index.md b/docs/installation/index.md index 6f5e8b6..51fa8ec 100644 --- a/docs/installation/index.md +++ b/docs/installation/index.md @@ -31,7 +31,7 @@ Using curl command above, the files are extracted into the `thalos` subdirectory ### Compiling from source -You will need golang version `1.20` or later to compile the source. +You will need golang version `1.21` or later to compile the source. #### Compile using make