Initial Commit

docs/api/clients.md

@@ -0,0 +1,8 @@
+# Clients
+
+List of officially supported client libraries
+
+| Name                                                                | Language |
+| ------------------------------------------------------------------- | -------- |
+| [thalos-go](https://github.com/eosswedenorg/thalos/tree/master/api) | Golang   |
+| [thalos-nodejs](https://github.com/eosswedenorg/thalos-nodejs)      | Nodejs   |

docs/api/messages.md

@@ -0,0 +1,62 @@
+# Messages
+
+This document describes the different messages that are sent
+
+## Encoding
+
+All messages are encoded in `json` format
+
+## Types
+
+### HeartBeat
+
+Heartbeat messages are posted to the heartbeat channel periodically.
+
+| Field                      | Datatype | Description                                 |
+| -------------------------- | -------- | ------------------------------------------- |
+| blocknum                   | int      | Current block number                        |
+| head_blocknum              | int      | Head block number                           |
+| last_irreversible_blocknum | int      | block number of the last irreversible block |
+
+### Transaction
+
+
+### ActionTrace
+
+| Field          | Datatype          | Description                                                       |
+| -------------- | ----------------- | ----------------------------------------------------------------- |
+| tx_id          | string            | Transaction ID                                                    |
+| blocknum       | int               | Block number where this action trace (and transaction) belongs to |
+| blocktimestamp | time              | Block timestamp                                                   |
+| receipt        | ActionReceipt     | Action receipt                                                    |
+| receiver       | string            | Receiver account                                                  |
+| contract       | string            | Contract account                                                  |
+| action         | string            | What action was executed on the contract                          |
+| data           | any               | Contract specific data (decoded using the contracts abi)          |
+| authorization  | PermissionLevel[] | Authorization                                                     |
+
+### ActionReceipt
+
+| Field           | Datatype              | Description        |
+| --------------- | --------------------- | ------------------ |
+| receiver        | string                | Actor account name |
+| act_digest      | string                | Action digest      |
+| global_sequence | int                   | Global sequence    |
+| recv_sequence   | int                   | Receive sequence   |
+| auth_sequence   | AccountAuthSequence[] | Auth sequence      |
+| code_sequence   | int                   | Code sequence      |
+| abi_sequence    | int                   | ABI sequence       |
+
+### PermisssionLevel
+
+| Field      | Datatype | Description                      |
+| ---------- | -------- | -------------------------------- |
+| actor      | string   | Actor account name               |
+| permission | string   | Permission (for example: active) |
+
+### AccountAuthSequence
+
+| Field    | Datatype | Description  |
+| -------- | -------- | ------------ |
+| account  | string   | Account name |
+| sequence | int      | Sequence     |

docs/api/redis-channels.md

@@ -0,0 +1,38 @@
+# Redis channels
+
+This document describes the redis channels used by thalos to deliver messages.
+
+## Namespace
+
+First. all channels have a namespace attached to them. this is done to prevent other application to clash with the keys.
+
+The namespace have the following format: `<prefix>::<chain_id>`
+
+* `prefix` is per default `ship` but can be configured to be something else.
+* `chain_id` is the chain's id and is used to separate transactions if multiple chains are setup in the same redis database.
+
+## Transactions
+
+All transactions are posted to the following channel:
+
+`<namespace>::transactions`
+
+## Actions
+
+there is 4 types of channels for actions.
+
+The channel where all actions are posted is:
+
+`<namespace>::actions`
+
+Channel where only specific actions are posted:
+
+`<namespace>::actions/name/<action>`
+
+Channel where only actions on a specific `<contract>` is posted:
+
+`<namespace>::actions/contract/<contract>`
+
+Channel where only `<action>` on a specific `<contract>` is posted:
+
+`<namespace>::actions/contract/<contract>/name/<action>`

docs/architecture/benchmark.md

@@ -0,0 +1,71 @@
+# Benchmark
+
+Thalos demonstrates impressive speed, and we have solid data to support this claim.
+
+We conducted two distinct types of benchmark tests:
+
+* **Live Benchmark**: Thalos was thoroughly benchmarked against streaming data from the current head block. This benchmark provides a highly realistic evaluation, although it operates at a slightly slower pace due to real-time processing limitations. Notably, the potential bottleneck arises from the fact that a new block is generated only once every 0.5 seconds.
+
+* **Replay Benchmark**: Thalos was operated in replay mode, utilizing data from a previous block.
+This mode allows for higher throughput as it eliminates real-time constraints since historical blocks have already been generated.
+Therefore, Thalos does not have to wait 0.5 second for the next block.
+
+In addition to the aforementioned test types, it is important to assess the impact of multiple clients on performance. Therefore, we categorize the results as `NxHardware`, where `N` represents the number of clients and `Hardware` signifies the hardware configuration used. For example, the designation `10xlow` indicates 10 clients using low-end hardware.
+
+All tests were diligently conducted on the WAX Mainnet.
+
+## Hardware
+
+The following hardware was used.
+
+1. **Low-end**: Raspberry Pi 3 Model B Rev 1.2
+2. **Mid-end**: Intel(R) Xeon(R) W-2125 CPU 4-Core (8 threads) @ 4.0GHz
+3. **High-end**: 2x Intel(R) Xeon(R) Gold 6128 CPU 6-cores (12 threads) @ 3.4GHz
+
+Please note that during the tests conducted on mid and high-end hardware, additional services that consume resources were present.
+This was due to the unavailability of idle servers for testing purposes.
+Consequently, it is important to consider that the performance on these hardware configurations could potentially be even better than the results obtained.
+
+## Live
+
+![Live Data msg per sec](./live.png)
+
+::: details
+\* Redis discards messages for clients because they cannot be processed in time. resulting in dataloss.
+:::
+
+
+Raw data:
+
+| 10xhigh           | 100xhigh          | 10xmid            | 100xmid           | 10xlow             | 100xlow*           |
+| ----------------- | ----------------- | ----------------- | ----------------- | ------------------ | ------------------ |
+| 531.9520324320212 | 517.2177668044619 | 549.1276634032015 | 553.6179173960732 | 432.66369389719085 | 17.004179093675067 |
+
+These results are quite interesting. High and mid-end hardware are about equal even for 100 clients. And the low-end hardware is not so far of. capable of handling atleast 10 clients.
+
+Note that one block contains alot of transactions and actions resulting in alot more then one redis message per block. therefore even if there is just 2 blocks per second. There is alot more redis messages sent out.
+
+## Replay
+
+![Replay Data msg per sec](./replay.png)
+
+::: details
+\* Redis discards messages for clients because they cannot be processed in time. resulting in dataloss.
+:::
+
+
+Raw data:
+
+| 10xhigh           | 100xhigh*          | 10xmid             | 100xmid*           | 10xlow*            | 100xlow*           |
+| ----------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |
+| 3853.401198489492 | 1614.2395356540624 | 4002.5386670846724 | 1297.8602672923382 | 211.21962626224587 | 0                  |
+
+The benchmarking of 100 clients on low-end hardware was intentionally excluded. The system's responsiveness was severely compromised during the test with 100 clients using live data, and running the same test on 100 clients with replay data would not make it better.
+it is just as 10xlow, not usable.
+
+## Conclusion
+
+By looking at the live data graphs they process roughly the same amount of messages per second. That is because the bottleneck is the blockchain
+itself. If we look at the replay data, redis clients can't keep up as number of clients increases. however, it is still pretty fast. consider that all 10x clients performs roughly equal.
+
+Also the fact that the benchmark tests fetchees **all** messages. That is pretty unrealistic as applications should in 99% of the cases only care about actions on a subset of contracts. Only case where you would want all actions is if you are building some sort of blockexplorer.

docs/architecture/index.md

@@ -0,0 +1,43 @@
+# Architecture
+
+Below is the diagram of how Thalos processes messages from a SHIP Node
+
+![overview](./overview.jpg)
+
+## Decoding Binary Format
+
+When leveraging the Antelope State History plugin (SHIP), developers have the capability to stream blockchain data via a WebSocket connection.
+Nevertheless, the process of decoding the binary format and managing the WebSocket connection can prove to be laborious and time-intensive.
+Thalos alleviates these challenges by assuming the responsibility of decoding the binary messages on behalf of developers.
+It seamlessly transforms the data received from the SHIP node into plain JSON format (and can be adapted to other prevalent formats if desired).
+JSON, being widely supported, has many implementations across various programming languages.
+
+## Contract ABIs
+
+Thalos simplifies the decoding of contract-specific data by maintaining an internal cache of contract ABIs. This cache, which is stored in Redis, allows for efficient retrieval of contract information. There are two ways in which the cache can be populated:
+
+1. **API Node**: Thalos can query the contract ABI through an API node, populating the cache with the retrieved information.
+2. **SHIP Node Messages**: Thalos keeps track of updates from the SHIP node, and when a contract is updated, it automatically updates its internal cache with the latest ABI information.
+
+## Redis - PubSub vs Stream
+
+Thalos utilizes Redis's publish/subscribe model known as Pub/Sub, which offers the following advantages:
+
+1. **Decoupled communication**: Pub/Sub enables loose coupling between publishers and subscribers. Publishers are unaware of the subscribers, and subscribers can join or leave independently, without affecting the publishing process.
+
+2. **Scalability**: Redis Pub/Sub is designed to handle high message throughput and efficiently distribute messages to a large number of subscribers. It can handle a high volume of messages without compromising performance.
+
+3. **Simple implementation**: Redis's Pub/Sub mechanism is easy to implement, involving basic operations such as subscribing to channels, publishing messages, and receiving messages. This simplicity facilitates development and integration with other systems.
+
+4. **Asynchronous communication**: Pub/Sub operates asynchronously, allowing publishers to send messages without waiting for subscribers to receive them. This decoupling is advantageous in scenarios where immediate response or synchronization is not necessary.
+
+However, these advantages can also be considered drawbacks in certain situations. Pub/Sub may encounter issues with message persistence:
+
+- **Message loss**: If clients disconnect and reconnect, they may miss messages published during their absence. Similarly, if a client cannot process messages quickly enough, it may not receive all messages.
+
+- **Immediate message consumption**: Pub/Sub requires subscribers to consume messages immediately upon publication. If a client is unable to read a message promptly, it is discarded.
+
+In contrast, Streams in Redis provide built-in message persistence. Messages are stored as entries in the stream and can be consumed at any time, even if they were published before the subscriber connected.
+This feature addresses the drawbacks of immediate message consumption and message loss associated with Pub/Sub.
+
+Currently, Streams are not implemented in Thalos. However, there are plans to consider implementing Streams in the future, based on the demand for such functionality. It's important to note that Streams introduce additional complexity to the client implementation.

docs/configuration.md

@@ -0,0 +1,41 @@
+# Configure and run the server
+
+The configuration file is located at `config.yml` in the installed directory and contains an example configuration with extensive documentation. Below are the essential fields that you need to modify. You can adjust the settings according to your preferences.
+
+```yaml
+name: MyShipReader
+api: "http://api.example.com:8888"
+
+ship:
+  url: "ws://ship.example.com:8080"
+```
+
+## Start using scripts
+
+Start the server using the `start.sh` script.
+
+```shell
+./start.sh
+```
+
+The logs can be found in `logs` directory (unless specified otherwise in the configuration).
+
+Stopping the server again is as simple as running.
+
+```shell
+./stop.sh
+```
+
+## Starting Manually
+
+If desired, Thalos can also be started manually for quick configuration testing.
+
+```shell
+./bin/thalos-server
+```
+
+or if you want to specify another config file then the default
+
+```shell
+./bin/thalos-server --config /path/to/thalos.yml
+```

docs/index.md

@@ -0,0 +1,11 @@
+# Documentation
+
+Thalos is a application that makes it easy for users to stream blockchain data from an Antelope SHIP node.
+
+It handles all the technical stuff for you:
+
+ * Decoding of antelope's binary format.
+ * Websocket connection (with reconnection)
+ * Decoding of action data according to contract ABI
+
+And then sends the data over redis in plain json (or other popular formats if you want!)

docs/installation/debian.md

@@ -0,0 +1,68 @@
+---
+next:
+  text: Configuration
+  link: /docs/configuration
+---
+
+# Debian
+
+The following documentation assumes that you have already set up a `redis` server, a `leap SHIP` node, and a running `leap API` node.
+
+## 1. Installing the package
+
+### Using Sw/edens apt repository
+
+First, obtain the key.
+
+```sh
+sudo apt-get install software-properties-common
+curl -sS https://apt.eossweden.org/key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/eossweden-2023.gpg > /dev/null
+```
+
+Next, install the package.
+
+#### For bash shell
+
+```shell
+sudo apt-add-repository -y "deb [arch=amd64] https://apt.eossweden.org/main `lsb_release -cs` stable"
+sudo apt-get install thalos
+```
+
+#### For fish shell
+
+```shell
+sudo apt-add-repository -y "deb [arch=amd64] https://apt.eossweden.org/main "(lsb_release -cs)" stable"
+sudo apt-get install thalos
+```
+
+### Manual installation
+
+Alternatively, you can manually install the package by downloading the .deb file from the [latest](https://github.com/eosswedenorg/thalos/releases/latest) release.
+
+```shell
+curl https://github.com/eosswedenorg/thalos/releases/download/<version>/thalos_<version>_amd64.deb
+sudo apt-get install ./thalos_<version>_amd64.deb
+```
+
+## 2. Configuration
+
+The configuration file is located at `/etc/thalos/config.yml` and contains an example configuration with extensive documentation.
+
+See [Configuration](/docs/configuration) for more information
+
+## 3. Starting the Server via systemd
+
+```shell
+sudo systemctl enable thalos-server
+sudo systemctl start thalos-server
+```
+
+After executing these commands, the server should be up and running. You can check the logs at `/var/log/thalos.log` (unless specified otherwise in the configuration), or by running `sudo systemctl status thalos-server`.
+
+### Starting Manually
+
+If desired, Thalos can also be started manually for quick configuration testing or in cases where running systemd is not preferable:
+
+```shell
+/usr/bin/thalos-server --config /etc/thalos/thalos.yml
+```

docs/installation/index.md

@@ -0,0 +1,53 @@
+---
+next:
+  text: Configuration
+  link: /docs/configuration
+---
+
+
+# Installation
+
+There are several ways to install thalos, via package manager, downloading a pre-built binary or building directly from source.
+
+## Package Managers
+
+* [Debian/Ubuntu based (apt)](/docs/installation/debian)
+
+## Manually
+
+### Bundled binaries
+
+You can get the latest archive package [here](https://github.com/eosswedenorg/thalos/releases/latest)
+
+Simply download using your webbrowser or via curl:
+
+```shell
+curl -Ls https://github.com/eosswedenorg/thalos/releases/download/<version>/thalos-server-<version>-linux-amd64.tar.gz | tar -z --one-top-level=thalos -xvf -
+```
+
+::: info
+Using curl command above, the files are extracted into the `thalos` subdirectory of the current directory where the command is run.
+:::
+
+### Compiling from source
+
+You will need golang version `1.18` or later to compile the source.
+
+#### Compile using make
+
+```shell
+make
+```
+
+or using go directly if you dont have make installed.
+
+```shell
+go build -o build/thalos-server cmd/main/main.go
+```
+
+#### Install
+
+After building the binary you can install it along with basic config file and start/stop scripts using `install.sh`
+
+```shell
+./install.sh /path/to/your/directory/of/choice