Initial Commit

docs/architecture/benchmark.md

@@ -0,0 +1,71 @@
+# Benchmark
+
+Thalos demonstrates impressive speed, and we have solid data to support this claim.
+
+We conducted two distinct types of benchmark tests:
+
+* **Live Benchmark**: Thalos was thoroughly benchmarked against streaming data from the current head block. This benchmark provides a highly realistic evaluation, although it operates at a slightly slower pace due to real-time processing limitations. Notably, the potential bottleneck arises from the fact that a new block is generated only once every 0.5 seconds.
+
+* **Replay Benchmark**: Thalos was operated in replay mode, utilizing data from a previous block.
+This mode allows for higher throughput as it eliminates real-time constraints since historical blocks have already been generated.
+Therefore, Thalos does not have to wait 0.5 second for the next block.
+
+In addition to the aforementioned test types, it is important to assess the impact of multiple clients on performance. Therefore, we categorize the results as `NxHardware`, where `N` represents the number of clients and `Hardware` signifies the hardware configuration used. For example, the designation `10xlow` indicates 10 clients using low-end hardware.
+
+All tests were diligently conducted on the WAX Mainnet.
+
+## Hardware
+
+The following hardware was used.
+
+1. **Low-end**: Raspberry Pi 3 Model B Rev 1.2
+2. **Mid-end**: Intel(R) Xeon(R) W-2125 CPU 4-Core (8 threads) @ 4.0GHz
+3. **High-end**: 2x Intel(R) Xeon(R) Gold 6128 CPU 6-cores (12 threads) @ 3.4GHz
+
+Please note that during the tests conducted on mid and high-end hardware, additional services that consume resources were present.
+This was due to the unavailability of idle servers for testing purposes.
+Consequently, it is important to consider that the performance on these hardware configurations could potentially be even better than the results obtained.
+
+## Live
+
+![Live Data msg per sec](./live.png)
+
+::: details
+\* Redis discards messages for clients because they cannot be processed in time. resulting in dataloss.
+:::
+
+
+Raw data:
+
+| 10xhigh           | 100xhigh          | 10xmid            | 100xmid           | 10xlow             | 100xlow*           |
+| ----------------- | ----------------- | ----------------- | ----------------- | ------------------ | ------------------ |
+| 531.9520324320212 | 517.2177668044619 | 549.1276634032015 | 553.6179173960732 | 432.66369389719085 | 17.004179093675067 |
+
+These results are quite interesting. High and mid-end hardware are about equal even for 100 clients. And the low-end hardware is not so far of. capable of handling atleast 10 clients.
+
+Note that one block contains alot of transactions and actions resulting in alot more then one redis message per block. therefore even if there is just 2 blocks per second. There is alot more redis messages sent out.
+
+## Replay
+
+![Replay Data msg per sec](./replay.png)
+
+::: details
+\* Redis discards messages for clients because they cannot be processed in time. resulting in dataloss.
+:::
+
+
+Raw data:
+
+| 10xhigh           | 100xhigh*          | 10xmid             | 100xmid*           | 10xlow*            | 100xlow*           |
+| ----------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |
+| 3853.401198489492 | 1614.2395356540624 | 4002.5386670846724 | 1297.8602672923382 | 211.21962626224587 | 0                  |
+
+The benchmarking of 100 clients on low-end hardware was intentionally excluded. The system's responsiveness was severely compromised during the test with 100 clients using live data, and running the same test on 100 clients with replay data would not make it better.
+it is just as 10xlow, not usable.
+
+## Conclusion
+
+By looking at the live data graphs they process roughly the same amount of messages per second. That is because the bottleneck is the blockchain
+itself. If we look at the replay data, redis clients can't keep up as number of clients increases. however, it is still pretty fast. consider that all 10x clients performs roughly equal.
+
+Also the fact that the benchmark tests fetchees **all** messages. That is pretty unrealistic as applications should in 99% of the cases only care about actions on a subset of contracts. Only case where you would want all actions is if you are building some sort of blockexplorer.

docs/architecture/index.md

@@ -0,0 +1,43 @@
+# Architecture
+
+Below is the diagram of how Thalos processes messages from a SHIP Node
+
+![overview](./overview.jpg)
+
+## Decoding Binary Format
+
+When leveraging the Antelope State History plugin (SHIP), developers have the capability to stream blockchain data via a WebSocket connection.
+Nevertheless, the process of decoding the binary format and managing the WebSocket connection can prove to be laborious and time-intensive.
+Thalos alleviates these challenges by assuming the responsibility of decoding the binary messages on behalf of developers.
+It seamlessly transforms the data received from the SHIP node into plain JSON format (and can be adapted to other prevalent formats if desired).
+JSON, being widely supported, has many implementations across various programming languages.
+
+## Contract ABIs
+
+Thalos simplifies the decoding of contract-specific data by maintaining an internal cache of contract ABIs. This cache, which is stored in Redis, allows for efficient retrieval of contract information. There are two ways in which the cache can be populated:
+
+1. **API Node**: Thalos can query the contract ABI through an API node, populating the cache with the retrieved information.
+2. **SHIP Node Messages**: Thalos keeps track of updates from the SHIP node, and when a contract is updated, it automatically updates its internal cache with the latest ABI information.
+
+## Redis - PubSub vs Stream
+
+Thalos utilizes Redis's publish/subscribe model known as Pub/Sub, which offers the following advantages:
+
+1. **Decoupled communication**: Pub/Sub enables loose coupling between publishers and subscribers. Publishers are unaware of the subscribers, and subscribers can join or leave independently, without affecting the publishing process.
+
+2. **Scalability**: Redis Pub/Sub is designed to handle high message throughput and efficiently distribute messages to a large number of subscribers. It can handle a high volume of messages without compromising performance.
+
+3. **Simple implementation**: Redis's Pub/Sub mechanism is easy to implement, involving basic operations such as subscribing to channels, publishing messages, and receiving messages. This simplicity facilitates development and integration with other systems.
+
+4. **Asynchronous communication**: Pub/Sub operates asynchronously, allowing publishers to send messages without waiting for subscribers to receive them. This decoupling is advantageous in scenarios where immediate response or synchronization is not necessary.
+
+However, these advantages can also be considered drawbacks in certain situations. Pub/Sub may encounter issues with message persistence:
+
+- **Message loss**: If clients disconnect and reconnect, they may miss messages published during their absence. Similarly, if a client cannot process messages quickly enough, it may not receive all messages.
+
+- **Immediate message consumption**: Pub/Sub requires subscribers to consume messages immediately upon publication. If a client is unable to read a message promptly, it is discarded.
+
+In contrast, Streams in Redis provide built-in message persistence. Messages are stored as entries in the stream and can be consumed at any time, even if they were published before the subscriber connected.
+This feature addresses the drawbacks of immediate message consumption and message loss associated with Pub/Sub.
+
+Currently, Streams are not implemented in Thalos. However, there are plans to consider implementing Streams in the future, based on the demand for such functionality. It's important to note that Streams introduce additional complexity to the client implementation.