Arsyncer Implementation

Architecture

Technical Principle

Pull Block Information Concurrently

Arsyncer uses the golang ant concurrency library to fetch blocks within the specified height range from arweave in batches.

When the block data is acquired concurrently, it will be output in sequence according to the height of the block. The implementation of arsyncer is to specify the starting height and ending height of the block in each concurrent task, and construct a fixed-length array (len = end - start). Enter the specified position of the array, to implement the sequential output of block data.

Note: Each block must be successfully fetched on arweave. The arsyncer first pulls blocks from the arweave.net gateway, and if it fails, it uses goar's getBlockFromPeers() method to request blocks from all network nodes. If it still fails, the program will panic.

Pull Transactions Concurrently

After arsyncer obtains block data, it needs to pull transaction information concurrently from each block.

In order to achieve high-performance transaction synchronization, it is necessary to start multiple threads to pull transactions concurrently in multiple blocks.

Problem to be solved: Transactions in blocks that are pulled concurrently need to be output in order of block height.

After concurrently pulling transactions from a single block, it is also necessary to output the transactions in the order in which they are arranged in the blocks, which is the same as the concurrently pulling block information described above.

In order to output the transactions in the concurrently pulled blocks in the order of block height, arsyncer sets a global variable nextSubscribeTxBlock to determine whether the current thread should execute the transaction output of the current block height. nextSubscribeTxBlock is incremented by 1 each time the thread successfully executes the transaction output. The transaction pulled by each thread is at a certain block height. Only when the block height bound to the thread is equal to nextSubscribeTxBlock will the transaction be output, which ensures that the transaction is Output in block height order.

Note: arsyncer first pulls transactions from the arweave gateway. If it fails, it uses goar's getTxFromPeers() method to pull transactions from all nodes in the Arweave network. If it still fails, only the arId will be included in the returned struct. The purpose of this is that because Arweave has some lost transactions, returning an empty transaction will let the arsyncer user know that the transaction failed.

Transaction Filtering

Users can customize the filtering parameters of arsyncer, so that arsyncer will only return filtered transactions.

After arsyncer pulls the transactions in the block, it first puts these transactions into the filter for filtering, and then returns the filtered transactions.

Filters can currently be configured with three types of parameters, ownerAddress , target and tags.