Table of Contents

• Table of Contents
• Overview
  • Features Unavailable in Snapshotter Lite Node
  • Epoch Generation
  • Snapshot Generation
  • Snapshot Finalization
• Major Components
  • System Event Detector
  • Processor Distributor
  • RPC Helper
  • Core API
• Setup
  • Using Docker
  • Without Docker
• Monitoring and Debugging
  • Monitoring
  • Debugging
• For Contributors
• Case Studies
  • 1. Pooler: Case study and extending this implementation
    • Extending pooler with a Uniswap v2 data point
      • Step 1. Review: Base snapshot extraction logic for trade information
• Find us

Overview

A snapshotter peer as part of Powerloom Protocol does exactly what the name suggests: It synchronizes with other snapshotter peers over a smart contract running on Powerloom Prost chain. It follows an architecture that is driven by state transitions which makes it easy to understand and modify.

Because of its decentralized nature, the snapshotter specification and its implementations share some powerful features that can adapt to your specific information requirements on blockchain applications:

• Each data point is calculated, updated, and synchronized with other snapshotter peers participating in the network
• synchronization of data points is defined as a function of an epoch ID(identifier) where epoch refers to an equally spaced collection of blocks on the data source blockchain (for eg, Ethereum Mainnet/Polygon Mainnet/Polygon Testnet -- Mumbai). This simplifies the building of use cases that are stateful (i.e. can be accessed according to their state at a given height of the data source chain), synchronized, and depend on reliable data. For example,
  • dashboards by offering higher-order aggregate datapoints
  • trading strategies and bots
• a snapshotter peer can load past epochs, indexes, and aggregates from a decentralized state and have access to a rich history of data
  • all the datasets are decentralized on IPFS/Filecoin
  • the power of these decentralized storage networks can be leveraged fully by applying the principle of composability

Snapshotter Lite Node is a lightweight implementation of the Snapshotter Peer that can be used to build simpler use cases that do not require the full functionality of the Snapshotter Peer. Snapshotter Lite Node only has a single dependency on Python environment and hence has significantly lower resource requirements than the Snapshotter Peer. It is suitable for use cases where no aggregation is required and the data can be directly used from the base snapshots.

Architecturally, the Snapshotter Lite Node is similar to the Snapshotter Peer, it still has the same modular and highly configurable architecture, allowing for easy customization and seamless integration. It consists of three core components:

1. Main Snapshotter Codebase:

   • This foundational component defines all the essential interfaces and handles a wide range of tasks, from listening to epoch release events to distributing tasks and managing snapshot submissions.

2. Configuration Files:

   • Configuration files, located in the /config directory are linked to snapshotter-configs repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.

3. Compute Modules:

   • The heart of the system resides in the snapshotter/modules directory are linked to snapshotter-computes, where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.

The architecture has been designed to facilitate the seamless interchange of configuration and modules. To achieve this, we maintain these components in separate Git repositories, which are then integrated into the Snapshotter Peer using Git Submodules. As a result, adapting the system to different use cases is as straightforward as changing a Git branch, offering unparalleled flexibility and versatility.

For more information on using Git Submodules, please refer to the Git Submodules Documentation.

Features Unavailable in Snapshotter Lite Node

Snapshotter Lite Node is a lightweight implementation of the Snapshotter Peer and hence does not support the following features:

• Aggregation and data composition
• Redis Caching
• Worker Pool for parallel processing
• Preloading Architecture
• Epoch Processing state transitions
• Internal Status APIs
• Data Source Signalling

If you require any of these features, please consider using the Snapshotter Peer instead. If you are unsure about which implementation to use, please reach out to us on Discord and we will be happy to help you out.

Epoch Generation

An epoch denotes a range of block heights on the EVM-compatible data source blockchain, for eg Ethereum mainnet/Polygon PoS mainnet/testnet. This makes it easier to collect state transitions and snapshots of data on equally spaced block height intervals, as well as to support future work on other lightweight anchor proof mechanisms like Merkle proofs, succinct proofs, etc.

The size of an epoch is configurable. Let that be referred to as size(E)

• A trusted service keeps track of the head of the chain as it moves ahead, and a marker h₀ against the max block height from the last released epoch. This makes the beginning of the next epoch, h₁ = h₀ + 1

• Once the head of the chain has moved sufficiently ahead so that an epoch can be published, an epoch finalization service takes into account the following factors

  • chain reorganization reports where the reorganized limits are a subset of the epoch qualified to be published
  • a configurable ‘offset’ from the bleeding edge of the chain

and then publishes an epoch (h₁, h₂) by sending a transaction to the protocol state smart contract deployed on the Prost Chain (anchor chain) so that h₂ - h₁ + 1 == size(E). The next epoch, therefore, is tracked from h₂ + 1.

Each such transaction emits an EpochReleased event

event EpochReleased(uint256 indexed epochId, uint256 begin, uint256 end, uint256 timestamp);

The epochId here is incremented by 1 with every successive epoch release.

Snapshot Generation

Processors as configured in config/projects.json calculate snapshots for each task type based on the filtering or any criteria defined for snapshot generation against this epochId which corresponds to collections of state observations and event logs between the blocks at height in the range [begin, end].

The project ID is ultimately generated in the following manner:

snapshotter-lite/snapshotter/utils/snapshot_worker.py

Lines 40 to 60 in 79e251b

	def _gen_project_id(self, task_type: str, data_source: Optional[str] = None, primary_data_source: Optional[str] = None):
	"""
	Generates a project ID based on the given task type, data source, and primary data source.
	Args:
	task_type (str): The type of task.
	data_source (Optional[str], optional): The data source. Defaults to None.
	primary_data_source (Optional[str], optional): The primary data source. Defaults to None.
	Returns:
	str: The generated project ID.
	"""
	if not data_source:
	# For generic use cases that don't have a data source like block details
	project_id = f'{task_type}:{settings.namespace}'
	else:
	if primary_data_source:
	project_id = f'{task_type}:{primary_data_source.lower()}_{data_source.lower()}:{settings.namespace}'
	else:
	project_id = f'{task_type}:{data_source.lower()}:{settings.namespace}'
	return project_id

The snapshots generated are the fundamental data models on which higher-order aggregates and richer data points are built (By Full Nodes)

For situations where data sources are constantly changing or numerous, it is impractical to maintain an extensive list of them. In such cases, specific data sources need not be defined explicitly. You can just define a filtering criteria for the data source contract address and the snapshotter will automatically generate snapshots for all data sources that match the criteria.

For complex use cases with a lot of data requirements, it is recommended to use the Snapshotter Peer instead of the Snapshotter Lite Node because it employs a more efficient and scalable architecture.

Snapshot Finalization

All snapshots per project reach consensus on the protocol state contract which results in a SnapshotFinalized event being triggered.

event SnapshotFinalized(uint256 indexed epochId, uint256 epochEnd, string projectId, string snapshotCid, uint256 timestamp);

Major Components

The Snapshotter Lite Node consists of the following major components:

System Event Detector

System Event Detector, defined in system_event_detector.py, is the main entry point for Snapshotter Lite Node. It tracks events being triggered on the protocol state contract running on the anchor chain and calls appropriate classes to handle them.

Events being tracked by the system event detector are:

• EpochReleased - This event is emitted by the Protocol State Contract when an epoch is released. It is used to trigger the snapshot generation process.
• allSnapshottersUpdated - This event is emitted by the Protocol State Contract when a new snapshotter peer is added or removed from the network. It is used to enable or disable the snapshot generation process (Might be removed closer to mainnet).
• DailyTaskCompletedEvent - Each snapshotter lite peer needs to complete a daily task to be eligible for rewards. This event is emitted by the Protocol State Contract when a snapshotter lite peer completes its daily task, making it inactive for the rest of the day.
• DayStartedEvent - This event is emitted by the Protocol State Contract when a new day starts. This is used to re-enable the snapshot generation process for all snapshotter lite peers.

Processor Distributor

The Processor Distributor, defined in processor_distributor.py, acts upon the events received from the System Event Detector and distributes the processing tasks to the appropriate snapshot processors. It is also responsible for acting on allSnapshottersUpdated, DailyTaskCompletedEvent and DayStartedEvent events to manage the snapshot generation process.

snapshotter-lite/snapshotter/processor_distributor.py

Lines 300 to 345 in 79e251b

	async def process_event(
	self, type_: str, event: Union[
	EpochReleasedEvent,
	SnapshotFinalizedEvent,
	SnapshottersUpdatedEvent,
	DayStartedEvent,
	DailyTaskCompletedEvent,
	],
	):
	"""
	Process an event based on its type.
	Args:
	type_ (str): The type of the event.
	event (Union[EpochReleasedEvent, SnapshotFinalizedEvent, SnapshottersUpdatedEvent,
	DayStartedEvent, DailyTaskCompletedEvent]): The event object.
	Returns:
	None
	"""
	if type_ == 'EpochReleased':
	if self._snapshotter_enabled:
	await self._epoch_release_processor(event)
	else:
	self._logger.info('System is not active, ignoring released Epoch')
	elif type_ == 'allSnapshottersUpdated':
	if event.snapshotterAddress == to_checksum_address(settings.instance_id):
	self._snapshotter_enabled = event.allowed
	elif type_ == 'DayStartedEvent':
	self._logger.info('Day started event received, setting active status to True')
	self._snapshotter_active = True
	elif type_ == 'DailyTaskCompletedEvent':
	self._logger.info('Daily task completed event received, setting active status to False')
	self._snapshotter_active = False
	else:
	self._logger.error(
	(
	'Unknown message type {} received'
	),
	type_,
	)

RPC Helper

Extracting data from the blockchain state and generating the snapshot can be a complex task. The RpcHelper, defined in utils/rpc.py, has a bunch of helper functions to make this process easier. It handles all the retry and caching logic so that developers can focus on efficiently building their use cases.

Core API

This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in core_api.py.

The pooler-frontend that serves the Uniswap v2 dashboards hosted by the PowerLoom foundation on locations like https://uniswapv2.powerloom.io/ is a great example of a frontend specific web application that makes use of this API service.

Among many things, the core API allows you to access the finalized CID as well as its contents at a given epoch ID for a project.

The main endpoint implementations can be found as follows:

snapshotter-lite/snapshotter/core_api.py

Lines 237 to 289 in 79e251b

	@app.get('/data/{epoch_id}/{project_id}/')
	async def get_data_for_project_id_epoch_id(
	request: Request,
	response: Response,
	project_id: str,
	epoch_id: int,
	):
	"""
	Get data for a given project and epoch ID.
	Args:
	request (Request): The incoming request.
	response (Response): The outgoing response.
	project_id (str): The ID of the project.
	epoch_id (int): The ID of the epoch.
	Returns:
	dict: The data for the given project and epoch ID.
	"""
	if not settings.ipfs.url:
	response.status_code = 500
	return {
	'status': 'error',
	'message': f'IPFS url not set, /data API endpoint is unusable, please use /cid endpoint instead!',
	}
	try:
	data = await get_project_epoch_snapshot(
	request.app.state.protocol_state_contract,
	request.app.state.anchor_rpc_helper,
	request.app.state.ipfs_reader_client,
	epoch_id,
	project_id,
	)
	except Exception as e:
	rest_logger.exception(
	'Exception in get_data_for_project_id_epoch_id',
	e=e,
	)
	response.status_code = 500
	return {
	'status': 'error',
	'message': f'Unable to get data for project_id: {project_id},'
	f' epoch_id: {epoch_id}, error: {e}',
	}
	if not data:
	response.status_code = 404
	return {
	'status': 'error',
	'message': f'No data found for project_id: {project_id},'
	f' epoch_id: {epoch_id}',
	}
	return data

snapshotter-lite/snapshotter/core_api.py

Lines 293 to 340 in 79e251b

	@app.get('/cid/{epoch_id}/{project_id}/')
	async def get_finalized_cid_for_project_id_epoch_id(
	request: Request,
	response: Response,
	project_id: str,
	epoch_id: int,
	):
	"""
	Get finalized cid for a given project_id and epoch_id.
	Args:
	request (Request): The incoming request.
	response (Response): The outgoing response.
	project_id (str): The project id.
	epoch_id (int): The epoch id.
	Returns:
	dict: The finalized cid for the given project_id and epoch_id.
	"""
	try:
	data = await get_project_finalized_cid(
	request.app.state.protocol_state_contract,
	request.app.state.anchor_rpc_helper,
	epoch_id,
	project_id,
	)
	except Exception as e:
	rest_logger.exception(
	'Exception in get_finalized_cid_for_project_id_epoch_id',
	e=e,
	)
	response.status_code = 500
	return {
	'status': 'error',
	'message': f'Unable to get finalized cid for project_id: {project_id},'
	f' epoch_id: {epoch_id}, error: {e}',
	}
	if not data:
	response.status_code = 404
	return {
	'status': 'error',
	'message': f'No finalized cid found for project_id: {project_id},'
	f' epoch_id: {epoch_id}',
	}
	return data

The first endpoint in GET /last_finalized_epoch/{project_id} returns the last finalized EpochId for a given project ID and the second one is GET /data/{epoch_id}/{project_id}/ which can be used to return the actual snapshot data for a given EpochId and ProjectId.

These endpoints along with the combination of a bunch of other helper endpoints present in Core API can be used to build powerful Dapps and dashboards.

You can observe the way it is used in pooler-frontend repo to fetch the dataset for the aggregate projects of top pairs trade volume and token reserves summary:

try {
      response = await axios.get(API_PREFIX+`/data/${epochInfo.epochId}/${top_pairs_7d_project_id}/`);
      console.log('got 7d top pairs', response.data);
      if (response.data) {
        for (let pair of response.data.pairs) {
          pairsData7d[pair.name] = pair;
        }
      } else {
        throw new Error(JSON.stringify(response.data));
      }
    }
    catch (e){
      console.error('7d top pairs', e);
    }

Setup

There are multiple ways to set up the Snapshotter Lite Node. You can either use the Docker image or run it directly on your local machine. However, it is recommended to use the Docker image as it is the easiest and most reliable way to set up the Snapshotter Lite Node.

Using Docker

1. Install Docker on your machine. You can find the installation instructions for your operating system on the official Docker website.

2. Clone this repository using the following command:

   git clone https://github.com/PowerLoom/snapshotter-lite powerloom

   This will clone the repository into a directory named powerloom.

3. Change your working directory to the powerloom directory:

   cd powerloom

4. Run build.sh to start the snapshotter lite node:

   ./build.sh

   If you're a developer and want to play around with the code, instead of running build.sh, you can run the following command to start the snapshotter lite node:

   ./build-dev.sh

5. When prompted, enter $SOURCE_RPC_URL, SIGNER_ACCOUNT_ADDRESS, SIGNER_ACCOUNT_PRIVATE_KEY (only required for the first time), this will create a .env file in the root directory of the project.

6. This should start your snapshotter node and you should see something like this in your terminal logs

snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:17 | INFO | Current block: 2208370| {'module': 'EventDetector'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:18 | DEBUG | Set source chain block time to 12.0| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter enabled: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter slot is set to 1| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter enabled: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:21 | INFO | Snapshotter active: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 0|core-api          | February 5, 2024 > 15:10:22 | INFO | 127.0.0.1:59776 - "GET /health HTTP/1.1" 200 | {} 

7. To stop the node, you can press Ctrl+C in the terminal where the node is running or docker-compose down in a new terminal window from the project directory.

NOTE: It is recommended to run build.sh in a screen or tmux session so that the process continues running even after you close the terminal.

Without Docker

If you want to run the Snapshotter Lite Node without Docker, you need to make sure that you have Git, and Python 3.10.13 installed on your machine. You can find the installation instructions for your operating system on the official Python website.

1. Clone this repository using the following command:

   git clone https://github.com/PowerLoom/snapshotter-lite powerloom

   This will clone the repository into a directory named powerloom.

2. Change your working directory to the powerloom directory:

   cd powerloom

3. Run init.sh to start the snapshotter lite node:

   ./init.sh

4. When prompted, enter $SOURCE_RPC_URL, SIGNER_ACCOUNT_ADDRESS, SIGNER_ACCOUNT_PRIVATE_KEY (only required for the first time), this will create a .env file in the root directory of the project.

5. Your node should start in background and you should start seeing logs in your terminal.

6. To stop the node, you can run pkill -f snapshotter in a new terminal window.

Monitoring and Debugging

Monitoring

TODO - Enter Node dashboard screenshots and information here

Debugging

Usually the easiest way to fix node related issues is to restart the node. If you're facing issues with the node, you can try going through the logs present in the logs directory. If you're unable to find the issue, you can reach out to us on Discord and we will be happy to help you out.

For Contributors

We use pre-commit hooks to ensure our code quality is maintained over time. For this contributors need to do a one-time setup by running the following commands.

• Install the required dependencies using pip install -r dev-requirements.txt, this will set up everything needed for pre-commit checks.
• Run pre-commit install

Now, whenever you commit anything, it'll automatically check the files you've changed/edited for code quality issues and suggest improvements.

Case Studies

1. Pooler: Case study and extending this implementation

Pooler is a Uniswap specific implementation of what is known as a 'snapshotter' in the PowerLoom Protocol ecosystem. It synchronizes with other snapshotter peers over a smart contract running on the present version of the PowerLoom Protocol testnet. It follows an architecture that is driven by state transitions which makes it easy to understand and modify. This present release ultimately provide access to rich aggregates that can power a Uniswap v2 dashboard with the following data points:

• Total Value Locked (TVL)
• Trade Volume, Liquidity reserves, Fees earned
  • grouped by
    • Pair contracts
    • Individual tokens participating in pair contract
  • aggregated over time periods
    • 24 hours
    • 7 days
• Transactions containing Swap, Mint, and Burn events

Extending pooler with a Uniswap v2 data point

In this section, let us take a look at the data composition abilities of Pooler to build on the base snapshot being built that captures information on Uniswap trades.

Step 1. Review: Base snapshot extraction logic for trade information

Required reading:

• Snapshot Generation and

As you can notice in config/projects.example.json, each project config needs to have the following components

• project_type (unique identifier prefix for the usecase, used to generate project ID)
• projects (smart contracts to extract data from, pooler can generate different snapshots from multiple sources as long as the Contract ABI is same)
• processor (the actual compuation logic reference, while you can write the logic anywhere, it is recommended to write your implementation in snapshotter/modules folder)

There's currently no limitation on the number or type of usecases you can build using snapshotter. Just write the Processor class and pooler libraries will take care of the rest.

{
  "config": [{
      "project_type": "uniswap_pairContract_pair_total_reserves",
      "projects":[
        "0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc",
        "0xae461ca67b15dc8dc81ce7615e0320da1a9ab8d5",
        "0x0d4a11d5eeaac28ec3f61d100daf4d40471f1852",
        "0x3041cbd36888becc7bbcbc0045e3b1f144466f5f",
        "0xd3d2e2692501a5c9ca623199d38826e513033a17",
        "0xbb2b8038a1640196fbe3e38816f3e67cba72d940",
        "0xa478c2975ab1ea89e8196811f51a7b7ade33eb11"
        ],
      "processor":{
        "module": "pooler.modules.uniswapv2.pair_total_reserves",
        "class_name": "PairTotalReservesProcessor"
      }
    },
    {
      "project_type": "uniswap_pairContract_trade_volume",
      "projects":[
        "0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc",
        "0xae461ca67b15dc8dc81ce7615e0320da1a9ab8d5",
        "0x0d4a11d5eeaac28ec3f61d100daf4d40471f1852",
        "0x3041cbd36888becc7bbcbc0045e3b1f144466f5f",
        "0xd3d2e2692501a5c9ca623199d38826e513033a17",
        "0xbb2b8038a1640196fbe3e38816f3e67cba72d940",
        "0xa478c2975ab1ea89e8196811f51a7b7ade33eb11"
        ],
        "processor":{
          "module": "pooler.modules.uniswapv2.trade_volume",
          "class_name": "TradeVolumeProcessor"
        }
    }
  ]
}

If we take a look at the TradeVolumeProcessor class present at snapshotter/modules/computes/trade_volume.py it implements the interface of GenericProcessorSnapshot defined in snapshotter/utils/callback_helpers.py.

There are a couple of important concepts here necessary to write your extraction logic:

• compute is the main function where most of the snapshot extraction and generation logic needs to be written. It receives the following inputs:

• msg_obj (SnapshotProcessMessage instance, contains all the necessary epoch related information to generate snapshots)
• rpc_helper (RpcHelper instance to help with any calls to the data source contract's chain)
• anchor_rpc_helper (RpcHelper instance to help with any calls to the protocol state contract's chain)
• ipfs_reader (async IPFS client to read the data from IPFS)
• protocol_state_contract (protocol state contract instance to read the finalized snapshot CID or anything else from the protocol state contract required for snapshot generation)

Output format can be anything depending on the usecase requirements. Although it is recommended to use proper pydantic models to define the snapshot interface.

The resultant output model in this specific example is UniswapTradesSnapshot as defined in the Uniswap v2 specific modules directory: utils/models/message_models.py. This encapsulates state information captured by TradeVolumeProcessor between the block heights of the epoch: min_chain_height and max_chain_height.

Find us

• Discord
• Twitter
• Github
• Careers
• Blog
• Medium Engineering Blog