# Crynux Network

Truly permissionless DeAI on GPU\@edge

[![](https://dcbadge.limes.pink/api/server/https://discord.gg/y8YKxb7uZk)](https://discord.gg/y8YKxb7uZk) [![X](https://img.shields.io/badge/@crynuxio-%23000000.svg?style=for-the-badge\&logo=X\&logoColor=white)](https://x.com/crynuxio) [![GitHub Org's stars](https://img.shields.io/github/stars/crynux-network?style=for-the-badge\&logo=github)](https://github.com/crynux-network)

Crynux Network is a decentralized AI compute network that turns edge GPUs into a shared cloud for modern LLM/VLM inference and fine-tuning tasks. Its vssML consensus protocol keeps the network permissionless and open to large-scale node participation while making malicious behavior detectable and punishable, bringing decentralized execution close to centralized-platform efficiency. On top of this compute layer, Crynux enables model and data assets that can support new AI-native DeFi applications.

### Truly Permissionless

The key component of Crynux is a robust consensus protocol that enables the permissionless joining and using of the decentralized network by millions.

The ability to identify and penalize all malicious behaviors ensures the ecosystem's sustainability while facilitating healthy growth in the long term.

The innovative [vssML](https://docs.crynux.io/system-design/verifiable-secret-sampling) technology significantly enhances network efficiency, rivaling centralized platforms while remaining decentralized and permissionless.

### Production-Ready AI Services Cloud, on Edge

As the foundation layer, Crynux Network is composed of the edge nodes, including home computers and mobile devices, who provide hardware to the network in exchange for tokens.

Applications could run tasks such as GPT text generation and Stable Diffusion image generation using various models hosted on the Crynux Network. The integration could be implemented in one-line of code using Crynux SDK.

Model developers use Crynux Network to train/fine-tune their models, and provide models as a service for applications and other developers, earning from the usage of their models.

Mobile devices could also be AI-enhanced by running larger and faster models beyond their current capabilities.

### DeFi Ecosystem built on the Tokenized Model and Data Assets

Building on top of the AI services, an innovative DeFi ecosystem could emerge around "Model Assets" and "Data Assets". All the current DeFi applications could be reimagined using the brand-new assets as their base assets.

For example, the developers of AI models can tokenize the models using Crynux, sharing the rewards from model usage with the model token holders.

Model tokens can be used as collateral in various DeFi applications. These applications can be deployed directly on the Crynux Blockchain or as modular L2 chains that connect to Crynux via cross-chain communication. Existing DeFi applications on other blockchains are also supported.

By utilizing the Blockchain, Zero-knowledge Proofs and Privacy Preserving Computation technologies, Crynux aims to build a completely decentralized and trustless infrastructure that is always accessible to everyone.

## Lithium Network

Lithium Network is the first mainnet release of Crynux Network. It turns Crynux into a production AI computing network where applications can use decentralized GPU nodes for LLM inference, vision-language model tasks, image generation, and model fine-tuning.

### Production AI Workloads

Applications can use Crynux for LLM inference, Vision Language Model tasks, image generation, and model fine-tuning through familiar APIs.

{% content-ref url="/pages/jtaLXzjcG1XwPkQU6Mhq" %}
[How to Run LLM using Crynux Network](/application-development/how-to-run-llm-using-crynux-network)
{% endcontent-ref %}

### AI Ecosystem Integration

Lithium integrates with the AI ecosystem developers already use. Through the OpenAI-compatible Crynux Bridge API, applications can connect Crynux to agent frameworks, tool-use workflows, LangChain, LangGraph, Hermes Agent, and Vision Language Model applications without rebuilding their stack around a new protocol.

{% content-ref url="/pages/d3Y2YDAR411um8SH0b5e" %}
[Vision Language Models (VLM)](/application-development/how-to-run-llm-using-crynux-network/vision-language-models)
{% endcontent-ref %}

{% content-ref url="/pages/aEuPw0R8uRvksFPmFXzp" %}
[Hermes Agent Integration](/application-development/how-to-run-llm-using-crynux-network/hermes-agent-integration)
{% endcontent-ref %}

{% content-ref url="/pages/tlRR30Y1MoplaOYmaQwF" %}
[Integration with LangChain & LangGraph](/application-development/how-to-run-llm-using-crynux-network/langchain)
{% endcontent-ref %}

### Delegated Staking

Delegated staking lets CNX holders participate in network rewards without running a node. Stake CNX to reliable node operators through [Crynux Portal](https://portal.crynux.io), support the compute providers you trust, and start sharing in the rewards generated by the network.

Read more about Lithium Network:

{% content-ref url="/pages/Zv2ZBLRLmw8LGZOk0obs" %}
[Lithium Network](/releases/lithium-network)
{% endcontent-ref %}

## Getting Started

### Start a Node

Download the package according to your platform, and follow the tutorials below:

<table><thead><tr><th width="131">Blockchain</th><th width="131">Platform</th><th width="261">Requirements</th><th data-type="content-ref">Download Link</th></tr></thead><tbody><tr><td>Base</td><td>Windows</td><td>Nvidia GPU with 8GB VRAM</td><td><a href="https://drive.google.com/uc?id=11xaq4TYq9N4cYP70o8OEjH4MaP1I3GhN&#x26;export=download">https://drive.google.com/uc?id=11xaq4TYq9N4cYP70o8OEjH4MaP1I3GhN&#x26;export=download</a></td></tr><tr><td>Base</td><td>Mac</td><td>M1/M2/M3 and later</td><td><a href="https://github.com/crynux-network/crynux-node/releases/download/v3.5.1/crynux-node-lithium-v3.5.1-base-mac-arm64-unsigned.dmg">https://github.com/crynux-network/crynux-node/releases/download/v3.5.1/crynux-node-lithium-v3.5.1-base-mac-arm64-unsigned.dmg</a></td></tr><tr><td>Near</td><td>Windows</td><td>Nvidia GPU with 8GB VRAM</td><td></td></tr><tr><td>Near</td><td>Mac</td><td>M1/M2/M3 and later</td><td></td></tr></tbody></table>

To start a node on your Windows computer:

{% content-ref url="/pages/QquuDgRAcmcQEouD8mKn" %}
[Start a Node - Windows](/node-hosting/start-a-node/start-a-node-windows)
{% endcontent-ref %}

If you are using Mac with Apple Silicon Chips (M1/M2/M3 and later):

{% content-ref url="/pages/P9OcReROZ6L0k0AyWb2W" %}
[Start a Node - Mac](/node-hosting/start-a-node/start-a-node-mac)
{% endcontent-ref %}

To start a node on cloud services based on Docker:

*Vast.ai*

{% content-ref url="/pages/dXGIWk0bz3awx1FdrDSX" %}
[Start a Node - Vast](/node-hosting/start-a-node/start-a-node-vast)
{% endcontent-ref %}

*Octa.space*

{% content-ref url="/pages/j120dUkR4XnvHUN5UJjw" %}
[Start a Node - Octa](/node-hosting/start-a-node/start-a-node-octa)
{% endcontent-ref %}

You can also start the node using Docker:

{% content-ref url="/pages/gGypoNA8XJ1TX4aGfQmE" %}
[Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker)
{% endcontent-ref %}

### Develop an application

If you are an application developer who want to utilize the AI abilities provided by the Crynux Network, follow the tutorial below:

{% content-ref url="/pages/xoQmd5cgfUHmklSz6yyC" %}
[Application Workflow](/application-development/application-workflow)
{% endcontent-ref %}

## Research

Checkout our latest research paper about Crynux Network here:

{% embed url="<https://www.researchgate.net/publication/380564678_A_Review_on_Decentralized_Artificial_Intelligence_in_the_Era_of_Large_Models>" %}

{% embed url="<https://www.researchgate.net/publication/377567611_Crynux_Hydrogen_Network_H-Net_Decentralized_AI_Serving_Network_on_Blockchain>" %}

## Links

Join the Crynux community on Discord:

[![](https://dcbadge.limes.pink/api/server/https://discord.gg/y8YKxb7uZk)](https://discord.gg/y8YKxb7uZk)

All the codes are open sourced at GitHub, feel free to submit issues and PRs:

{% embed url="<https://github.com/crynux-network>" %}

The Crynux Blog contains the technical explanations and our latest progress:

{% embed url="<https://blog.crynux.io>" %}

And follow us on Twitter:

{% embed url="<https://x.com/crynuxio>" fullWidth="true" %}


# Lithium Network

\[Jun 17, 2026] The First Mainnet

Lithium Network is the first mainnet release of Crynux Network.

Lithium turns Crynux from a public test network into a production AI computing network. Applications can use decentralized GPU nodes for LLM inference, vision-language model tasks, image generation, and model fine-tuning, while node operators earn CNX by providing compute capacity.

With the vssML consensus protocol upgrade, Lithium greatly reduces validation overhead and improves network efficiency while preserving a permissionless network where nodes can join at scale and malicious behavior remains detectable and punishable.

## Verifiable Secret Sampling

vssML, or Verifiable Secret Sampling for Machine Learning, is the core efficiency improvement in the consensus protocol behind Lithium. Previous consensus protocol sends every task to three nodes and compares the results, which provides strong security but consumes triple compute capacity. vssML validates only a secretly selected sample of tasks, and nodes must submit results before knowing whether they were selected for validation, so cheating still risks detection and slashing.

This greatly improves the efficiency of the whole network, bringing decentralized execution close to the speed of centralized platforms.

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

## Staking Score

Lithium introduces staking score to connect task probabilities with economic commitment. Nodes with more stake can receive more tasks and earn more rewards, while dishonest behavior puts more capital at risk. This makes the network harder to attack, rewards operators who commit long-term resources, and keeps task dispatching aligned with network security.

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

## Delegated Staking

Delegated staking lets CNX holders participate in network rewards without running their own hardware. Users can delegate stake to trusted node operators, similar to cloud mining or cloud compute rental, while operators with better GPUs, stronger uptime, and more reliable service can attract more delegated stake and earn more task income. This creates a new market where capital and computing power work together to grow the network.

## Latest LLM and VLM Support

Lithium expands Crynux's OpenAI-compatible AI service from text-only LLMs to both LLM and Vision Language Model workloads. Applications can use latest Hugging Face models, including examples such as `Qwen/Qwen3.6-27B` and `Qwen/Qwen3.5-9B`, while keeping the same chat completion workflow through Crynux Bridge.

{% content-ref url="/pages/jtaLXzjcG1XwPkQU6Mhq" %}
[How to Run LLM using Crynux Network](/application-development/how-to-run-llm-using-crynux-network)
{% endcontent-ref %}

{% content-ref url="/pages/d3Y2YDAR411um8SH0b5e" %}
[Vision Language Models (VLM)](/application-development/how-to-run-llm-using-crynux-network/vision-language-models)
{% endcontent-ref %}

## AI Ecosystem Integration

Lithium works with the tools developers already use. Through the OpenAI-compatible Crynux Bridge API, existing AI applications and agent frameworks can use Crynux as a decentralized model backend. Hermes Agent can connect to Crynux as a custom LLM provider, and LangChain or LangGraph applications can use Crynux through either `langchain-crynux` or standard OpenAI-compatible integrations.

{% content-ref url="/pages/aEuPw0R8uRvksFPmFXzp" %}
[Hermes Agent Integration](/application-development/how-to-run-llm-using-crynux-network/hermes-agent-integration)
{% endcontent-ref %}

{% content-ref url="/pages/tlRR30Y1MoplaOYmaQwF" %}
[Integration with LangChain & LangGraph](/application-development/how-to-run-llm-using-crynux-network/langchain)
{% endcontent-ref %}

## Multi-chain Architecture

Lithium launches Crynux as a multi-chain network. Crynux runs as dedicated Layer 2 blockchains, uses CNX bridged from the corresponding Layer 1 chain as the native gas token, and keeps the wallet experience EVM-compatible. Users can connect wallets, add networks, and move CNX between networks through Crynux Portal.

{% content-ref url="/pages/6Woj3nzIp5m05NKVuw1o" %}
[Multi-chain Architecture](/crynux-blockchains/multi-chain)
{% endcontent-ref %}


# Helium Network

\[Jan 30, 2024] Decentralized GPT Task Execution Engine

Helium Network adds the support of the LLM text generation tasks. The Crynux Network now supports running both the Stable Diffusion image generation tasks and the LLM text generation tasks.

OpenAI-compliant APIs are implemented through the Crynux Bridge. Official OpenAI SDKs can be directly used. And [most of the LLM models on the Huggingface](https://huggingface.co/models?pipeline_tag=text-generation\&sort=trending) are supported.

To get started, follow the guide below:

{% content-ref url="/pages/jtaLXzjcG1XwPkQU6Mhq" %}
[How to Run LLM using Crynux Network](/application-development/how-to-run-llm-using-crynux-network)
{% endcontent-ref %}

## AI Chatbot Application

An AI chatbot application has been released to demonstrate the abilities. The app provides a simple chat UI in the browser, and the text generation task is sent to the [Crynux Bridge](https://github.com/crynux-network/crynux-bridge) at the backend, and then sent to the Crynux Network for execution. The task fees are paid from the wallet inside the Crynux Bridge so that the users won't have to deal with the wallet themselves.

Try the application yourself at: [https://chat.crynux.io](https://chat.crynux.io/)

The source code of the application is located on the GitHub:

<https://github.com/crynux-network/chat-web>

## GPT Task Framework

A general framework to define and execute the GPT tasks is developed to be used in the Helium Network. A wide range of the common task types and configurations are supported. Just describe the task using JSON, and send it to the inference API:

* Unified task definition for various different large language model
* Apply model specific chat templates to input prompts automatically
* Model quantizing (INT4 or INT8)
* Fine grained control text generation arguments
* ChatGPT style response

To find out more about how to write a GPT task, go to the following page:

{% content-ref url="/pages/wbQGyrKGcOdpNuWNNgqU" %}
[Text-to-Text Task](/application-development/execute-tasks/text-to-text-task)
{% endcontent-ref %}

## GPT Task Verification On-chain

The consensus protocol now supports the validation of the GPT tasks.

To support the validation of the GPT tasks, the network will select 3 nodes with the same card model to run a single task, which ensures the results will be exactly the same on all the 3 nodes.

The node selection for stable diffusion tasks remain the same, which does not require the same cards, which gives the task more candidates to use and makes the network safer.

## Task Queue & Task Pricing

The order of the task execution is now determined by the task price set by the task creator. In general, tasks with higher prices will be executed first. Task with a lower priority will be put into the task queue to be executed later.

The order is not simply determined by the total price of a task. Instead, the task execution time is also taken into account to maximize the total income of a node in a fixed time range. The network will estimate a unit value in "CNX per second" of the task to determine the actual order of the task. The details can be found in the following docs:

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

{% content-ref url="/pages/QAZOlU2eJGyB4PPBPOXI" %}
[Task Pricing](/system-design/task-pricing)
{% endcontent-ref %}

## Quality of Service (QoS)

The Helium Network will calculate the Submission Speed score for each node. The score will be used in the following 2 scenarios:

* **Task fee distribution among the participating nodes**: the node that submits the result faster will get larger portion of the task fee.
* **Bad node kick out**: the node that has a lower score below the threshold will be forced to quit the network.

The details can be found in the following doc:

{% content-ref url="/pages/1kTI3u88IvoiLtjy3IkD" %}
[Quality of Service (QoS)](/system-design/quality-of-service-qos)
{% endcontent-ref %}

## Mac Support

The Crynux Node could now be started on Mac with Apple Silicon Chips (m1, m2 and m3 series). Both the Stable Diffusion and GPT tasks are supported. All the mac users could now join the network to earn CNX tokens.

To start a node on Mac, just follow the tutorial below:

{% content-ref url="/pages/P9OcReROZ6L0k0AyWb2W" %}
[Start a Node - Mac](/node-hosting/start-a-node/start-a-node-mac)
{% endcontent-ref %}

## Multi-chain Architecture

Crynux now supports applications and nodes running on multiple blockchains. Base and Near are now supported, and more will follow. Please visit the following document for details:

{% content-ref url="/pages/6Woj3nzIp5m05NKVuw1o" %}
[Multi-chain Architecture](/crynux-blockchains/multi-chain)
{% endcontent-ref %}


# Hydrogen Network

\[Nov 8, 2023] Decentralized Stable Diffusion Task Execution Engine

Hydrogen Network is the first testnet of the Crynux Network. Hydrogen Network implements an AI inference task execution engine that supports running the Stable Diffusion image generation tasks for the applications.

The computation power comes from a decentralized network of home computers and servers that are coordinated by a consensus protocol running on the Blockchain. The individuals who have the spared computation power could connect their devices to the network, exchanging the computation power for tokens by running the inference tasks for the applications.

## Inference API

To the applications, Hydrogen Network is an inference API service that could be used to generate images using the Stable Diffusion.

The application should prepare a wallet, to pay the tokens for the inference task. But other than that, the invocation of the API is no different than the invocation of a traditional API service on AWS. The decentralized execution process is completely invisible to the applications.

If you are an application developer, get started from here:

{% content-ref url="/pages/xoQmd5cgfUHmklSz6yyC" %}
[Application Workflow](/application-development/application-workflow)
{% endcontent-ref %}

#### Stable Diffusion Task Framework

A general framework to define and execute the Stable Diffusion tasks is developed to be used in the Hydrogen Network. A wide range of the common task types and configurations are supported. Just describe the task using JSON, and send it to the inference API:

* Unified task definition for Stable Diffusion 1.5, 2.1 and Stable Diffusion XL
* SDXL - Base + Refiner ([ensemble of expert denoisers](https://research.nvidia.com/labs/dir/eDiff-I/)) and standalone Refiner
* Controlnet and various preprocessing methods
* LoRA
* VAE
* Textual Inversion
* Long prompt
* Prompt weighting using [Compel](https://github.com/damian0815/compel)
* Auto LoRA and Textual Inversion model downloading from non-Huggingface URL

The following document gives more information on how to write a Stable Diffusion task:

{% content-ref url="/pages/1DaHVxBpm1NeVj27j6if" %}
[Text-to-Image Task](/application-development/execute-tasks/text-to-image-task)
{% endcontent-ref %}

And more examples can be found in the GitHub repository:

{% embed url="<https://github.com/crynux-network/stable-diffusion-task/tree/main/examples>" %}

#### The Image Generator

The Image Generator is a showcase application that provides a web interface (just like [`stable-diffusion-webui`](https://github.com/AUTOMATIC1111/stable-diffusion-webui)) for the users to generate images in the browser.

The users could select between different versions of the Stable Diffusion models, such as Stable Diffusion 1.5 and Stable Diffusion XL, and apply a LoRA model on it by specifying the download link of the LoRA model on Civitai.

Thanks to the Hydrogen Network, the application could be used on the devices that do not have a capable GPU integrated. If the browser exists, the Image Generator could be used.

Give it a try:

{% embed url="<https://ig.crynux.io>" %}

The Image Generator also serves as a reference implementation for the traditional centralized applications who want to integrate the inference API. The source code of the Image Generator is also hosted on GitHub:

**The backend:**

The Crynux Bridge is serving as the backend of the Image Generator. The bridge transparently handles the blockchain transaction processing and wallet signatures at the backend, and provides simple traditional APIs to the web UI:

{% embed url="<https://github.com/crynux-network/crynux-bridge>" %}

**The frontend:**

{% embed url="<https://github.com/crynux-network/ig-web>" %}

## Node Hosting

The contributor of the spared computation power could join the network by hosting a node on the local computer. The node could be easily started in just a few steps:

{% content-ref url="/pages/0kY58V9ydRvF5ABSeFMI" %}
[Start a Node](/node-hosting/start-a-node)
{% endcontent-ref %}

## Consensus Protocol

The consensus protocol ensures that all the malicious behaviors could be identified and panelized in the network. Thanks to the consensus protocol, the Hydrogen Network allows everyone to join freely as the computation power contributor, without asking for permissions.

The consensus protocol works by asking the node to stake certain amount of tokens before joining the network, and if the malicious behavior is detected from the node, the staked tokens will be slashed. By a calculation based on the probability, the attacking against the network will highly likely to cause the attacker to loose money rather than earn.

The malicious behaviors are discovered by dispatching the same task to 3 nodes randomly at the same time, and compare the results returned by the nodes on the Blockchain. A similarity score is used to overcome the randomness problem in the inference computation.

The consensus protocol is described in detail in the following doc:

{% content-ref url="/pages/8RH26qh31tfzLKVxeCfN" %}
[Consensus Protocol](/system-design/consensus-protocol)
{% endcontent-ref %}

## The Blockchain In Use

The Hydrogen Network is running on a private blockchain whose node can be accessed using the RPC endpoint:

```url
https://block-node.crynux.io/rpc
```

The reason for a private Blockchain is that public Blockchains with strong consensus protocol, such as Ethereum, is not fast enough alone to support the throughput of the Hydrogen Network, or any other networks of Crynux in the future. The solution will be a layer 2 scaling tech such as [ZK-Rollups](https://blockworks.co/news/zk-rollups-future-of-smart-contract-blockchains). We will be either using a generalized solution that is well known to the industry, or developing our own for better performance(under the limit of our use cases).

Our focus right now, however, is to support more features, such as the GPT tasks and training tasks. And we will launch a network on the public blockchain when the network has a rich set of features, and is ready to be used by a large number of applications. The layer 2 solution will be implemented when we are close to it.

When the test networks are running on the private blockchain, the test tokens are free to acquire from our community. The node providers are contributing their computation power for free in a belief of the open and democratized future. And their contributions are recorded by the private blockchain. We believe their efforts will be paid out eventually.

The test tokens are required for both starting a node, or calling the inference API. To get the test tokens, just join the Discord of Crynux and bind your wallet address using the bot:

{% embed url="<https://docs.crynux.ai/happyaigen#bind-the-wallet-address>" %}

The private blockchain in use in the Hydrogen Network is built using the [Frontier project](https://paritytech.github.io/frontier/), which contains an EVM running on top of the [Substrate Blockchain](https://substrate.io/). The Blockchain is Ethereum compatible, most of the existing tools for the Ethereum can be used directly.

The Hydrogen Network is coordinated by three smart contracts on the Blockchain:

<table><thead><tr><th width="186">Contract</th><th>Address</th></tr></thead><tbody><tr><td>Token</td><td>0x95E7e7Ed5463Ff482f61585605a0ff278e0E1FFb</td></tr><tr><td>Node</td><td>0xB0E9A451Ce0CC181EA9888C7B42BB8Ad90b73C78</td></tr><tr><td>Task</td><td>0xba2489a25A5f542877D3825Ab802651f28878C4a</td></tr></tbody></table>

The CNX token is just an standard ERC20 token. The tokens will be operated by the other contracts to implement the required functions.

The source code of the smart contracts is hosted on the GitHub:

{% embed url="<https://github.com/crynux-network/crynux-contracts>" %}

## The Relay In Use

The relay server could be accessed at:

```url
https://relay.h.crynux.io
```

The source code of the relay is hosted at:

{% embed url="<https://github.com/crynux-network/crynux-relay>" %}


# Network Architecture

Crynux Network Overview

Crynux Network is illustrated in the graph below:

<figure><img src="/files/La7q9cMWZ8lnQclx2uP4" alt=""><figcaption><p>The Crynux Network Architecture</p></figcaption></figure>

The core participants in the network are the **Nodes** and the **Applications**. The nodes provide computing power to the network, executing the Stable Diffusion image generation tasks from the applications, and receive tokens as the reward. The applications send the tasks to the nodes, paying with tokens, and get the images back.

Each of the nodes and the applications will start a **blockchain node**, and communicate with each other using it. The blockchain executes a consensus mechanism to make sure no one is cheating: the nodes could never use the fake images to get rewards, and the applications could never get the images without paying.

Beside the blockchain, the nodes and applications will also communicate through the **Relay**, to send data such as the task arguments and the images, which are too large to be stored on chain. These data are sent between the applications and the nodes directly, thus causes the data availability problem and the network reachability problem. The Relay stands between the nodes and the applications as a reliable intermediate to solve these problems.

## The Node

The node, once started, constantly monitors the blockchain for new tasks.

When a new task arrives from the blockchain, the node connects to the Relay to get the task arguments, such as the ID of the base model on Huggingface, and the URL of the LoRA model on Civitai. Then the node executes the task on the local hardware, producing the result images.

> A general framework has been developed to support most of the popular configurations in a Stable Diffusion image generation task, such as LoRA, Controlnet and Textual Inversion. The details on how to define a Stable Diffusion task can be found in the [Stable Diffusion Task introduction](/application-development/execute-tasks/text-to-image-task).

After the images are generated, the node executes the consensus protocol to proof to the blockchain that it is not cheating. The consensus protocol requires the node to calculate the p-hash of the images and disclose the p-hash on the blockchain. The p-hash is compared to the p-hashes generated by the other two nodes on-chain. If the p-hashes are the same (similar under a given threshold), the tokens are paid to the nodes. Otherwise the node will be slashed, the tokens staked by the node will be transferred to the incentivization pool, and the node will be kicked out of the network.

More about the running workflow of a task can be found in the task lifecycle introduction:

{% content-ref url="/pages/iPukPMh2AXLB0TPkE1Wt" %}
[Task Lifecycle](/system-design/task-lifecycle)
{% endcontent-ref %}

More explanations about the design of the consensus protocol can be found here:

{% content-ref url="/pages/8RH26qh31tfzLKVxeCfN" %}
[Consensus Protocol](/system-design/consensus-protocol)
{% endcontent-ref %}

The source code of the Node of the Crynux Network could be found in the repository [https://github.com/crynux-network/crynux-node](https://github.com/crynux-network/h-node).

## The Application

The applications are developed by the third-parties. The applications treat Crynux Network as an API service to enhance their abilities.

The application constructs the arguments of the Stable Diffusion/GPT task, and sends the hash of the task arguments to the blockchain to create the task, alongside with the tokens to be paid to the nodes.

After the blockchain confirmation, the application sends the task arguments to the relay, and then wait for the notification of task success on the blockchain.

Once the task success event has been emitted on the blockchain, the application could fetch the images/texts from the relay, and continue with its own subsequent business logics.

A showcase application, the [Image Generator](https://ig.crynux.io), has been developed to demonstrate the workflow. The showcase application is quite similar to the [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui), which is a web interface for the users to generate images using different models and text prompt. The difference is that, our application does not require the presence of a local GPU, thus could be used on any devices.

The Image Generator could be accessed at: <https://ig.crynux.io>.

The Image Generator is designed to be a traditional centralized application. The wallet is created and operated transparently by the application backend. To the end users, the blockchain, the tokens, are completely invisible, which makes it easier for the users to get started, comparing to a DApp where the users have to install the Metamask, and prepare a wallet with enough tokens before using the app.

However, the DApp is absolutely supported. the DApp could interact with the blockchain directly, sign the transaction with the user's wallet using Metamask. The workflow with the relay remains the same.

A detailed explanation of the application workflow is described here:

{% content-ref url="/pages/xoQmd5cgfUHmklSz6yyC" %}
[Application Workflow](/application-development/application-workflow)
{% endcontent-ref %}

The source code of the Image Generator could be found at:

Backend: <https://github.com/crynux-network/crynux-bridge>

Web UI: <https://github.com/crynux-network/ig-web>

## The Blockchain

The blockchain ensures that the consensus protocol is executed correctly. A list of all the nodes and their status are maintained. No central party is controlling the network. The nodes could join and quit the network freely at any time. As long as there are enough nodes, the network will operate normally.

Certain amount of tokens must be staked on-chain in order to join the network. If the node is found cheating, the staked tokens are slashed.

When a task is submitted by an application, the blockchain randomly selects 3 available nodes to execute the task. When the node discloses their image hashes on-chain, the blockchain compares the hashes of the 3 nodes, and slash the node whose result is different.

Crynux Network could be deployed on any blockchain system that supports the smart contracts. The source code of the smart contracts is in this repository:

<https://github.com/crynux-network/crynux-contracts>

## The Relay

The relay is actually a compromization on the decentralization of the network, in exchange for the network usability and efficiency.

Since the task arguments and the result images are too large to be stored directly on-chain, the data can only be stored at some other place that is accessible by the nodes. However, if the data becomes unavailable, due to for example, the storage system crashing, the nodes cannot retrieve the task arguments and thus cannot finish the task. Since the blockchain has no way to verify whether the data is accessible by the node or not, it can not tell whether the node is cheating, which is a situation that the system fails to handle. This is known as the data availability problem of the blockchain.

Ideally a decentralized storage network that is closely coupled with the blockchain could solve the problem. The data, once stored, can never be lost, and the smart contract could invoke a function such as `getData(hash)` to verify the integrity of these data. Unfortunately we don't have such a solution at this time.

The relay in Crynux Network stores the task arguments and the images, making them available to the relevant parties. The network should assume that the data stored in the relay is reliable and always accessible.

Given that the data is useless after the task is completed, the relay needs to keep the data available only during the task execution process.

Another problem is the network connectivity. Often the applications and the nodes are located under different subnets, which makes the direct connection impossible. This is also a well recognized problem in the P2P network.

The relay in Crynux Network is located at the public network, where everyone could access it. The relay serves as the intermediate channel for the nodes and applications to communicate.

The source code of the relay could be found at:

<https://github.com/crynux-network/crynux-relay>


# Consensus Protocol

Decentralize the Infrastructure

The consensus protocol in a decentralized system ensures the integrity of the network, allowing permissionless participation without the possibility of fraudulent activities. The consensus protocol is the most important component in any decentralized system, since it is where "decentralization" comes from.

The hardest part about the consensus protocol design is that **Everyone Could Be Malicious**. If a leader is selected, the leader could be malicious. If validators are chosen, the validators could be malicious. The goal of every participant is the same: maximizing the income while at the same time reducing the cost as much as possible. If vulnerability exists, even a minor one, it will be exploited, resulting in the losses for the honest participants. This situation can compel these participants to exit the network, leading to a network downfall eventually.

For example, consider a scenario in Crynux Network, where a malicious node submits a random image to the network without actually performing any computation. If we rely on the user to detect this fraud, allowing them to withhold payment until they have verified the result, it opens a loophole. A dishonest user could exploit this by denying all payments, effectively using the network services without paying.

The consensus protocol in the Crynux Network aims to verify the correctness of a task's output based on its input arguments. Additionally, it ensures that the node submitting the correct result gets the payment.

The consensus protocol must be enforced by the blockchain, which eliminates the need for a centralized authority. This decentralized approach safeguards against potential abuse of power by removing the temptation for any single party to cheat, given their control.

## Verifiable Secret Sampling (VSS) of Validation Tasks

When the application sends a task to the blockchain, the blockchain will decide whether to validate the task based on a pre-defined probability (e.g., 10%). If chosen for validation, the task is sent to 3 nodes for independent execution. The computation results from all 3 nodes will be cross-validated on-chain to prevent cheating. If a node submits a fake result, it will be punished by slashing its staked tokens on the blockchain.

The random sampling result should be kept secret from nodes until they submit their computation results. If a node knows in advance whether a task will be validated, it could cheat by submitting fake results for tasks that won't be validated.

Hiding the random sampling process from the public while keeping it verifiable on-chain is a challenging task, given that all data on the blockchain is public and transparent. Crynux achieved this using a combination of VRF (Verifiable Random Function) and ZKP (Zero-Knowledge Proofs).

Comparing to validating all the tasks on chain, the secret task sampling significantly enhances network efficiency, rivaling centralized platforms while remaining decentralized and permissionless by effectively preventing fraudulent activities. Please find the details of the sampling algorithm in the following document:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

## Cross Validation by Multiple Result Comparison

### Deterministic Execution of AI Tasks

For 3-task cross validation to function correctly, the execution of AI tasks needs to be deterministic. This means that, regardless of the GPU types, hardware, or operating systems used across different nodes, identical task parameters should consistently yield the same results.

The non-deterministic behaviors observed in current AI computations stem from two main sources:

#### Hardware

For different types of GPUs, the non-determinism observed in AI computations can be pinpointed to specific nuances like floating-point precision disparities, execution strategies, and the tailored optimizations within math libraries and drivers.

The architectural distinctions across different GPUs can introduce slight precision variations, particularly noticeable when leveraging reduced precision formats (e.g., FP16 or BF16) to enhance computational speed. This approach, while efficient, may result in minor discrepancies after numerous calculations, a common scenario in deep learning tasks.

Moreover, GPUs exhibit unique processing strategies, where the scheduling and load management of parallel computations can differ, affecting the determinism due to the non-associative and non-distributive nature of floating-point arithmetic under rounding errors.

Additionally, Nvidia's continuous refinement of its CUDA toolkit, including specialized libraries like `cuDNN` for deep learning, introduces optimization-driven differences. These libraries are engineered to maximize efficiency and performance on hardware through sophisticated algorithmic choices and task partitioning strategies, which, while largely beneficial, can subtly influence the consistency of computational results.

#### Framework

The frameworks commonly used in AI computation, such as `PyTorch`, introduce non-deterministic behaviors through their handling of random number generation and the use of inherently non-deterministic algorithms. This randomness is pivotal in various stages, from initializing neural network weights to shuffling data for training.

Moreover, certain `PyTorch` operations and layers, especially those executed on GPUs, are designed with non-deterministic algorithms for efficiency, such as specific convolution implementations and atomic operations in parallel reductions. Although these features enrich `PyTorch`'s flexibility and performance, they also sow the seeds of variability in outcomes, making exact reproducibility a challenge despite the ability to set global random seeds. This nuanced dance between enhancing performance and managing unpredictability underscores the complexity of achieving deterministic results in AI models developed with `PyTorch`.

{% hint style="info" %}
More details about the non-deterministic behavior of `PyTorch` can be found in its [docs](https://pytorch.org/docs/main/notes/randomness.html) and [discussions](https://github.com/pytorch/pytorch/issues/15359).
{% endhint %}

Despite the aforementioned challenges, Crynux succeeded in achieving deterministic execution for specific AI tasks on identical GPU models. This was accomplished by thoroughly dissecting the frameworks to capture and control the random numbers, alongside substituting the non-deterministic algorithms with their deterministic counterparts.

Restricting the execution of validation tasks to the same GPU models curtails network performance by narrowing the pool of eligible candidates for a task, and it compromises network security by diminishing the number of honest nodes, thereby making it easier for attackers to launch Sybil attacks with fewer counterfeit nodes.

By tolerating slight discrepancies in computation results and employing specific similarity comparison methods, it becomes feasible to permit the execution of certain tasks across all GPU models, thereby optimizing both performance and security while still facilitating cross-validation of tasks.

### Inference Tasks

Image generation tasks, including text-to-image and image-to-image, can be executed across a variety of GPU models. However, text generation tasks utilizing Large Language Models (LLMs) are restricted to identical model types. Further information is provided in the document below:

{% content-ref url="/pages/V53v3lIBfiAKNYz3WobE" %}
[Inference Task Validation](/system-design/consensus-protocol/inference-task-validation)
{% endcontent-ref %}

### Training/Fine-tuning Tasks

The Stable Diffusion fine-tuning tasks can be executed across a variety of GPU models. Read more in the document below:

{% content-ref url="/pages/VoGhFHXWRRakI9NRk5or" %}
[Training/FT Task Validation](/system-design/consensus-protocol/training-ft-task-validation)
{% endcontent-ref %}

## Random Number Generation on the Blockchain

Generating random numbers on the blockchain is then a critical step to the security of the whole network. Ethereum 2.0 has `prevrando`, which can be used as the source of the random number. On the other blockchains, the block hash of the last confirmed block is usually used. More advanced (and complex) methods exist such as the Verifiable Random Functions. Strictly speaking, however, none of these methods are safe enough in our scenario.

The attack one could perform, given that the result validation is effective, is for an attacker to host more nodes by himself, and try to have two or more of his own nodes selected for a single task. In which case the attacker could submit two identical fake results to cheat the blockchain.

If an attacker is hosting the blockchain node (and producing the blocks) himself, the last block hash, or `prevrando`, or the selection of the VRF, is known to him before the `CreateTask` transaction has been confirmed by the next block. This leaves a chance for the attacker to find out if his nodes are selected for a task ahead of time.

The attacker could then reject the `CreateTask` transactions in which it can not cheat, i.e. not having two or more of his own nodes selected in the task.

By carefully constructing and organizing more adjacent blocks, the attacker could even control who will be selected in the next task. Note that this does not apply to the VRF method, where the source of the randomness is not from the blockchain. Which is immune to this kind of attack, but introduces other risks which we will not cover in this article.

Considering that to make this attack **practical**, the attacker must control a significant large number of nodes in the whole network by himself. The Crynux Network chooses to ignore this problem and uses the `prevrando` on the supported blockchains, and uses the last block hash on other blockchains.

## Staking based Penalization

Nodes are required to stake a certain amount of tokens on the blockchain before joining the network. If a node exhibits malicious behavior, its tokens will be slashed.

Given the VSS task validation scheme above, it is then a calculation of the required number of tokens to stake to prevent attacking attempts. If the staked tokens are not enough, the attacker can still make profit even if some tokens will be slashed.

### Sybil Attack

The attacker will start as many malicious nodes as he could. All the malicious nodes will do one thing: submitting the identical fake result for every task they received.

1. If the task is not selected for validation, the attacker gets the reward for free.
2. If the task is selected for validation:
   1. If 2 or 3 nodes from the same attacker are selected for the task, the attacker gets the rewards for free.
   2. If there is only 1 node from the attacker is selected, the attacker loses staked tokens.

{% @mermaid/diagram content="graph LR
task(Task) --> sampling{Selected for validation?}
sampling -- No --> attacker((Attacker gets rewards))
sampling -- Yes --> twonode{2 or 3 nodes from the same attacker?}
twonode -- Yes --> attacker
twonode -- No --> slash((Attacker penalized))
classDef node fill: #00B0F0, stroke: none, color: #fff" %}

### Expectation of the Rewards from Sybil Attack

The probability of an attacker getting more than 2 nodes of himself selected in a task could be calculated as:

$$
p(h, d) = \frac{ C\_d^2 \* C\_h^1 + C\_d^3}{C\_{d+h}^3}
$$

Where $$h$$ is the number of the honest nodes, and $$d$$ is the number of the dishonest nodes the attacker starts.

And the expectation of the rewards from sybil attack is given by:

$$
E =  (1 - r) \* k + r \* (p \* k - (1-p) \* s)
$$

Where $$r$$ is the sampling rate given in VSS, $$k$$ is the price of the task, and $$s$$ is the number of the staked tokens for a node.

By increasing the number of the staked tokens $$s$$, we could decrease the expectation $$E$$ down to zero or even below. If $$E$$ is below zero, there is no benefit to attack the system by starting more malicious nodes. The attacking will highly likely cause the attacker to lose money rather than earn.

The safety of the network now depends on the calculated value of the amount of the staked tokens $$s$$. Given a network size (the number of the total nodes in the network), and a target ratio of the malicious nodes (under which the network is safe), the probability of a successful attack $$p$$ is then fixed. Setting $$E$$ to zero, the amount of the staked tokens required for a single node $$s$$ is determined by:

$$
s = \frac{(1-r) \* k  + r \* p \* k}{r \* (1-p)}
$$

### Identifying the Validation Task Groups

An attacker could identify the validation task group by decrypting and comparing the task parameters received by all the malicious nodes. If parameters are identical for two adjacent tasks from the same application, they likely belong to the same validation group. The attacker might then return identical fake results to gain rewards without effort.

However, identifying task groups doesn't provide the attacker with additional advantages in a Sybil attack. The attacker already receives rewards by submitting two identical fake results for all tasks, without needing to identify the validation groups.

{% @mermaid/diagram content="graph TD
task(Task) --> sampling{Selected for validation?}
sampling -- No --> nogroup(No group identified)
nogroup --> fake(Submit fake result)
fake --> attacker((Attacker gets rewards))
sampling -- Yes --> twonode{2 or 3 nodes from the same attacker?}
twonode -- Yes --> groupidentify(Group identified)
groupidentify --> fake
twonode -- No --> nogroup2(No group identified)
nogroup2 --> fake2(Submit fake result)
fake2 --> slash((Attacker penalized))
classDef node fill: #00B0F0, stroke: none, color: #fff" %}

Another attack method involves submitting fake results only when the validation group is detected, while behaving normally otherwise. The network cannot identify this behavior.

{% @mermaid/diagram content="graph TD
task(Task) --> sampling{Selected for validation?}
sampling -- No --> nogroup(No group identified)
nogroup --> normal(Execute normally)
normal --> noattack(( No attack ))
fake --> attacker((Attacker gets rewards))
sampling -- Yes --> twonode{2 or 3 nodes from the same attacker?}
twonode -- Yes --> groupidentify(Group identified)
groupidentify --> fake(Submit fake result)
twonode -- No --> nogroup2(No group identified)
nogroup2 --> normal2(Execute normally)
normal2 --> noattack2(( No attack ))
classDef node fill: #00B0F0, stroke: none, color: #fff" %}

For this attack to be effective, all malicious nodes must be equipped with GPUs, significantly increasing the cost compared to the Sybil attack mentioned earlier.

Given that only a small portion of the network's tasks will be validated (targeted by this attack), and the chance of an attacker discovering the identification groups is even smaller, the attacker would need to control a significant portion of the nodes, making the attack impractical with low potential income. This scenario is therefore excluded in the consensus protocol.

Additionally, although the task parameters may be identical, the attacker cannot be certain that the tasks are part of the same validation group. There's still a possibility that they are independent tasks. If the attacker submits two fake results, they will be penalized.

## Task Error and Timeout

Given that the network is a loosely coupled P2P system composed of home computers and laptops, we cannot assume the nodes are reliable. A node may lose contact with the network at any moment, even if it is still marked as available or executing a task on the blockchain.

The applications are also unreliable. Tasks submitted might be entirely inexecutable, such as combining the SD1.5 base model with an SDXL LoRA model.

### Task Error Reporting

When an exception occurred during the task execution on the node, if the exception is not recoverable, the node will report the error to the blockchain.

The error reporting will also be cross validated in a validation task group to prevent malicious behaviors from the nodes. If one of nodes reports error while the other two send the normal computation results, it will be penalized.

Crynux Network allows model downloads through an external link. However, network issues may occur during the download. It's challenging to determine if these issues affect all three nodes or if they are temporary.

To prevent mistakenly slashing honest nodes, reporting errors should only be used when the node is certain it's an issue with the task arguments, not a network problem. All other cases should be handled by the timeout mechanism below.

If errors are reported by the nodes, the task will be aborted. And the task fee will be refunded. The small cost of the transaction fee will prevent the applications from sending the invalid task parameters intentionally.

### Task Cancellation on Timeout

The consensus protocol requires the submission of the commitments of all the 3 nodes. If a selected node goes offline before submitting the commitment to the blockchain, the other 2 nodes will have to wait for an unlimited time, which is not tolerable for both the nodes and the applications.

The timeout mechanism is introduced to solve this problem. After a pre-defined period, all the 3 nodes, and the application, are allowed to submit the request to cancel the task on the blockchain. Once submitted, the blockchain will abort the task immediately.

### Timeout Attack under VSS

The timeout mechanism introduces a new vulnerability to the network. An attacker could exploit this by returning fake results only when task validation groups are found. In other scenarios, rather than executing the tasks, the node could simply wait for the timeout to avoid penalties. And similar to a Sybil attack, the attacker can execute this attack without needing GPUs.

{% @mermaid/diagram content="graph TD
task(Task) --> sampling{Selected for validation?}
sampling -- No --> nogroup(No group identified)
nogroup --> timeout(Wait for timeout)
timeout--> escape(( Attacker escaped ))
fake --> attacker((Attacker gets rewards))
sampling -- Yes --> twonode{2 or 3 nodes from the same attacker?}
twonode -- Yes --> groupidentify(Group identified)
groupidentify --> fake(Submit fake result)
twonode -- No --> nogroup2(No group identified)
nogroup2 --> timeout2(Wait for timeout)
timeout2 --> escape2(( Attacker escaped ))
classDef node fill: #00B0F0, stroke: none, color: #fff" %}

[Similar to the discussion earlier](#identifying-the-validation-task-groups), the risk of this attack is low and therefore it is excluded from the consensus protocol.


# Inference Task Validation

## Stable Diffusion Image Generation

Stable Diffusion image generation tasks are allowed to be executed using a combination of all types of GPU models.

The non-deterministic behavior in the Stable Diffusion pipeline is minimized to keep the result images as close as possible. There will still be minor differences when executed on different GPU models due to technical limitations, such as [this](https://github.com/pytorch/pytorch/issues/87992).

The [Perceptual Hash](https://apiumhub.com/tech-blog-barcelona/introduction-perceptual-hashes-measuring-similarity/), or pHash, is further adopted to calculate the image similarity. The node submits the pHash of the images to the blockchain, and the blockchain calculates the [Hamming Distance](https://en.wikipedia.org/wiki/Hamming_distance) between two pHashes as the similarity score. Two images with the similarity score under a given threshold are considered the same image.

Raising the threshold heightens the likelihood of successful attacks with counterfeit images. This risk can be mitigated by increasing the amount of tokens required for staking.

## LLM Text Generation

LLM text generation tasks are limited to be execute on the same GPU models.

In LLM text generation tasks, the words are generated one after another, each output word will be used as the input for the next word. This means the error will be accumulated during the whole generation process. If two different words are generated on two different cards in the middle of a text sequence, the rest parts of the sequence will highly likely to be completely different. As a result, no differences could be tolerated in the LLM tasks.

By managing the random number generation and swapping out the non-deterministic algorithms in the text creation process, Crynux ensures consistent execution of LLM tasks across identical GPU models.

When joining the network, nodes will declare their card models to the blockchain, which will then pair nodes with identical card models for specific LLM tasks. It's important to note that submitting false card model information to the blockchain offers no advantage to the nodes and will result in penalties.


# Training/FT Task Validation

## Stable Diffusion Model Fine-tuning

The SD model fine-tuning task could be executed using a combination of all types of GPU models.

Rather than directly validating the result models, multiple images are produced using the models and a random prompt (seed) provided by the blockchain. [The method](https://docs.crynux.ai/system-design/consensus-protocol/inference-task-validation#stable-diffusion-image-generation) for validating image generation tasks is applied to assess the similarity between images created by two models.

The average similarity score of these images serves as the measure of similarity between the two models. And models with the similarity score under a given threshold is considered the same model.


# Verifiable Secret Sampling

Reduce the Task Validation Overhead

To prevent any malicious node from sending fake computing results to the network for free rewards, Crynux Network validates task results by selecting three random nodes to independently execute the same task and compare their outputs to detect any cheating.

However, using triple computing power for every task wastes resources, cutting network capacity by two-thirds and increasing wait times. This inefficiency is compounded if one node is slow due to network issues, further lengthening task execution times.

Instead of validating every task, an effective approach is to randomly sample a small portion of tasks for validation. As long as the node is unaware of whether its task will be validated, it cannot execute any attacks.

The random sampling process must be enforced by the blockchain using smart contracts to ensure no participants can manipulate the selection or alter the sampling rate for personal gain. The challenge then is to conceal the sampling results from the public despite the transparency of blockchain data.

The complexity of the problem further increases because the task arguments and results are too large to store directly on the blockchain. This makes it difficult for the blockchain to verify the correctness and delivery of certain data, increasing the risk of participants cheating.

Crynux implements a Verifiable Secret Sampling algorithm using the [VRF (Verifiable Random Function)](https://en.wikipedia.org/wiki/Verifiable_random_function) and [ZKP (Zero-Knowledge Proofs)](https://en.wikipedia.org/wiki/Zero-knowledge_proof).

When a task is sent to the blockchain, it determines secretly whether the task will be validated based on a pre-defined sampling rate. If the task is selected for validation, three tasks will be sent to the nodes.

Nodes cannot know the sampling results before submitting their computations to the blockchain. Once the task execution is complete, the sampling results are revealed on-chain for verification. This ensures the sampling were generated prior to the task and prevents manipulation.

Comparing to validating all the tasks on chain, the secret task sampling significantly enhances network efficiency, rivaling centralized platforms while remaining decentralized and permissionless by effectively preventing fraudulent activities.

The algorithm's detailed description will be provided in the next sections.

{% hint style="info" %}
The sequence diagrams in this document focus solely on consensus-related parameters. For a comprehensive list of all parameters, please refer to the task lifecycle documentation:

[Task Lifecyle](/system-design/task-lifecycle)
{% endhint %}

## Task Creation

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant R as DA/Relay
Participant N as Node

```
activate A
A ->> A: Generate task ID
Note over A,A: Task GUID: a unique identifier<br/>Nonce: a random number<br/>Task ID Commmitment: hash(Task GUID, Nonce)
A ->> B: Create task
activate B
Note over A,B: Task ID Commitment<br/>Nonce
deactivate A

B ->> B: Generate sampling seed
Note over B,B: Sampling Seed: a random number
activate A
B -->> A: Return sampling seed
Note over A,B: Sampling Seed

par Upload task parameters

    B ->> B: Select node or enqueue until a node is selected

    B ->> A: Event: TaskStarted
    activate A
    Note over A,B: Task ID Commitment<br />Selected Node
    deactivate B
    A ->> R: Upload task parameters
    activate R
    note over A,R: Encrypted Task Parameters
    deactivate A
    R ->> B: Update Merkle root
    note over B,R: Merkle Root
    R -->> A: Return hash and Merkle proof
    activate A
    note over A,R: Hash of Encrypted Task Parameters<br/>Merkle Proof
    deactivate R
    A ->> B: Notify task parameters uploaded
    activate B
    note over A,B: Task ID Commitment<br/>Hash of Encrypted Task Parameters<br/>Merkle Proof
    deactivate A
        
    B ->> N: Event: TaskParametersUploaded
    note over B,N: Task ID Commitment<br/>Hash of Encrypted Task Parameters<br/>Selected Node
    deactivate B

and Send validation tasks
    activate A
    A ->> A: Generate Sampling Number<br/>Using VRF
    note over A,A: Sampling Number: VRF(Sampling Seed, Private Key)
    opt Last digit of the Sampling Number is 0
        loop Repeat 2 times
            A ->> B: Create validation task and upload the task parameters
            deactivate A
        end
    end    
end
" %}
```

### Task GUID and Task ID Commitment

To initiate a task, the application creates a unique `Task GUID`.

The `Task GUID` for each task is obscured by generating a `Task ID Commitment`. This commitment is a hash of the real `Task GUID` combined with a random number `Nonce`.

For three tasks within the same validation group, each `Task ID Commitment` is derived from the same `Task GUID` but uses different random numbers, making them appear unrelated in public data. Only the `Task ID Commitment` is sent to the blockchain, and is used to identify the task during the whole task lifecycle. This prevents the nodes from knowing whether the task will be validated or not.

After execution, the application will reveal the real `Task GUID` on the blockchain. This allows the blockchain to validate task relationships, preventing the application from fraudulently grouping unrelated tasks. This ensures honest nodes are not penalized.

### Secret Selection of Validation Tasks

The sampling process begins when the application sends the transaction to create a task on the blockchain. Upon receiving the task, the blockchain generates a random number to be used as the `Sampling Seed` for the VRF and return it to the application.

The application uses VRF locally to generate the `Sampling Number`, using the `Sampling Seed` and its own private key as the VRF inputs. With a 10% sampling ratio, the task will be selected for validation if the `Sampling Number` ends in 0.

The `Sampling Number` is only known to the application, since no one else knows its private key.

The application cannot cheat on the `Sampling Number` either, as the `Sampling Seed` is fixed on the blockchain. Additionally, the public key of the application is set before the task and is known to the blockchain.

### Sending the Validation Tasks

If the task is not selected for validation, the application will simply stop and await for the notification to upload the `Task Parameters`. However, if the task is selected, the application must send two additional tasks with the same `Task Parameters` for validation purposes.

{% hint style="info" %}
The application will not get the computing result if the validation tasks are not submitted or if they are submitted with inconsistent parameters. The blockchain will verify the correctness of the validation tasks before allowing the application to get the computation result. More details are provided in the next section.
{% endhint %}

The fees charged for the validation tasks will be refunded once the task is completed. This extra charge ensures that the validation tasks appear identical to regular tasks, preventing nodes from distinguishing them based on the fees.

If an application sends tasks infrequently, such as a human sending tasks to the blockchain via a browser-based DApp and Metamask, the node can monitor the user's address for new tasks. If no additional tasks are sent from the same address in a short period, the task is likely non-validation. However, if tasks are sent frequently, it becomes impossible for the node to determine if a task is for validation.

The higher probability of guessing correctly increases the chance of a node performing a successful [statistical attack](/system-design/consensus-protocol#expectation-of-the-income). Increasing the required amount of staking could solve this issue. A task mixer can also be designed to combine tasks from all applications before dispatching them to the nodes, thereby concealing the origin of the tasks from the nodes.

### Uploading Task Parameters to the DA/Relay

When a task is created on the blockchain, the blockchain will try to select a node based on the task's criteria. If no node is available, the task is added to a queue. Once a new node becomes available, the task is retrieved from the queue and executed. Details of this process are outlined in the following document:

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

When a node is selected for the task, the blockchain will emit `TaskStarted` event to notify the application to upload `Task Parameters`.

Knowing which node will execute the task, the application encrypts the `Task Parameters`, such as the prompt and image size, using the node's public key. It then sends the `Encrypted Task Parameters` to the DA/Relay service and gets the `Merkle Proof` in return.

{% hint style="info" %}
The DA/Relay will save the data and make it publicly available. A [Merkle Tree](https://en.wikipedia.org/wiki/Merkle_tree) is generated for a collection of recently submitted data, and the Merkle Root is sent to the smart contract on the blockchain. The application receives the Merkle Proof for the data. Using the correct Merkle Proof, the blockchain can verify data availability under a specific hash, confirming its existence and public accessibility.
{% endhint %}

The `Encrypted Task Parameters` can only be decrypted by the assigned node. Nodes cannot decrypt the parameters of other tasks, making it impossible to determine if a task will be validated by comparing task parameters.

After sending the `Encrypted Task Parameters` to the DA/Relay and obtaining the `Merkle Proof`, the application notifies the blockchain by sending the `Merkle Proof` to the blockchain. The blockchain verifies the `Merkle Proof`, and emits `TaskParametersUploaded` event to notify the node to start the execution.

The verification of the `Merkle Proof` only makes sure **some** **data** is uploaded to the DA/Relay and is claimed to be the encrypted `Task Parameters` for the given `Task ID Commitment`, it doesn't guarantee the correctness of the `Task Parameters`. The task parameters may still be inconsistent across tasks in a validation group, may be in an invalid format, or may be undecryptable by the node at all.

If the `Task Parameters` are invalid, the node will report error to the blockchain, and the task will be aborted. The task fee is returned to the application, but the transaction fee is still charged, which will stop the application from sending the invalid `Task Parameters` intentionally.

The consistency of the `Task Parameters` across the validation group will be verified later using Zero-Knowledge Proof (ZKP).

## Task Execution

Upon receiving the `TaskParametersUploaded` event from the blockchain, the node retrieves the `Encrypted Task Parameters` from the DA/Relay service, decrypts them, and executes the task on the local GPU.

After the computation is finished, the node will calculate the `Sim Hash` of the result, and send the `Sim Hash` to the blockchain. Then the node should wait for a future notification from the blockchain. If the wait exceeds the timeout period, the node may abort the task. The task fee will then be refunded to the application.

The node cannot send the task result to the DA/Relay service at this stage. If the result is transmitted, the application could retrieve it prematurely and disrupt subsequent processes. The blockchain lacks mechanisms to identify and penalize the application in such scenarios.

When the `Sim Hash` is sent, the blockchain emits `TaskResultReady` event to notify the application. If the `Task Parameters` are invalid, the node will report error to the blockchain, and the blockchain will emit `TaskErrorReported`. In both cases, the task enters the **Result Validation** stage.

## Result Validation

When the `TaskResultReady` or `TaskErrorReported` event is received, the application should proceed with one of two different strategies based on the previously generated `Sampling Number`.

### Tasks Require Validation

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node

```
activate B
loop Until events from all three tasks are received
    alt
        B ->> A: Event: TaskResultReady
        activate A
        Note over A,B: Task ID Commitment<br />Sim Hash
    else
        B ->> A: Event: TaskErrorReported
        activate A
        Note over A,B: Task ID Commitment
        deactivate B
    end
    A ->> A: Wait for other validation tasks
end 

    A ->> B: Finish task
    activate B
    Note over A,B: Task ID Commitment<br/>Task GUID<br/>Sampling Number<br/>VRF Proof<br/>Hash of Task Parameters<br/>ZK Proof
    deactivate A
    B ->> B: Validate task
    
    break Task error reported
        B ->> A: Event: TaskAborted
    end
    
    alt Sim Hash identical
        B ->> N: Event: TaskValidated
        Note over B,N: Task ID Commitment
    else One Sim Hash different
        B ->> N: Event: NodeSlashed
    else All Sim Hash different
        B ->> A: Event: TaskAborted
    end

    deactivate B
" %}
```

If the `Sampling Number` ends with 0, the task requires validation. The application will wait for the submission of all the 3 `Sim Hash` (or error reporting) on the blockchain, and then disclose the relationship of the tasks, and other relevant proofs for the blockchain to validate:

#### Sampling Number Validation

The application sends the `VRF Proof` and the `Sampling Number` to the blockchain, and the blockchain validates the `Sampling Number` using the `VRF Proof` and the `Application Public Key`. This ensures that the `Sampling Number` is generated from the on-chain `Sampling Seed` and the application's private key.

{% @mermaid/diagram content="graph LR
sn\[Sampling Number] --> |VRF Proof| ss\[Sampling Seed]
ss --> bc\[Blockchain]
sn --> |VRF Proof| pk\[Application Public Key]
pk --> bc" %}

If the `VRF Proof` is valid, the blockchain will verify whether the `Sampling Number` ends in 0. If valid, the blockchain confirms that the task was genuinely selected during the secret task selection.

#### Task Relationship Validation

The application sends the `Task ID Commitment` of all three tasks along with the actual `Task GUID` to the blockchain. The blockchain validates the `Task ID Commitment` with the previously uploaded `Nonce`, ensuring they are generated from the same `Task GUID`.

The task relationship validation ensures the application does not send misleading information to the blockchain, such as the combination of three irrelevant tasks, which could cause honest nodes to be penalized.

#### Task Parameters Validation

This validation ensures the `Task Parameters` provided by the application are consistent across all three nodes. Inconsistent parameters given to different nodes result in different `SimHash` being submitted to the blockchain, causing honest nodes to be penalized.

The validation is implemented using Zero-Knowledge Proofs. The application sends the hash of the `Task Parameters`, along with a `ZK Proof` to the blockchain.

{% @mermaid/diagram content="graph TD
tp\[Task Parameters] --> |ZK Proof| hash\[Hash of Task Parameters]
hash --> |Compare| ot\[Other Tasks]
tp --> |ZK Proof| etp\[Encrypted Task Parameters]
pk\[Node Public Key] --> |ZK Proof| etp
etp --> |ZK Proof| hetp\[Hash of Encrypted Task Parameters]
hetp --> |Merkle Proof| bc\[Blockchain]" %}

The `ZK Proof` is constructed to use the plain text `Task Parameters` as the private input, the `Public Key` of the node as the public input, and publicly outputs the hash of the `Task Parameters` and the hash of the `Encrypted Task Parameters`. A valid `ZK Proof` ensures that:

1. The `Task Parameters` has the given hash value `Hash of Task Parameters`.
2. The `Task Parameters` are encrypted using the `Node Public Key`, and the cipher text has the given hash value `Hash of Encrypted Task Parameters`.

If the `ZK Proof` is valid, the blockchain verifies that the three `Hash of Encrypted Task Parameters` match those provided by the application in the **Task Creation** stage, which are previous verified using the `Merkle Proof`. This ensures that the `Task Parameters` referenced in the `ZK Proof` are identical to those actually executed by the nodes.

The blockchain then compares the three `Hash of Task Parameters` to ensure they are identical. This prevents the application from submitting inconsistent `Task Parameters` to different nodes, which could lead to the penalization of the honest nodes.

There is no way to penalize the application for submitting inconsistent `Task Parameters` for different tasks in a validation group. The application could always escape from the penalization by not sending the validation transaction, and simply waiting for the timeout of the tasks.

The application will not send inconsistent tasks intentionally though, since there is a small cost of the transaction fee, and there is no benefit at all.

#### Task Result Validation

The blockchain uses three `Sim Hash` values to verify task results. If one node submits a `Sim Hash` significantly different from the other two, it will be penalized. If all the three `Sim Hash` are different, the task will be aborted.

### Tasks Do Not Require Validation

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node

```
alt
    B ->> A: Event: TaskResultReady
    activate A
    Note over A,B: Task ID Commitment<br />Sim Hash
else
    B ->> A: Event: TaskErrorReported
    Note over A,B: Task ID Commitment
end

A ->> B: Finish task
activate B
Note over A,B: Task ID Commitment<br/>Sampling Number<br/>VRF Proof
deactivate A
B ->> B: Validate Sampling Number

break Task error reported
    B ->> A: Event: TaskAborted
end

B ->> N: Event: TaskValidated
Note over B,N: Task ID Commitment
    
deactivate B" %}
```

If the `Sampling Number` does not end in 0, which means the task does not require validation, the validation will be much simpler.

The [Relationship Validation](#task-relationship-validation) and the [Parameters Validation](#task-parameters-validation) are both skipped. Only the `Sampling Number` needs validation to ensure the task doesn't require result validation.

The [Sampling Number Validation](#sampling-number-validation) remains unchanged, with the exception that the blockchain must ensure the `Sampling Number` does not end in 0.

## Result Retrieval

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node
Participant D as DA/Relay

```
B ->> N: Event: TaskValidated
activate N
Note over B,N: Task ID Commitment
N ->> D: Send task result
activate D
Note over N,D: Encrypted Task Result
deactivate N
D ->> B: Update Merkle root
Note over B,D: Merkle Root
D -->> N: Return Merkle proof
activate N
Note over N,D: Hash of Encrypted Task Result<br/>Merkle Proof
deactivate D
N ->> B: Report result uploaded
deactivate N
activate B
Note over B,N: Task ID Commitment<br/>Hash of Encrypted Task Result<br/>Merkle Proof<br/>ZK Proof

B ->> B: Validate proofs
B ->> B: Settle task fee
B ->> A: Event: TaskSuccess 
activate A
Note over A,B: Task ID Commitment
deactivate B
A ->> D: Get task result
deactivate A
activate D
D -->> A: Return task result
note over A,D: Encrypted Task Result
deactivate D
```

" %}

After the node receives the `TaskValidated` event, it encrypts the task result using the public key of the application, and sends the cipher text to the DA/Relay service.

After receiving the `Merkle Proof`, the node generates a `ZK Proof` and submits it to the blockchain.

{% @mermaid/diagram content="graph TD
tr\[Task Result] --> |ZK Proof| sh\[Sim Hash]
tr --> |ZK Proof| etr\[Encrypted Task Result]
pk\[Application Public Key] --> |ZK Proof| etr
etr --> |ZK Proof| hetr\[Hash of Encrypted Task Result]
sh --> bc\[Blockchain]
hetr --> |Merkle Proof| bc
" %}

The `ZK Proof` uses the `Task Result` as the private input, the `Application Public Key` as the public input, and publicly outputs the `Sim Hash`, and the `Hash of Encrypted Task Result`. A valid `ZK Proof` makes sure:

1. The `Task Result` has the given `Sim Hash`.
2. The `Task Result` is encrypted using the `Application Public Key`, and the cipher text has the given hash `Hash of Encrypted Task Result`.

The blockchain verifies the `Sim Hash` against the previously submitted one from the node to pass the result validation. This ensures that the `Task Result` produces the correct `Sim Hash`.

The blockchain verifies the `Hash of Encrypted Task Result` using the `Merkle Proof` against the `Merkle Root` submitted by the DA/Relay. This ensures that the correct cipher texts have been uploaded to the DA/Relay service and are accessible to the application.

If all the validation passes, the blockchain distributes the task fee to all participating nodes based on their [QoS scores](/system-design/quality-of-service-qos) and notifies the application to retrieve the task result. Once the application retrieves the result, the task is marked as completed.


# Task Lifecycle

From the task creation to the task success

## Overview

Tasks are central to the Crynux Network. Each application use case is represented as a different task. Applications interact with the network by sending various tasks, and nodes are responsible solely for executing these tasks.

A task consists of a group of `Task Parameters`. For instance, in a Stable Diffusion image generation task, the `Task Parameters` might include:

* **Text Prompt:** The description or scene you want to generate.
* **Image Size:** Dimensions of the generated image.
* **Guidance Scale:** Controls the strength of the prompt on the image generation.
* **Controlnet Image:** An image used as the reference in the Controlnet.

Here is a concrete example of the `Task Parameters` of an SD image generation task:

```json
{
    "version": "2.0.0",
    "base_model": {
        "name": "stabilityai/sdxl-turbo"
    },
    "prompt": "best quality, ultra high res, photorealistic++++, 1girl, desert, full shot, dark stillsuit, "
              "stillsuit mask up, gloves, solo, highly detailed eyes,"
              "hyper-detailed, high quality visuals, dim Lighting, ultra-realistic, sharply focused, octane render,"
              "8k UHD",
    "negative_prompt": "no moon++, buried in sand, bare hands, figerless gloves, "
                       "blue stillsuit, barefoot, weapon, vegetation, clouds, glowing eyes++, helmet, "
                       "bare handed, no gloves, double mask, simplified, abstract, unrealistic, impressionistic, "
                       "low resolution,",
    "task_config": {
        "num_images": 9,
        "steps": 1,
        "cfg": 0
    },
    "lora": {
        "model": "https://civitai.com/api/download/models/178048"
    },
    "controlnet": {
        "model": "diffusers/controlnet-canny-sdxl-1.0",
        "image_dataurl": "data:image/png;base64,12FE1373...",
        "preprocess": {
            "method": "canny"
        },
        "weight": 70
    },
    "scheduler": {
        "method": "EulerAncestralDiscreteScheduler",
        "args": {
            "timestep_spacing": "trailing"
        }
    }
}
```

The lifecycle of a task consists of four stages: **Task Creation**, **Task Execution**, **Result Validation**, and **Result Retrieval**.

In the **Task Creation** stage, the application initiates a task by sending a transaction to the blockchain. The `Task Parameters` are not sent to the blockchain due to size constraints. Instead, the application sends the task's consensus-related metadata to the blockchain to create the task. Once the task is dispatched to a node, the application encrypts the `Task Parameters` using the node's public key and sends them to the DA/Relay.

To ensure successful cross-validation for the nodes, the blockchain may require the application to send two additional tasks with identical `Task Parameters`. The application will be unable to obtain the computation results if the additional tasks are not sent.

In the **Task Execution** stage, the node is notified about the task by the blockchain. It then receives the task metadata from the blockchain, fetches the `Task Parameters` from the DA/Relay, and executes the task locally.

Upon a successful run, the node computes the similarity hash of the result and submits it to the blockchain for validation.

In the **Result Validation** stage, the application either completes the task directly or waits for other validation tasks to complete, based on the VSS selection result. In both scenarios, it must submit the relevant proofs to the blockchain to initiate validation. The application will not be able to get the computation result if the proofs are not submitted.

The blockchain will perform the validation. Once validation is complete, the task proceeds to the **Result Retrieval** stage.

The node will upload the actual computation result to the DA/Relay, and claim the task fee from the blockchain by proving the availability of the computation result to the application. The node will get the task fee immediately when the validation completes on-chain. No interaction from the application is required.

After the validation, the application is notified to download the result from the DA/Relay, and the task is completed.

The subsequent sections detail all the stages. This document focuses on listing the interaction steps between components, the parameters required for each step, and the possible status and return values. Explanations on why a parameter is required are given in other documents. For the validation related parameters, refer to the following document:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

For the node criteria related parameters, refer to the following document:

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

And the pricing related parameters:

{% content-ref url="/pages/QAZOlU2eJGyB4PPBPOXI" %}
[Task Pricing](/system-design/task-pricing)
{% endcontent-ref %}

The task lifecycle is modeled and implemented as the [Finite State Machine (FSM)](https://en.wikipedia.org/wiki/Finite-state_machine) in the smart contract. All the states and possible transitions are given in the document below:

{% content-ref url="/pages/OQilzRJn2d1IYfRJPv3u" %}
[Task State Transitions](/system-design/task-lifecycle/task-state-transitions)
{% endcontent-ref %}

## Task Creation

### Create Task On-Chain

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain

```
A ->> B: Create Task
activate B
Note over A,B: Task ID Commitment<br/>Nonce<br/>Model ID<br/>Minimum VRAM<br/>Required GPU<br/>Task Fee<br/>Task Version

break Task Fee == 0 or Nonce is not unique
    B -->> A: Tx reverted
end

B ->> A: Event: TaskCreated
activate A
Note over A,B: Sampling Seed
deactivate B

A ->> A: Generate Sampling Number<br/>Using VRF
opt Last digit of the Sampling Number is 0
    loop Repeat 2 times
        A ->> B: Create the validation task and upload Task Parameters
    end
end    
deactivate A
activate B

alt Node available
    B ->> A: Event: TaskStarted
    Note over A,B: Task ID Commitment<br/>Selected Node
else No node available
    B ->> A: Event: TaskQueued
    Note over A,B: Task ID Commitment
    loop New node available
        break Node available for task
           B ->> B: Select node
           B ->> A: Event: TaskStarted
           Note over A,B: Task ID Commitment<br/>Selected Node
           deactivate B
        end
    end
end" fullWidth="false" %}
```

The application starts a task by signing a transaction, invoking the smart contract to create the task on the Blockchain.

The application must set the task fee it is willing to pay in the `value` field of the transaction.

The transaction might be reverted, due to several reasons:

* The transaction value is not set (task fee is not paid).
* The Nonce has already been used before.

If the transaction is confirmed, the application receives a `Sampling Seed`. The application then uses the VRF algorithm with this `Sampling Seed` to generate a `Sampling Number`. If the last digit of the `Sampling Number` is 0, the application should create two additional tasks to form a task validation group. The details of the task validation are described in the following document:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

For each of the tasks, the blockchain will attempt to locate a suitable node that is available to execute the task. If such a node is found, the task starts immediately. Otherwise, the task is added to the queue and `TaskQueued` event is emitted. When a new node becomes available, it will retrieve the task from the queue and begin execution. In both cases, the blockchain emits a `TaskStarted` event when the task begins, including the node's address. Details of this process are outlined in the following document:

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

### Upload Task Parameters

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant R as DA/Relay
Participant N as Node

```
activate B
B ->> A: Event: TaskStarted
activate A
Note over A,B: Task ID Commitment<br />Selected Node
deactivate B

A ->> R: Upload task parameters
activate R
note over A,R: Encrypted Task Parameters
deactivate A
        
R ->> B: Update Merkle root
note over B,R: Merkle Root
R -->> A: Return hash and Merkle proof
activate A
note over A,R: Hash of Encrypted Task Parameters<br/>Merkle Proof
deactivate R
        
A ->> B: Notify task parameters uploaded
activate B
note over A,B: Task ID Commitment<br/>Hash of Encrypted Task Parameters<br/>Merkle Proof
deactivate A
        
break Validation failed
    B -->> A: Validation failed
end

B ->> N: Event: TaskParametersUploaded
note over B,N: Task ID Commitment<br/>Hash of Encrypted Task Parameters<br/>Selected Node
deactivate B" fullWidth="false" %}
```

Upon receiving the `TaskStarted` event, the application should encrypt the `Task Parameters` using the node's public key and send them to the DA/Relay. The DA/Relay will update the `Merkle Root` to the blockchain for validation, and return the `Merkle Proof` to the application.

The application sends the hash and `Merkle Proof` to the blockchain. The blockchain verifies the proof against the `Merkle Root` submitted by the DA/Relay, ensuring the `Task Parameters` are uploaded. It then emits the `TaskParametersUploaded` event to notify the node to start execution.

## Task Execution

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node
Participant R as DA/Relay

```
B ->> N: Event: TaskParametersUploaded
Note over B,N: Task ID Commitment<br/>Hash of Encrypted Task Parameters<br/>Selected Node
activate N
N ->> R: Get task parameters
activate R
Note over N,R: Hash of Encrypted Task Parameters
deactivate N

R -->> N: Return the encrypted task parameters
activate N
Note over N,R: Encrypted Task Parameters
deactivate R

N ->> N: Execute the task locally

break Task not executable
    N ->> B: Report task error
    activate B
    B ->> A: Event: TaskErrorReported
    deactivate B
end

N ->> N: Calculate the task score
N ->> B: Submit the score
activate B
Note over N,B: Task ID Commitment<br/>Task Score
B ->> A: Event: TaskScoreReady
Note over A,B: Task ID Commitment<br/>Task Score
deactivate B

break Waiting exceeds timeout
    N ->> B: Abort task
    deactivate N
    activate B
    B ->> A: Event: TaskAborted
    deactivate B
end
```

" fullWidth="false" %}

When the node receives the `TaskStarted` event, it will start to execute the task locally.

The execution starts by fetching the `Encrypted Task Parameters` from the DA/Relay. After the parameters are received, the node decrypts them using its own private key, and starts the execution.

The first step is to download the models. The node will check the local existence of the models specified in the `Task Parameters`. If the models are not cached locally, they will be downloaded from the network.

If there are network issues during the download, the node will retry the download several times until the timeout period is reached. The task will be cancelled by the node if the timeout is reached.

If the model download link is confirmed to be invalid, such as a 404 response from Civitai, the node will report error to the blockchain.

The task is then sent to the execution engine of the node. If the execution engine finds out that the task is misconfigured, such as an SDXL LoRA model combined with an SD1.5 base model, it will report the error to the blockchain.

When the task has finished execution successfully, the node has the final computation result such as the images. It will calculate the score of the result, and then submit it to the blockchain.

The blockchain will emit `TaskScoreReady` event to the application, and wait for the application to perform the validation process.

The node will also wait for the task validation. If validation isn't completed within the timeout period, the node might abort the task to accept new ones instead of waiting indefinitely.

## Result Validation

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node

```
alt
    B ->> A: Event: TaskScoreReady
    activate A
    Note over A,B: Task ID Commitment<br />Task Score
else
    B ->> A: Event: TaskErrorReported
    Note over A,B: Task ID Commitment
else
    B ->> A: Event: TaskAborted
    Note over A,B: Task ID Commitment
end

alt Validation not required
    A ->> B: Validate single task
    activate B
    Note over A,B: Task ID Commitment<br/>Sampling Number<br/>VRF Proof
    deactivate A
    B ->> B: Validate Sampling Number
    break Validation failed
        B -->> A: Validation error 
    end
    break Task error reported
        B ->> A: Event: TaskAborted
    end

    B ->> N: Event: TaskValidated
    Note over B,N: Task ID Commitment
    
    deactivate B
else Validation required
    activate A
    activate B

    loop Until events from all three tasks are received
        A ->> A: Wait for other validation tasks
        alt
            B ->> A: Event: TaskScoreReady
            Note over A,B: Task ID Commitment<br />Task Score
        else
            B ->> A: Event: TaskErrorReported
            Note over A,B: Task ID Commitment
        else
            B ->> A: Event: TaskAborted
            Note over A,B: Task ID Commitment
        end            
    end

    deactivate B

    A ->> B: Validate task group
    activate B
    Note over A,B: Task ID Commitments<br/>Task GUID<br/>Sampling Number<br/>VRF Proof<br/>Hash of Task Parameters<br/>ZK Proof
    deactivate A
    B ->> B: Validate task
    break Invalid proofs
        B -->> A: Validation error 
    end
    break Task error reported
        B ->> A: Event: TaskAborted
    end
    
    alt Task Score identical
        B ->> N: Event: TaskValidated
        Note over B,N: Task ID Commitment
    else One Task Score different
        B ->> N: Event: NodeSlashed
    else All Task Score different
        B ->> A: Event: TaskAborted
    end

    deactivate B
end
" fullWidth="false" %}
```

Upon receiving the `TaskResultReady` event, the application's response varies based on the need for task validation:

### Task does not Require Validation

If the task does not require validation, the application should send the "Complete Task" transaction directly to the blockchain, including proofs of the `Sampling Number`.

The blockchain will then validate the proofs. If the validation passes, the blockchain will emit `TaskValidated` event to the node to notify it to disclose the actual computation result. The transaction will fail if the validation does not pass.

For more information on the validation process, please see the following document:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

### Task Requires Validation

If validation is required, the application should wait for the `TaskResultReady` event from the other two tasks in the validation group. Once all three tasks have submitted their similarity hashes, the application will disclose their relationship for blockchain validation.

There are more validations to be performed by the blockchain, comparing to the validation of tasks that do not require validation. For more information on the validation process, please see the following document:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

If the validation passes, the blockchain will emit `TaskValidated` event to all the three nodes. The transaction will fail if the proofs provided by the application are invalid.

If the `Sim Hash` are different across the nodes, if two of them are identical, the other node will be slashed. If all three `Sim Hash`are different, the task will be aborted.

## Result Retrieval

{% @mermaid/diagram content="sequenceDiagram
Participant A as Application
Participant B as Blockchain
Participant N as Node
Participant D as DA/Relay

```
B ->> N: Event: TaskValidated
activate N
Note over B,N: Task ID Commitment
N ->> D: Send task result
activate D
Note over N,D: Encrypted Task Result
deactivate N
D -->> B: Update Merkle root
note over B,D: Merkle Root
D -->> N: Return Merkle proof
activate N
Note over N,D: Hash of Encrypted Task Result<br/>Merkle Proof
deactivate D
N ->> B: Report result uploaded
activate B
Note over B,N: Task ID Commitment<br/>Hash of Encrypted Task Result<br/>Merkle Proof<br/>ZK Proof
deactivate N
B ->> B: Validate proofs
break Validation failed
    B -->> N: Validation failed
end
B ->> B: Settle task fee
B ->> A: Event: TaskSuccess 
activate A
Note over A,B: Task ID Commitment
deactivate B
A ->> D: Get task result
deactivate A
activate D
D -->> A: Return task result
note over A,D: Encrypted Task Result
deactivate D
```

" fullWidth="false" %}

Upon receiving the `TaskValidated` event, the node can upload the computation result to the DA/Relay service and obtain the task fee by proving to the blockchain that the upload was correct. The proving is implemented using ZKP, the details are described in the following section of the documentation:

{% content-ref url="/pages/pxDilxJ3o0ya0wNKpnTu" %}
[Verifiable Secret Sampling](/system-design/verifiable-secret-sampling)
{% endcontent-ref %}

The computation result is encrypted with the application's public key before being sent to the DA/Relay, ensuring that only the application can decrypt and access the actual result.

Once the node submits the proofs to the blockchain, and they are verified, the blockchain will transfer the task fee to the node and emit a `TaskSuccess` event to the application. The application can then retrieve the computation result from the DA/Relay service, completing the task.


# Task State Transitions

Task as Finite State Machine (FSM)

## State Transition Graph

Relay owns task deadlines. A task can be aborted only by the deadline rule for its current state, or by its creator while it is still `Queued`. A selected node cannot abort a task.

The task state transition graph is given below. Deadline transitions to `EndAborted` are listed separately after the graph:

```mermaid
stateDiagram-v2
  state "Queued" as q
  state "Started" as s
  state "Parameters Uploaded" as pu
  state "Error Reported" as er
  state "Score Ready" as sr
  state "Validated" as v
  state "Group Validated" as gv
  state "End Group Refund" as egr
  state "End Success" as es
  state "End Group Success" as egs
  state "End Invalidated" as ei
  state "End Aborted" as ea

  [*] --> q: App - create task
  [*] --> s: App - create task
  q --> s: Blockchain - start task
  s --> pu: Relay - report parameters uploaded
  pu --> sr: Node - submit task score
  pu --> er: Node - report task error
  sr --> ei: App - validate task group
  sr --> v: App - validate single task
  sr --> gv: App - validate task group
  gv --> egs: Relay - report result uploaded
  sr --> egr: App - validate task group
  er --> ea: App - validate single task<br/>App - validate task group
  er --> ei: App - validate task group
  v --> es: Relay - report result uploaded
  ea --> ea: App - validate task group
  es --> [*]
  egr --> [*]
  egs --> [*]
  ei --> [*]
  ea --> [*]
  
```

## Deadline Transitions

Only the row matching the current state applies. A state transition invalidates the preceding deadline.

| Current state                     | Deadline                                                                                                                                   | Transition                                                               | Fee handling                                                              | Node handling                                      |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------- | -------------------------------------------------- |
| `Queued`                          | Ordinary GPT and Stable Diffusion: `CreateTime + queue_timeout_seconds`. Stable Diffusion fine-tuning: `CreateTime + 3 minutes + Timeout`. | `Queued → EndAborted`, reason `TaskAbortTimeout`                         | Full refund                                                               | No selected node and no penalty                    |
| `Started` or `ParametersUploaded` | `StartTime + Timeout`                                                                                                                      | Current state `→ EndAborted`, reason `TaskAbortTimeout`                  | Full refund                                                               | Execution-timeout health penalty, then release     |
| `ScoreReady` or `ErrorReported`   | `ScoreReadyTime + app_validation_timeout_seconds`                                                                                          | Current state `→ EndAborted`, reason `TaskAbortCreatorValidationTimeout` | No refund; distribute this task's fee using the successful-task fee split | No penalty, then release                           |
| `Validated` or `GroupValidated`   | `ValidatedTime + result_upload_timeout_seconds`                                                                                            | Current state `→ EndAborted`, reason `TaskAbortResultUploadTimeout`      | Full refund                                                               | Result-upload-timeout health penalty, then release |

For ordinary GPT and Stable Diffusion tasks, Relay calculates execution `Timeout` after selecting the exact GPU variant. The execution parameters are estimates only; they do not determine validation, consensus, payment, or slashing.

`TaskAbortCreatorValidationTimeout` means the creator did not complete validation by the deadline. Its fee distribution compensates the node operator and eligible delegators for node time already occupied. The task remains `EndAborted`, and the distribution does not mean the result was correct. This applies to both `ScoreReady` and `ErrorReported`.

If any validation-group member reaches `EndAborted` for `TaskAbortCreatorValidationTimeout`, Relay permanently rejects validation of the whole group. The group cannot be validated by excluding that member. Other members retain their own deadlines and current states until they independently reach terminal states.

The creator can cancel only a task whose current state is `Queued`. This transition uses the creator-cancellation reason, gives a full refund, and does not penalize a node.

## Group Validation Results

When a task is validated in a validation group, its result state is determined according to the table below:

Rows containing `EndAborted` apply only when no member has ended with `TaskAbortCreatorValidationTimeout`. That reason permanently makes the group ineligible for validation.

<table data-full-width="true"><thead><tr><th width="191">Task 1 Before</th><th width="182">Task 2 Before</th><th width="165">Task 3 Before</th><th width="165">Task 1 After</th><th width="172">Task 2 After</th><th>Task 3 After</th></tr></thead><tbody><tr><td>ScoreReady (A)</td><td>ScoreReady (A)</td><td>ScoreReady (A)</td><td>GroupValidated</td><td>EndGroupRefund</td><td>EndGroupRefund</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (A)</td><td>ScoreReady (B)</td><td>GroupValidated</td><td>EndGroupRefund</td><td>EndInvalidated</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (B)</td><td>ScoreReady (C)</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (A)</td><td>ErrorReported</td><td>GroupValidated</td><td>EndGroupRefund</td><td>EndInvalidated</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (B)</td><td>ErrorReported</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (A)</td><td>EndAborted</td><td>GroupValidated</td><td>EndGroupRefund</td><td>EndAborted</td></tr><tr><td>ScoreReady (A)</td><td>ScoreReady (B)</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ScoreReady</td><td>ErrorReported</td><td>ErrorReported</td><td>EndInvalidated</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ScoreReady</td><td>ErrorReported</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ScoreReady</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ErrorReported</td><td>ErrorReported</td><td>ErrorReported</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ErrorReported</td><td>ErrorReported</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>ErrorReported</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr><tr><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td><td>EndAborted</td></tr></tbody></table>

## Actions for Each State

<table><thead><tr><th width="248">State</th><th>Action</th></tr></thead><tbody><tr><td>Group Validated</td><td>Record the address of all the 3 nodes in the validation group.</td></tr><tr><td>End Success</td><td>Settle the payment. Release the node.</td></tr><tr><td>End Group Refund</td><td>Refund the payment. Release the node.</td></tr><tr><td>End Group Success</td><td>Distribute payment to 3 nodes. Release the node.</td></tr><tr><td>End Invalidated</td><td>Refund the payment. Slash the node.</td></tr><tr><td>End Aborted: CreatorValidationTimeout</td><td>Distribute this task's fee using the successful-task fee split as compensation for occupied node time. Do not mark the result correct. Do not penalize the node. Release the node.</td></tr><tr><td>End Aborted: all other reasons</td><td>Refund the payment. Apply a node health penalty only when the specific reason assigns the timeout to the node. Release a selected node.</td></tr></tbody></table>


# Task Dispatching

Find the best node to execute the task

When creating a task, the application can specify criteria such as the minimum VRAM requirement or restrict the node selection to a specific GPU model. The blockchain will identify all eligible nodes that meet the criteria and then randomly select one from these candidates.

If no available nodes meet the criteria, the task will be added to the queue to await more nodes. When a node satisfying the criteria is freed, the highest-price task from the queue will be assigned to this available node.

To optimize task execution speed while maintaining consensus strength, nodes are selected randomly from candidates with different probabilities. Factors influencing a node's selection probability include its local model cache and QoS score. Nodes with faster GPUs, superior networks, fewer timeouts, or locally cached models needed for the task will have a higher likelihood of selection.

## Node Selection Algorithm

The node selection algorithm first determines a pool of candidate nodes, then selects one from the pool using a weighted random process. If no candidates are available, the task is added to the queue.

### Candidate Pool

The nodes on the blockchain are first grouped by their card model, such as the Nvidia RTX 4090 group and the RTX 3080 group. These card model groups are then further grouped by their VRAM size. For example, the 16GB VRAM group may include the RTX 4080, RTX 3080, and RTX 4000 Ada groups. The blockchain will use these card groups to select candidates for a task.

If the application sets `Required GPU` in the task parameters, only the group with the required GPU model will be selected as the candidates. Otherwise, all the groups with VRAM equal to or larger than the task's requirement are chosen as candidates.

To ensure the fastest possible task execution, the system further narrows the candidate pool based on model locality. Crynux Network uses a registry to determine which nodes have already downloaded the required models, and applies the following logic:

* If **at least one** eligible node has the required models locally, the selection is **restricted** to only those nodes with local models. Nodes without local models are excluded from the candidate pool entirely.
* If **no** eligible nodes have the required models locally, the selection falls back to the full set of eligible nodes.

This ensures that tasks are preferentially routed to nodes that can begin execution immediately without downloading models, reducing startup latency for applications.

Additionally, to ensure that enough nodes have in-demand models available, the system triggers a **model pre-download mechanism** every time a task starts. When fewer than 3 available nodes have a required model, additional eligible nodes are prompted to download it proactively. For more details, refer to the following document:

{% content-ref url="/pages/5FB3KVLsniwm46kyd1um" %}
[Model Distribution](/system-design/model-distribution)
{% endcontent-ref %}

The process is illustrated in the following diagram:

```mermaid
graph TD
    A[New Task] --> B{Any eligible node with local model?};
    B -- Yes --> C[Restrict candidates to nodes with local models];
    C --> D[Select node using weighted probability];
    D --> E[Dispatch Task];
    B -- No --> F{Any eligible nodes at all?};
    F -- Yes --> G[Use all eligible nodes as candidates];
    G --> D;
    F -- No --> H[Enqueue Task];
    A --> I[Trigger Model Pre-download if needed];
```

### Selection Weight

Once the candidate pool has been determined, one node is chosen to execute the task. The selection is made through a weighted random process, where each node's probability of being chosen is proportional to a weight calculated from the factors described below. This method ensures that nodes that are better suited for the task are more likely to be selected.

*1. Model Locality Boost*

A task may require one or more models (e.g., a Stable Diffusion task might need a base model plus LoRA models; an LLM task typically needs a single model). The system boosts nodes based on how closely their local state matches the task's requirements to reduce startup latency.

There are two levels of locality:

* **On-disk locality**: The model is already downloaded to the node's disk. This saves significant time and bandwidth by avoiding downloads.
* **In-memory locality**: The model is loaded in the GPU memory. This further reduces startup time by skipping the model loading process.

The Model Locality Boost ($$M\_i$$) for a node $$i$$ is calculated as:

$$
M\_i = 1 + 0.7 \times \frac{localCnt}{total} + 0.3 \times \frac{inUseCnt}{total}
$$

Where:

* $$localCnt$$ is the number of required models available locally on disk.
* $$inUseCnt$$ is the number of required models already loaded in GPU memory.
* $$total$$ is the total number of models required by the task.

This formula gives more weight (0.7) to on-disk locality because avoiding downloads is the primary bottleneck, while in-memory locality provides an additional bonus (0.3).

*2. Staking*

To align a node's economic incentives with the long-term health and security of the network, the amount of staked tokens is a key factor in the selection probability. Nodes with a higher stake are given a higher probability of being assigned tasks.

A Staking Score ($$S\_i$$) for a node $$i$$ is calculated by normalizing the square root of its staked amount against the maximum square root of stake in the network:

$$
S\_i = \frac{ \sqrt{s\_i} } { \max( \sqrt{s\_j} \mid j \in N ) }
$$

Where $$s\_i$$ is the amount staked by node $$i$$, and $$\max( \sqrt{s\_j} \mid j \in N )$$ is the maximum square root of the staked amount among all nodes $$N$$.

This square-root staking dampens the marginal advantage of very large stakes, similar in spirit to quadratic voting. Doubling the stake increases the score by only $$\sqrt{2}$$ rather than 2, which reduces large-holder dominance and helps prevent monopolization, while still rewarding meaningful economic commitment and preserving Sybil resistance.

This design is fundamental to network security, as it significantly raises the cost of a successful Sybil attack. To successfully disrupt the network, an attacker's malicious nodes must be selected to perform tasks. Because the network prioritizes nodes with a higher stake for task assignment, an attacker cannot rely on a large number of cheap, low-stake nodes. Instead, they are forced to consolidate their capital into high-stake nodes just to be considered for selection.

This directly ties the cost of an attack to the cost of controlling the network's most trusted and active participants. It forces the attacker to lock up significant funds in the very nodes they wish to use for malicious purposes, dramatically increasing the economic risk and capital required to disrupt a meaningful portion of the network's operations. This makes the entire system more resilient by making attacks economically impractical.

*3. QoS Score*

A node's performance is determined by its underlying hardware; for example, GPUs with higher clock speeds execute tasks more quickly, and superior network connectivity leads to faster result submission. To encourage faster task execution, Crynux Network prioritizes faster nodes by giving them higher selection probabilities.

To prevent nodes from reporting fake frequencies and GPU models, Crynux Network uses the measured task execution speed rather than self-reported hardware specs. The QoS system produces a single score ($$QoS\_i$$, range 0 to 1) for each node that reflects both its long-term performance and short-term reliability. It captures whether a node is consistently fast and whether it is currently dependable. Nodes that frequently time out or perform poorly will see their QoS score drop, reducing their chance of being selected for tasks. For more details on how the QoS score is calculated, see:

{% content-ref url="/pages/1kTI3u88IvoiLtjy3IkD" %}
[Quality of Service (QoS)](/system-design/quality-of-service-qos)
{% endcontent-ref %}

*4. Final Selection Weight*

The final selection weight for a node is calculated by combining all the scores from the factors above.

To ensure a node is both secure (high stake) and performant (high QoS), the Staking and QoS scores are first combined using the harmonic mean. This method penalizes imbalance; a node cannot compensate for a very low QoS score with a high stake, or vice-versa. The result is then multiplied by the Model Locality Boost.

$$
W\_i = \frac{M\_i \cdot S\_i \cdot QoS\_i}{S\_i + QoS\_i}
$$

Where:

* $$W\_i$$ is the final selection weight for node $$i$$.
* $$M\_i$$ is the node's Model Locality Boost (1 to 2).
* $$S\_i$$ is the node's Staking Score (0 to 1).
* $$QoS\_i$$ is the node's QoS Score (0 to 1), reflecting both long-term performance and short-term reliability.

The probability of a node being selected is then its individual weight divided by the sum of the weights of all candidate nodes. Nodes are selected using weighted random sampling — higher-weighted nodes are more likely to be selected, but any eligible node can be chosen.

If there are not enough candidate nodes to be selected from, the task will be added to the task queue and wait for more nodes to become available.

## Task Queue

Tasks added to the queue are grouped based on VRAM and GPU model requirements. Initially, tasks are sorted into VRAM groups (e.g., 16GB, 24GB). Within these groups, tasks are further categorized by GPU model (e.g., 4090, A100). If no specific GPU model is required, tasks are placed in an "Any" group.

Tasks within the same group are sorted by **task priority**. When a task is taken from the queue, the task with the highest priority is prioritized. The task priority is calculated by dividing the task fee by the estimated resource consumption of the task. For more details on task priority calculation, refer to the following document:

{% content-ref url="/pages/QAZOlU2eJGyB4PPBPOXI" %}
[Task Pricing](/system-design/task-pricing)
{% endcontent-ref %}

### Dequeue a Task for a Newly Available Node

The blockchain will try to retrieve a task from the task queue when a new node becomes available. Which will happen when one of the following situations occurs:

* A running task is finished.
* A new node joins the network.
* A node resumes from the paused status.

> When a new task is sent to the blockchain, it attempts to dispatch the task immediately to the nodes, regardless of the task queue’s status. Tasks remain in the queue only if there are not enough **matching** nodes available. Even if the task queue isn't empty, there might still be available nodes in the network matching the new task, providing a chance for the new task to execute first.

Depending on the GPU model and the VRAM size of the node, the candidate task groups including:

* The task group of the same GPU model
* The "Any" groups that have a equal or smaller VRAM requirement

The first tasks of each candidate group are compared, and the task with the highest priority is selected.

### Max Size of the Task Queue

The max size of the task queue is estimated dynamically using the total number of nodes of the network:

$$
S = \alpha \* N
$$

Where $$N$$ is the number of nodes in the network, and $$\alpha$$ is a fixed multiplier that will be set as the network parameter.

If the max size is reached, when a new task is sent to the task queue, the task with the lowest priority in the queue will be removed and aborted. The task creator of the removed task will receive the `TaskAborted` event.


# Task Pricing

The capacity of the Crynux Network is limited by the total number of nodes (and the execution speed of the nodes). If there are more tasks than the network can handle, the tasks will have to wait in a queue for available nodes.

Crynux Network gives the task creator an option to pay more for a task to make it execute earlier than the others.

When the user creates a task, the total fee they are willing to pay for the task is given as an argument. The user can freely set the task fee to any value. Roughly speaking, a shorter waiting time is expected if the task fee is set higher.

However, the exact order of the tasks in the queue is not determined by the total fee directly, but by a **task priority**, which measures how much the task pays for each unit of node resource it is going to consume. A task that pays more for less resource consumption gets a higher priority.

This method allows for a more equitable distribution of the network resources across all tasks. Different tasks may differ significantly in how long they run and how powerful a node they require. By dividing the task fee by the estimated resource consumption, the system effectively identifies the tasks that provide optimal value—those that contribute a significant amount of fee without demanding an excessive portion of the network capacity. The calculation maintains a balance between efficient resource use and the satisfaction of the task creators.

## Task Priority

The task priority $$V$$ is calculated by:

$$
V = \frac{P}{T \times W}
$$

Where $$P$$ is the task fee given by the task creator, $$T$$ is the estimated task execution time, and $$W$$ is a weight representing the scarcity of the node capacity the task requires.

The priority is calculated once when the task is created, and stays fixed while the task is waiting in the queue. Tasks are dispatched in descending order of priority. If two tasks have exactly the same priority, the one created earlier goes first.

A task with a low priority will not wait forever: if it is still in the queue when its queue deadline is reached, the task is aborted and the task fee is fully refunded to the creator.

## Task Execution Time

Relay estimates only the selected node's execution time, from task start until the node submits a score or reports a task error. Waiting for application validation and uploading a validated result are separate deadline stages and are not included in this estimate.

For Stable Diffusion inference, the workload is:

$$
sd\_units = num\_images \times image\_width \times image\_height \times steps
$$

The estimated node time is the fixed execution overhead plus `sd_units` multiplied by the calibrated seconds per pixel-step.

For LLM inference, Relay deterministically encodes `messages`, `tools`, and `template_args` and measures the UTF-8 byte length as `input_bytes`. The estimated node time is:

$$
T = constant\_seconds + input\_bytes \times seconds\_per\_input\_byte + max\_new\_tokens \times seconds\_per\_output\_token
$$

The generation configuration uses its declared `max_new_tokens`, or the configured default when it is absent. Queue ordering does not reduce this value based on historical early stopping.

Stable Diffusion fine-tuning keeps its creator-supplied timeout and existing pricing rule.

### Automatic Calibration

Relay calibrates execution parameters from completed, validated tasks for each exact `(GPUName, GPUVram)` variant. A task that explicitly requires a GPU variant uses that variant's parameters directly.

A task without `RequiredGPU` uses an in-memory aggregate for its task type and VRAM demand. The aggregate includes calibrated variants whose VRAM is at least the demand and gives each compatible GPU variant equal weight. A successful calibration sample updates its exact GPU variant and immediately recalculates every initialized aggregate that includes that variant.

The parameter key intentionally excludes model ID, model architecture, dtype, quantization, scheduler, and other model configuration. Different models on the same exact GPU variant can therefore execute faster or slower than the estimate. This error affects queue priority and the Relay-owned execution timeout only. It does not affect validation, consensus, fee settlement, or slashing.

The priority and its workload values are fixed when the task is created. Later calibration changes do not reorder an existing queued task.

Queue priority changes only dispatch order. It does not extend or shorten the queue deadline.

## Node Capacity Weight

Besides the execution time, tasks also differ in the kind of node they require. A task demanding a large amount of VRAM can only run on the high-end nodes, which are scarcer in the network, while a lightweight task can run on almost any node.

If the queue ordering considered time alone, a task occupying a scarce high-end node would be treated the same as a task occupying an abundant low-end node for the same duration, even though the former consumes a much more valuable resource.

To account for this, the priority calculation applies a weight based on the VRAM requirement of the task: the more VRAM a task requires beyond the baseline, the proportionally larger its weight, and the more fee it needs to pay to reach the same priority. Tasks whose VRAM requirement is at or below the baseline all share the same weight of 1.

The weight only affects the ordering of the waiting queue. It changes neither the task fee charged to the creator, nor which nodes are eligible to execute the task.


# Quality of Service (QoS)

To encourage nodes to provide better service to the network — faster execution, reliable availability, and fewer failures — the QoS (Quality of Service) system evaluates each node's performance and directly influences its role in the network.

The QoS score of a node is continuously updated as it executes tasks. It combines two factors operating at different time scales — a long-term performance factor and a short-term reliability factor — so that the score reflects both a node's sustained hardware quality and its current operational status. The score is then used to influence several key aspects of the network's operation, including task allocation priority, reward distribution, and node removal decisions.

By giving more advantages to higher-scoring nodes, the network encourages nodes to improve their hardware and network environment, thereby improving the overall service quality for applications.

## QoS Score Usage

### Task Allocation Priority

A node's QoS score directly affects its probability of being selected for new tasks. Nodes with higher QoS scores are more likely to be chosen, ensuring that high-performing nodes handle more work and earn more rewards. For the detailed selection mechanism, see:

{% content-ref url="/pages/Y7oaQ7V3uKJTimOkHrhD" %}
[Task Dispatching](/system-design/task-dispatching)
{% endcontent-ref %}

### Bad Node Removal

If a node consistently underperforms over a sustained period, the node is permanently removed from the network. This prevents persistently underperforming nodes — such as those that have been shut down without properly leaving the network — from degrading application experience. See [Permanent Kickout](#permanent-kickout) for details.

## Node QoS Score

The QoS score evaluates node quality through two factors that operate at different time scales:

* **Long-term performance factor**: A rolling average of recent validation task scores that captures whether a node is consistently fast.
* **Short-term reliability factor**: A multiplier that reacts immediately to timeout failures, capturing whether a node is currently dependable.

A single long-term average alone would react too slowly to sudden failures — a node could time out on many consecutive tasks before its score drops meaningfully. Conversely, relying only on short-term signals would make the score too volatile and fail to reflect a node's true hardware quality. By combining both, the QoS score can immediately suppress unreliable nodes (protecting applications) while still accurately ranking nodes by their sustained performance (rewarding better hardware).

The final QoS score for a node $$i$$ is the product of both factors:

$$
QoS\_i = \frac{Q\_{long}}{Q\_{max}} \times H
$$

Where $$Q\_{long}$$ is the node's long-term performance score (rolling average of task scores), $$Q\_{max} = 10$$ is the maximum possible task score, and $$H$$ is the short-term reliability factor (range 0 to 1). New nodes that have not yet completed any validation tasks are assigned a default long-term score equivalent to $$Q\_{long} / Q\_{max} = 0.5$$.

### Long-term Performance Factor

The long-term factor measures a node's sustained execution speed across its recent validation tasks. It changes gradually and reflects the node's typical hardware and network quality.

#### Task Score

To measure performance objectively, the network uses **validation task groups** where the same task is assigned to multiple nodes (typically 3) simultaneously. The network measures the time each node takes to submit its result. The nodes in the group are ranked by execution speed, and each receives a fixed task score based on its ranking:

<figure><img src="/files/Ux7QuXfcAlf1VWfAfxyT" alt=""><figcaption><p>The task score of a node by its submission order and status</p></figcaption></figure>

Faster submissions earn a higher task score, directly rewarding nodes for improving all factors that affect submission speed — GPU performance, network quality, memory bandwidth, and system optimization.

{% hint style="info" %}
When group validation completes, an aborted member receives a task score of 0. If group validation determines that **all 3 tasks** must be aborted because the task cannot produce an accepted result, the scores are set to NULL and excluded from the rolling average.

`TaskAbortCreatorValidationTimeout` does not create a validation rank and does not update the rolling average. It means the creator failed to complete validation while the task was in `ScoreReady` or `ErrorReported`. If any group member has this reason, Relay permanently rejects validation of that group, and no partial group ranking is produced.
{% endhint %}

#### Rolling Average

Each node maintains a long-term performance score that represents its recent performance. This is a **rolling average** of the task scores from its most recent validation tasks.

The system keeps a rolling pool of the last **50 task scores** for each node. When a new task score arrives, it is appended to the pool. If the pool exceeds 50 entries, the oldest entry is removed. The long-term score is the **arithmetic mean** of all scores in the pool:

$$
Q\_{long} = \frac{\sum\_{j=1}^{n} {ts}\_j}{n}
$$

Where $$n$$ is the number of task scores in the pool (up to 50), and $${ts}\_j$$ is the task score for the $$j$$-th most recent validation task.

{% hint style="info" %}
The rolling pool approach means the long-term score reflects recent performance rather than lifetime history. A node that improves its hardware or network setup will see its score improve as new, higher scores push out older, lower ones.
{% endhint %}

### Short-term Reliability Factor

The long-term factor operates on a slow time scale — it takes many validation tasks to shift the 50-task rolling average. This means the long-term factor alone cannot protect applications from a node that suddenly starts failing. A node could time out on dozens of consecutive tasks before its long-term score degrades enough to trigger any consequence.

The short-term reliability factor addresses this gap. It is designed to balance two competing goals:

* **Application quality**: When a node times out, it should be immediately excluded from receiving further tasks so that applications are not affected by unreliable nodes.
* **Node protection**: An otherwise healthy node should not be permanently removed due to a short burst of failures (e.g., caused by invalid application tasks or transient network issues). It should be given a chance to recover and prove itself.

The mechanism achieves both by sharply reducing a failing node's QoS score on each timeout (protecting applications), while allowing the score to recover automatically over time and through successful task completions (protecting nodes). Each node carries a short-term reliability factor $$H$$ (range 0.0 to 1.0, default 1.0). This factor directly scales the node's QoS score:

* On each **node-owned timeout failure**, $$H$$ is immediately multiplied by a penalty factor (0.3), causing a sharp drop in the QoS score. Execution timeout and result-upload timeout are node-owned. A queue timeout has no selected node, and a creator-validation timeout is owned by the creator, so neither penalizes a node.
* When $$H$$ drops below a **hard exclusion threshold** (0.1), the node is completely excluded from task selection. It receives zero tasks, which from the application's perspective is equivalent to the node being offline.
* The penalty is **temporary**. $$H$$ recovers through two complementary mechanisms: passive time-based recovery (exponential decay back toward 1.0) and active success-based recovery (a discrete boost for each successfully completed task).
* When a node **joins or re-joins** the network, $$H$$ is reset to 1.0.

#### Penalty on Node-Owned Timeout

The short-term reliability factor is reduced when a selected node misses either of its Relay-owned deadlines:

* `Started` or `ParametersUploaded`: the node does not submit a score or report an error by the execution deadline.
* `Validated` or `GroupValidated`: the node does not upload the result by the result-upload deadline.

In either case:

$$
H\_{new} = H\_{current} \times 0.3
$$

The penalty compounds rapidly with consecutive timeouts:

| Consecutive Timeouts | Short-term Factor (H) | Effect                                 |
| -------------------- | --------------------- | -------------------------------------- |
| 0                    | 1.0                   | Normal QoS score                       |
| 1                    | 0.30                  | 70% reduction in QoS score             |
| 2                    | 0.09                  | Effectively excluded (below threshold) |
| 3                    | 0.027                 | Deep exclusion                         |

No health penalty applies when a queued task expires before node selection. No health penalty applies when a task in `ScoreReady` or `ErrorReported` expires with `TaskAbortCreatorValidationTimeout`.

For `TaskAbortCreatorValidationTimeout`, the task remains aborted, but its fee is distributed using the successful-task fee split to compensate the node operator and eligible delegators for execution and waiting time already occupied. This payment does not mean the result was correct. It does not add a validation rank, update `Q_long`, or apply a successful-result-upload boost.

#### Hard Exclusion

When a node's short-term reliability factor drops below **0.1**, it is completely excluded from task selection — it receives zero tasks. The node automatically becomes eligible again as the factor recovers above the threshold through the recovery mechanisms described below.

#### Recovery

The penalty is temporary. A node's short-term reliability factor recovers through two complementary mechanisms:

**1. Passive time-based recovery.** Even if no tasks are assigned to the node, the factor slowly drifts back toward 1.0 over time. This follows an exponential curve with a 30-minute time constant:

$$
H(t) = H\_{base} + (1 - H\_{base}) \cdot (1 - e^{-(t - t\_{base}) / \tau})
$$

Where $$\tau = 30$$ minutes. This means approximately 63% recovery after 30 minutes, 86% after 60 minutes, and 95% after 90 minutes. Passive recovery is critical because it is the **only** mechanism that works in the exclusion zone (where the node receives no tasks and therefore cannot earn success boosts).

**2. Active success-based recovery.** Every time the node completes a task successfully and uploads its validated result, the factor receives a discrete boost:

$$
H\_{new} = \min(1.0, \ H\_{current} + 0.15)
$$

This is faster than passive recovery and serves as a proof-of-work mechanism — a node that actively demonstrates it can complete tasks recovers faster than one that simply waits.

{% hint style="info" %}
The two recovery mechanisms are complementary across different ranges of H. In the **exclusion zone** (H < 0.1), the node receives no tasks, so only passive time-based recovery works — it slowly brings H back above the threshold. In the **low probability zone** (H = 0.1 \~ 0.3), the node starts receiving occasional tasks, and each success provides a meaningful relative boost. In the **moderate zone** (H > 0.3), success-based recovery becomes the dominant force, creating a positive feedback loop: each success increases H, which increases the QoS score, which increases selection probability, which leads to more tasks and more successes.
{% endhint %}

## Permanent Kickout

The permanent kickout mechanism removes nodes whose long-term performance demonstrates sustained poor quality. It uses the 50-task rolling average long-term score and evaluates two conditions, both of which must be true:

1. The node's long-term score has dropped below the kickout threshold (default: **2.0**).
2. The node has completed enough validation tasks to fill the rolling pool (default: **50 tasks**).

The second condition prevents premature removal of nodes that have only completed a few validation tasks — the system waits until there is a statistically meaningful sample before making a permanent decision.

When a node is kicked out:

* Its status is set to quit and it is removed from the active node pool.
* Its stake is returned (the node is **not slashed** — permanent kickout is distinct from cheating penalties).
* A kickout event is emitted on the blockchain.

Due to validation task sampling, a node must execute a large number of total tasks before the 50-task pool is full. Permanent kickout is a long-term backstop that catches nodes with genuinely persistent problems, while the short-term reliability factor handles immediate issues.


# Model Distribution

Distribute models across the nodes

## Model Hosting Service

Crynux will offer model hosting within the Lithium Network. Developers can upload their models to the network, enabling Model-as-a-Service for applications and other developers.

Upon upload, the model is initially stored on a few nodes. Tasks requiring the model are randomly distributed among these nodes. As demand grows, additional nodes are selected to store the model, enhancing service capabilities. Conversely, if demand decreases, the model is removed from some nodes to save disk space.

<figure><img src="/files/92cIxsOpDYA7tlB1S0BP" alt=""><figcaption></figcaption></figure>

## Model Download Cache

In the pre-Lithium network setup, the model required by a task is specified as a Huggingface or Civitai link in the task parameters. Upon task arrival on the node, if the model isn't already stored locally, the node needs to download it. This downloading process often takes a considerable amount of time, significantly reducing task execution speed and potentially causing task timeout.

Before model hosting is implemented, the model distribution mechanism has already been applied in the Helium Network to solve this issue. When a task is initiated on the blockchain, it assesses overall demand and may notify certain nodes to download the model, in addition to selecting nodes to execute the tasks.

## Impact on the Network Consensus

The node to execute a task will only be selected from nodes with locally stored models, significantly limiting the number of candidates. This increases the risk of Sybil Attacks, especially for less popular models. To mitigate this risk, a relatively large number of nodes should be selected for a new model initially.

In the Helium Network, when a task is initiated, 10 nodes are selected to implement a new model. If fewer than 3 nodes remain available for the model, the blockchain will notify 10 additional nodes to download the model.

## Built-in Model Storage V.S. External Decentralized Storage

Instead of storing the models on individual nodes, another option is to use a decentralized storage service to hold the models. Nodes can then retrieve the models as needed.

However, downloading the model takes significantly more time than executing tasks, as seen with the Helium Network. This highlights the "data locality" concept in computer science, which suggests moving computation (code) to the data because data is usually larger than the code.

Integrating model storage into nodes fosters a robust environment where tasks can be executed swiftly and reliably, aligning with the decentralized ethos of the Crynux Network:

1. **Increased Speed**: With models stored directly on nodes, the time taken to retrieve and execute the model is significantly reduced. This direct access minimizes latency and boosts overall network efficiency.
2. **Enhanced Data Locality**: By housing models on the nodes where computation occurs, the Crynux Network leverages data locality principles, reducing the need to move large model files across the network.
3. **Improved Reliability**: Storing models across multiple nodes increases redundancy. In the event of node failures, models remain accessible, ensuring continuous operation without interruptions.
4. **Cost Efficiency**: Eliminating the need for external storage services reduces operational costs. This built-in approach streamlines resource allocation and optimizes expenditures.

## Proof of Storage

In decentralized storage networks, ensuring nodes adhere to protocol rules is crucial for maintaining integrity and fairness. However, a situation may arise where nodes do not comply with the rules for retrieving model files and falsely claim compliance to obtain rewards.

For example, Filecoin addresses this issue by employing a consensus protocol known as "[Proof of Spacetime](https://docs.filecoin.io/storage-providers/filecoin-economics/storage-proving)," which utilizes zero-knowledge proofs.

In the Crynux Network, verifying model file storage does not require additional proof. The integrity of the model file is confirmed when a node successfully executes a task and produces the same result as other nodes. If a node delivers accurate computation results, it is assumed to have the correct model files.

If the node downloads the model file only when a task arrives, execution speed will be slower. This can lead to a [QoS penalty](/system-design/quality-of-service-qos), reducing the likelihood of receiving rewards and future tasks, and may eventually result in the node being removed from the network.


# Start a Node

1. ~~Fill a form to tell us your GPU type, location, network bandwidth~~ \[<mark style="color:blue;">**No application form, no sign up, you don’t need to tell us**</mark>]
2. ~~Join waitlist and wait for the email from us~~ \[<mark style="color:blue;">**No waitlist, just install the Crynux Node app, you can start earning CNX tokens right away**</mark>]
3. Just download the package according to your platform, and follow the tutorials below:

<table><thead><tr><th width="131">Blockchain</th><th width="131">Platform</th><th width="261">Requirements</th><th data-type="content-ref">Download Link</th></tr></thead><tbody><tr><td>Base</td><td>Windows</td><td>Nvidia GPU with 8GB VRAM</td><td><a href="https://drive.google.com/uc?id=11xaq4TYq9N4cYP70o8OEjH4MaP1I3GhN&#x26;export=download">https://drive.google.com/uc?id=11xaq4TYq9N4cYP70o8OEjH4MaP1I3GhN&#x26;export=download</a></td></tr><tr><td>Base</td><td>Mac</td><td>M1/M2/M3 and later</td><td><a href="https://github.com/crynux-network/crynux-node/releases/download/v3.5.1/crynux-node-lithium-v3.5.1-base-mac-arm64-unsigned.dmg">https://github.com/crynux-network/crynux-node/releases/download/v3.5.1/crynux-node-lithium-v3.5.1-base-mac-arm64-unsigned.dmg</a></td></tr><tr><td>Near</td><td>Windows</td><td>Nvidia GPU with 8GB VRAM</td><td></td></tr><tr><td>Near</td><td>Mac</td><td>M1/M2/M3 and later</td><td></td></tr></tbody></table>

## Tutorials

### Windows

{% content-ref url="/pages/QquuDgRAcmcQEouD8mKn" %}
[Start a Node - Windows](/node-hosting/start-a-node/start-a-node-windows)
{% endcontent-ref %}

### Mac with Apple Silicon Chips (M1/M2/M3 and later)

{% content-ref url="/pages/P9OcReROZ6L0k0AyWb2W" %}
[Start a Node - Mac](/node-hosting/start-a-node/start-a-node-mac)
{% endcontent-ref %}

### Cloud services based on Docker

*Vast.ai*

{% content-ref url="/pages/dXGIWk0bz3awx1FdrDSX" %}
[Start a Node - Vast](/node-hosting/start-a-node/start-a-node-vast)
{% endcontent-ref %}

*Octa.space*

{% content-ref url="/pages/j120dUkR4XnvHUN5UJjw" %}
[Start a Node - Octa](/node-hosting/start-a-node/start-a-node-octa)
{% endcontent-ref %}

### Docker

{% content-ref url="/pages/gGypoNA8XJ1TX4aGfQmE" %}
[Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker)
{% endcontent-ref %}


# Start a Node - Windows

Start a node to join the Crynux Network on Windows

## 1. Prerequisite

Before you start, make sure your device meets the following requirements:

<table><thead><tr><th width="187">Hardware</th><th>Requirements</th></tr></thead><tbody><tr><td>GPU</td><td>NVIDIA GPU with 8GB VRAM</td></tr><tr><td>Memory</td><td>16GB</td></tr><tr><td>Disk Space</td><td>60GB</td></tr><tr><td>Network</td><td>Public network access to Huggingface and Civitai</td></tr></tbody></table>

## 2. Install the software

### Install the latest NVIDIA driver

Make sure you have already installed the latest NVIDIA driver from the [NVIDIA official website](https://www.nvidia.com/Download/index.aspx?lang=en-us).

### Download the Crynux Node

Download the binary release version of the Crynux Node from the link below:

For Base users:

{% embed url="<https://drive.google.com/file/d/11xaq4TYq9N4cYP70o8OEjH4MaP1I3GhN/view?usp=drivesdk>" %}

For Near users:

{% hint style="info" %}
Coming soon...
{% endhint %}

{% hint style="info" %}
Starting a node on Windows using the binary release package, as described here, is still in **beta testing**. If you have trouble running the downloaded package, please use [the Docker version](/node-hosting/start-a-node/start-a-node-docker) instead.
{% endhint %}

## 3. Start the node

Unzip the downloaded package, double-click on the `Crynux Node.exe` to start the node:

<figure><img src="/files/VWzDuSbH0lA4Bgx48IdR" alt=""><figcaption></figcaption></figure>

## 4. Prepare the wallet

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/jsbRJlgIKKHGkbrubNMx" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 5. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/MD6EFoA1jyfzmk01sCuY" alt=""><figcaption></figcaption></figure>

## 6. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/PK0rVueHR90Nl0Y4l7h4" alt=""><figcaption></figcaption></figure>

Now you could just leave it there to execute the tasks. When you shutdown the Crynux Node app, it will try to quit the network before exiting, so that new tasks will not be sent to the node any more. And the next time the app is started, it will join the network to receive new tasks automatically.


# Start a Node - Mac

Start a node to join the Crynux Network on Mac

## 1. Prerequisite

The Crynux Node supports only the Macs with the M1, M2, M3 or newer versions. Make sure your device meets the requirement before running the node.

<table><thead><tr><th width="187">Hardware</th><th>Requirements</th></tr></thead><tbody><tr><td>Model</td><td>Mac M1, M2, M3 or newer</td></tr><tr><td>Memory</td><td>16GB</td></tr><tr><td>Disk Space</td><td>60GB</td></tr><tr><td>Network</td><td>Public network access to Huggingface and Civitai</td></tr></tbody></table>

## 2. Download the Crynux Node software

Download the DMG file using the following link. By default, it will be saved to your `Downloads` folder.

{% hint style="warning" %}
**Allow the Installer to Run**

Due to macOS security policies, you must remove the `quarantine` attribute from the downloaded DMG file before opening it. This prevents security warnings during installation.

Open the `Terminal` app and run the following command. Make sure to replace `crynux-node.dmg` with the actual name of the downloaded file.

`$ xattr -d com.apple.quarantine ~/Downloads/crynux-node.dmg`
{% endhint %}

For Base users:

{% embed url="<https://github.com/crynux-network/crynux-node/releases/download/v3.5.1/crynux-node-lithium-v3.5.1-base-mac-arm64-unsigned.dmg>" %}

For Near users:

{% hint style="info" %}
Coming soon...
{% endhint %}

## 3. Start the node

Double-click on the icon of the newly installed app to start the node:

<figure><img src="/files/zkU8JYQzaKsdMK4drkL3" alt=""><figcaption></figcaption></figure>

## 4. Prepare the wallet

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/lbPlnhmdTL5XjnRf3drT" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 5. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/6XZn6Fob33DLOQsxpjSH" alt=""><figcaption></figcaption></figure>

## 6. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/BKViJGeX4VL0SchWT35O" alt=""><figcaption></figcaption></figure>

Now you could just leave it there to execute the tasks. When you shutdown the Crynux Node app, it will try to quit the network before exiting, so that new tasks will not be sent to the node any more. And the next time the app is started, it will join the network to receive new tasks automatically.


# Start a Node - Linux

Start a node using binary package on Linux (Ubuntu)

The binary releases of the Crynux Node for the Linux distributions are still a work in progress. For now you could simply install Docker and use the Docker image to start Crynux Node on Linux:

{% content-ref url="/pages/gGypoNA8XJ1TX4aGfQmE" %}
[Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker)
{% endcontent-ref %}


# Start a Node - Docker

Start a node to join the Crynux Network using Docker images

{% hint style="info" %}
This guide is used to start the Docker container on Windows and Linux (Ubuntu, etc), or on a cloud based VM such as AWS EC2. **DO NOT** use this guide on Docker based clouds such as Vast, [follow the instructions in this doc instead](/node-hosting/start-a-node/start-a-node-vast).
{% endhint %}

## 0. Overview

* ~~Fill a form to tell us your GPU type, location, network bandwidth~~ \[<mark style="color:blue;">**No application form, no sign up, you don’t need to tell us**</mark>]
* ~~Join waitlist and wait for the email from us~~ \[<mark style="color:blue;">**No waitlist, just install the Docker image, you can start earning CNX tokens right away**</mark>]
* Follow the steps below:

## 1. Prerequisite

Before you start, make sure your device meets the following requirements:

<table><thead><tr><th width="187">Hardware</th><th>Requirements</th></tr></thead><tbody><tr><td>GPU</td><td>NVIDIA GPU with 8GB VRAM</td></tr><tr><td>Memory</td><td>16GB</td></tr><tr><td>Disk Space</td><td>60GB</td></tr><tr><td>Network</td><td>Public network access to Huggingface and Civitai</td></tr></tbody></table>

## 2. Install the software

### Install the latest NVIDIA driver

Download the latest NVIDIA driver from the [NVIDIA official website](https://www.nvidia.com/Download/index.aspx?lang=en-us), and finish the installation.

### Install the latest version of Docker

Download the latest version of the [Docker Desktop](https://docs.docker.com/get-docker/), and finish the installation.

<details>

<summary>If you have 16GB of memory and use Docker with WSL2 on Windows</summary>

The memory limit for WSL is default to 8GB, which is not enough to run the Node. You will have to change the default settings using a [`.wslconfig`](https://learn.microsoft.com/en-us/answers/questions/1296124/how-to-increase-memory-and-cpu-limits-for-wsl2-win) file to allow WSL to use 16GB memory.

</details>

<details>

<summary>If you are running on Linux (Ubuntu/Fedora/CentOS/...)</summary>

Install the latest version of NVIDIA Container Toolkit:

<https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html>

</details>

### Check the installation

**a. Run the following command in the terminal to check the version of the docker engine:**

```bash
$ docker --version
```

Make sure the returned version number is greater than 19.03.0:

```
Docker version 26.0.0, build 2ae903e
```

**b. Run the following command in the terminal:**

<pre class="language-bash"><code class="lang-bash"><strong>$ sudo docker run --rm --gpus all ubuntu nvidia-smi
</strong></code></pre>

You should get the info of the GPU from `nvidia-smi` like this:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10    Driver Version: 535.86.10    CUDA Version: 12.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

{% hint style="danger" %}
If something goes wrong on the above steps, the problem is on the Docker or your operating system, please search the error message online for solutions.
{% endhint %}

## 3. Start the node using the Docker Compose

#### a. Get the Crynux Docker Compose project

{% tabs %}
{% tab title="Base users" %}
you can use Git to clone the branch for Base of the following repository:

```bash
$ git clone -b base https://github.com/crynux-network/crynux-node-docker-compose.git
```

or simply download the files from GitHub:

<https://github.com/crynux-network/crynux-node-docker-compose/tree/base>
{% endtab %}

{% tab title="Near users" %}
{% hint style="info" %}
Coming soon. The Near network is still being deployed and will be available shortly.
{% endhint %}
{% endtab %}
{% endtabs %}

#### b. Start the container

In a terminal, navigate to the folder you just cloned or downloaded, and run the following command

```shell
$ cd crynux-node-docker-compose
$ docker compose up -d
```

#### c. Visit the WebUI in the browser

Open the browser and go to <http://localhost:7412>

You should see the WebUI of the Node:

<figure><img src="/files/TOVMJj2p62IyKvCBU6yc" alt=""><figcaption></figcaption></figure>

## 4. Prepare the wallet

{% hint style="danger" %}
**DO NOT** **use the Web UI to create or import private keys if you're accessing the Web UI from a remote machine.**

**You will loose your tokens!**

If you're using HTTP protocol to access the WebUI, the connection is not encrypted, and the private key might be intercepted by a malicious middle man.

Instead, use an SSH connection in the terminal to transfer your private key to the node.
{% endhint %}

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/sW96dKDW7zUIh6kXse1g" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 5. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/9ifO8ZhSDPqFBgpJeg0v" alt=""><figcaption></figcaption></figure>

## 6. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/jQlfE6EZrdMjZP3SUHv6" alt=""><figcaption></figcaption></figure>

Now the Node is fully up and running. You could just leave it there to run tasks automatically.

The Node could be paused or stopped at any time by clicking the control buttons. If the node is in the middle of running a task, after clicking the buttons, the node will go into the "pending" status and continue with the running task. When the task is finished, the node will pause/stop automatically.

The difference between pausing and stopping is that pausing will not cause the staked CNX tokens to be returned, so that the transaction costs less gas fee than stopping. If you have a plan of going back, you could use pausing rather than stopping.


# Start a Node - LXC

Start a node to join the Crynux Network using LXC images

{% hint style="info" %}
This guide is for starting a Crynux Node using LXC (Linux Containers) on a Linux machine with an NVIDIA GPU.
{% endhint %}

## 1. Prerequisite

Before you start, make sure your device meets the following requirements:

<table><thead><tr><th width="187">Hardware</th><th>Requirements</th></tr></thead><tbody><tr><td>GPU</td><td>NVIDIA GPU with 8GB VRAM</td></tr><tr><td>Memory</td><td>16GB</td></tr><tr><td>Disk Space</td><td>60GB</td></tr><tr><td>Network</td><td>Public network access to Huggingface and Civitai</td></tr></tbody></table>

## 2. Install the software

### Install the latest NVIDIA driver

Download the latest NVIDIA driver from the [NVIDIA official website](https://www.nvidia.com/Download/index.aspx?lang=en-us), and finish the installation.

The Crynux Node requires the `nvidia-smi` command to be installed on the host machine. You need to make sure this command is available on your host.

On Ubuntu, the command can be installed via the `nvidia-utils` package. For other Linux distributions, please find the package that provides the `nvidia-smi` command and install it.

You can verify the installation by running `nvidia-smi` on your host machine:

```bash
$ nvidia-smi
```

### Install NVIDIA Container Toolkit

Install the NVIDIA Container Toolkit by following the official guide: <https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html>

### Install LXD or Incus

Install your chosen container manager by following its official guide:

* **LXD:** <https://documentation.ubuntu.com/lxd/>
* **Incus:** <https://linuxcontainers.org/incus/docs/main/installing/>

After installation, initialize it according to its documentation.

## 3. Setup the Configuration Profile

The Crynux Node repository provides a script to generate a ready-to-use profile configuration file tailored to your system.

#### a. Get the profile script from GitHub

Clone the `crynux-node` repository and navigate to the script directory:

```bash
$ git clone https://github.com/crynux-network/crynux-node.git
$ cd crynux-node/build/lxc/crynux-profile
```

{% hint style="info" %}
Cloning the entire `crynux-node` repository can be time-consuming. As an alternative, you can download only the files from the `crynux-profile` directory.

Visit [this Github link](https://github.com/crynux-network/crynux-node/tree/main/build/lxc/crynux-profile) to download the files. Make sure you are in the `crynux-profile` directory in your terminal to proceed with the next steps.
{% endhint %}

#### b. Generate the profile configuration

Run the `create-profile.sh` script to generate the `profile.yaml` file. You must tell the script whether you are using `lxc` or `incus`.

```bash
# Using LXD
$ ./create-profile.sh lxc

# Using Incus
$ ./create-profile.sh incus
```

#### c. Create the profile

Now, create the `crynux-node` profile using the generated `profile.yaml` file:

```bash
# Using LXD
$ sudo lxc profile create crynux-node
$ cat profile.yaml | sudo lxc profile edit crynux-node

# Using Incus
$ sudo incus profile create crynux-node
$ cat profile.yaml | sudo incus profile edit crynux-node
```

## 4. Start the Node

### a. Add the Crynux LXC image remote

The Crynux Node LXC images are hosted on a public image server. Add it to your remotes:

```bash
# Using LXD
$ sudo lxc remote add --protocol simplestreams crynux https://lxc.crynux.io

# Using Incus
$ sudo incus remote add --protocol simplestreams crynux https://lxc.crynux.io
```

You can list the available images:

```bash
# Using LXD
$ sudo lxc image list crynux:

# Using Incus
$ sudo incus image list crynux:
```

### b. Launch the container

Now, launch the container using the profile you created. This is a clean, single command that applies all your configurations at once. Note that we apply both the `default` profile (for basic networking) and our new `crynux-node` profile.

Launch the Crynux Node container. There are different images for different blockchain networks.

{% tabs %}
{% tab title="Base users" %}
Use the `crynux-node:latest-base` image:

```bash
# Using LXD
$ sudo lxc launch crynux:crynux-node:latest-base crynux-node -p default -p crynux-node

# Using Incus
$ sudo incus launch crynux:crynux-node:latest-base crynux-node -p default -p crynux-node
```

{% endtab %}

{% tab title="Near users" %}
{% hint style="info" %}
Coming soon. The Near network is still being deployed and will be available shortly.
{% endhint %}
{% endtab %}
{% endtabs %}

### c. Visit the WebUI in the browser

Open the browser and go to <http://localhost:7412>

You should see the WebUI of the Node:

<figure><img src="/files/TOVMJj2p62IyKvCBU6yc" alt=""><figcaption></figcaption></figure>

## 5. Prepare the wallet

{% hint style="danger" %}
**DO NOT** **use the Web UI to create or import private keys if you're accessing the Web UI from a remote machine.**

**You will loose your tokens!**

If you're using HTTP protocol to access the WebUI, the connection is not encrypted, and the private key might be intercepted by a malicious middle man.

Instead, use an SSH connection in the terminal to transfer your private key to the node.
{% endhint %}

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/sW96dKDW7zUIh6kXse1g" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 6. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/9ifO8ZhSDPqFBgpJeg0v" alt=""><figcaption></figcaption></figure>

## 7. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/jQlfE6EZrdMjZP3SUHv6" alt=""><figcaption></figcaption></figure>

Now the Node is fully up and running. You could just leave it there to run tasks automatically.

The Node could be paused or stopped at any time by clicking the control buttons. If the node is in the middle of running a task, after clicking the buttons, the node will go into the "pending" status and continue with the running task. When the task is finished, the node will pause/stop automatically.

The difference between pausing and stopping is that pausing will not cause the staked CNX tokens to be returned, so that the transaction costs less gas fee than stopping. If you have a plan of going back, you could use pausing rather than stopping.

## 8. Updating the Node

### a. Pull the latest image

First, refresh your local image to pull the latest version from the remote server.

{% tabs %}
{% tab title="Base users" %}

```bash
# Using LXD
$ sudo lxc image refresh crynux:crynux-node:latest-base --alias

# Using Incus
$ sudo incus image refresh crynux:crynux-node:latest-base --alias
```

{% endtab %}

{% tab title="Near users" %}
{% hint style="info" %}
Coming soon. The Near network is still being deployed and will be available shortly.
{% endhint %}
{% endtab %}
{% endtabs %}

### b. Stop and delete the old container

```bash
# Using LXD
$ sudo lxc stop crynux-node
$ sudo lxc delete crynux-node

# Using Incus
$ sudo incus stop crynux-node
$ sudo incus delete crynux-node
```

Don't worry, if you have mounted the data and config directories, your data will be safe on the host machine as it is managed by the profile.

### c. Launch a new container with the latest image

Follow the instructions in step 4 to launch a new container. It will now use the latest image you just pulled, and automatically apply the `crynux-node` profile with all your settings.

{% tabs %}
{% tab title="Base users" %}

```bash
# Using LXD
$ sudo lxc launch crynux:crynux-node:latest-base crynux-node -p default -p crynux-node

# Using Incus
$ sudo incus launch crynux:crynux-node:latest-base crynux-node -p default -p crynux-node
```

{% endtab %}

{% tab title="Near users" %}
{% hint style="info" %}
Coming soon. The Near network is still being deployed and will be available shortly.
{% endhint %}
{% endtab %}
{% endtabs %}

Your node will restart with the new version, using your existing data and configuration.


# Start a Node - Vast

Start a node on Vast.ai

The Crynux Node can be easily started on cloud services, such as [Vast.ai](https://vast.ai/), who supports starting a VM using Docker images directly. The steps to start a node on those services are quite similar. We will use Vast.ai as an example to show the complete steps to start a Crynux Node.

## 1. Start the container using template

We have already created the template for the Crynux Node on Vast, just use this template to start the node:

{% embed url="<https://cloud.vast.ai/?ref_id=136043&template_id=bba3743eb66fac590ab6a9de83158f4b>" %}

The content of the template is shown below:

<figure><img src="/files/amqaRI636G2JXXHnai7K" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
**Please use the latest version tag to start the container**

you could find the available tags at:

[**https://github.com/crynux-network/crynux-node/pkgs/container/crynux-node/versions**](https://github.com/crynux-network/crynux-node/pkgs/container/crynux-node/versions)

For example, if you want to run the 3.5.1 version of the Crynux Node under Base Network, use the image link below:

`ghcr.io/crynux-network/crynux-node:3.5.1-base`
{% endhint %}

Some other config options that worth highlighting:

* Expose port `7412` for WebUI.
* Use the default docker ENTRYPOINT to start the container. Do not use interactive shells.

After selecting your desired hardware, and starting the instance, find the instance in the `INSTANCES` tab:

<figure><img src="/files/kV6r9sKiEa3cYFxQDwGD" alt=""><figcaption></figcaption></figure>

Wait until the container finishes initialization, and shows the `RUNNING` status.

## 2. Find the URL to access the WebUI

Click on the network info button to show the detailed ip address and ports:

<figure><img src="/files/pKiQuCXg2GktvVMxtWMT" alt=""><figcaption></figcaption></figure>

The URL to access the WebUI will be shown in the popup:

<figure><img src="/files/81owdY9Gj8cg26pWmIVO" alt=""><figcaption></figcaption></figure>

In this case, the URL of the WebUI is `http://213.181.122.2:40021`. Just open it in a browser window, you should see the WebUI of the Node:

<figure><img src="/files/TOVMJj2p62IyKvCBU6yc" alt=""><figcaption></figcaption></figure>

## 3. Prepare the wallet

{% hint style="danger" %}

#### Security Warning: Private key on third‑party machines

When you run a node on Vast, your Docker container is not on your own hardware. It runs on GPU machines owned and operated by other individual Vast users. Any private key you put into the container is therefore stored on those third-party machines, where a malicious host or malware on the host system could read the key and immediately transfer all funds controlled by it.

To reduce potential loss, you **MUST** set up a **Beneficial Address** so that all rewards and returned stake are paid to a separate cold wallet, and only keep the minimum necessary balance on the hot key used by the node. Please find the details of the Beneficial Address in the docs below:

[Private Key Security](/node-hosting/private-key-security)
{% endhint %}

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/sW96dKDW7zUIh6kXse1g" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 4. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/9ifO8ZhSDPqFBgpJeg0v" alt=""><figcaption></figcaption></figure>

## 5. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/jQlfE6EZrdMjZP3SUHv6" alt=""><figcaption></figcaption></figure>

Now the Node is fully up and running. You could just leave it there to run tasks automatically.

The Node could be paused or stopped at any time by clicking the control buttons. If the node is in the middle of running a task, after clicking the buttons, the node will go into the "pending" status and continue with the running task. When the task is finished, the node will pause/stop automatically.

The difference between pausing and stopping is that pausing will not cause the staked CNX tokens to be returned, so that the transaction costs less gas fee than stopping. If you have a plan of going back, you could use pausing rather than stopping.


# Start a Node - Octa

Start a node on Octa.space

The Crynux Node can be easily started on [Octa](https://marketplace.octa.space/) using Docker images.

## 1. Go to the Octa Marketplace and find the Crynux app

Visit the Octa Marketplace in a browser:

{% embed url="<https://marketplace.octa.space>" %}

Search for `Crynux` and click on the app:

<figure><img src="/files/gCPKbqegcDvYp7PPUNnD" alt=""><figcaption></figcaption></figure>

## 2. Select the GPU to start the Docker container

<figure><img src="/files/bdPu92c45v8Me0LvtX3c" alt=""><figcaption></figcaption></figure>

Select the GPU that fits your need. And then click "Configure".

## 3. Configure the Docker container

<figure><img src="/files/nbiCdlhOWpwkGAC5CnUl" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
**Please use the latest version tag to start the container**

you could find the available tags at:

[**https://github.com/crynux-network/crynux-node/pkgs/container/crynux-node/versions**](https://github.com/crynux-network/crynux-node/pkgs/container/crynux-node/versions)

For example, if you want to run the 3.5.1 version of the Crynux Node under Base Network, use the image link below:

`ghcr.io/crynux-network/crynux-node:3.5.1-base`
{% endhint %}

Expose port `7412` for the remote access of the WebUI.

`100 GB` of disk space will be enough for normal operations of the node.

After you're done, click "Deploy" to start the Docker container:

<figure><img src="/files/CqCCYnLOQ5KY47fuNv88" alt=""><figcaption></figcaption></figure>

Once Octa pulls and prepares the image on the node, it will start the container. To track progress, check the `Status` field in the session item. For more detailed insights, click the `View logs` button in the `Actions` column.

<figure><img src="/files/EbVhK2yDLdi62ge1dYZa" alt=""><figcaption></figcaption></figure>

## 4. Find the URL to access the WebUI

Once the container has started, the `Status` will change to `Service configured`. Then, click on the session item to find the URL for accessing the WebUI:

<figure><img src="/files/Hx4gloAfAXcdLU18ELN4" alt=""><figcaption></figcaption></figure>

Click on the link below `HTTP Services`, and you will be redirected to the WebUI in the browser:

<figure><img src="/files/pigpuZSEs17CBXtn7Ksq" alt=""><figcaption></figcaption></figure>

## 5. Prepare the wallet

A wallet with enough CNX tokens must be provided to the node. If this is the first time you start a node, click the "Create New Wallet" button and follow the instructions to create a new wallet and finish the backup of the private keys. Join the Crynux Discord to learn more about acquiring CNX tokens and starting a node.

<figure><img src="/files/7EUckTRp0bfLYj5p9ZyR" alt=""><figcaption></figcaption></figure>

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## 6. Wait for the system initialization to finish

If this is the first time you start a node, it could take quite a long while for the system to initialize. The most time consuming step is to download \~40GB of the commonly used model files from the Huggingface. The time may vary depending on your network speed.

After the models are downloaded, a test image generation task will be executed locally to examine the capability of your device. If the device is not capable to generate images, or the generation speed is too slow, the node will not be able to join the network. If the task is finished successfully, the initialization is completed:

<figure><img src="/files/9ifO8ZhSDPqFBgpJeg0v" alt=""><figcaption></figcaption></figure>

## 7. Join the Crynux Network

The Crynux Node will try to join the network automatically every time it is started. After the transaction is confirmed on-chain, the node has successfully joined the network. When the node is selected by the network to execute a task, the task will start automatically, and the tokens will be transferred to the node wallet after the task is finished.

<figure><img src="/files/jQlfE6EZrdMjZP3SUHv6" alt=""><figcaption></figcaption></figure>

Now the Node is fully up and running. You could just leave it there to run tasks automatically.

The Node could be paused or stopped at any time by clicking the control buttons. If the node is in the middle of running a task, after clicking the buttons, the node will go into the "pending" status and continue with the running task. When the task is finished, the node will pause/stop automatically.

The difference between pausing and stopping is that pausing will not cause the staked CNX tokens to be returned, so that the transaction costs less gas fee than stopping. If you have a plan of going back, you could use pausing rather than stopping.


# Private Key Security

## Beneficial Address

### Keep funds off the node key

The node’s operational private key is a hot key that must remain online to sign requests and transactions. Any hot key is exposed to risks from malware, misconfiguration, or a compromised host. If that key also controls funds, an attacker can move them immediately after a breach. To safeguard funds, the system design should keep the node key strictly operational and separate from any address that holds or receives tokens.

### Beneficial address: concept and setup

A beneficial address is a separate cold wallet that receives all tokens associated with your node, including emissions, stake refunds, and Relay withdrawals. You bind your node address to this beneficial address with a one-time on-chain transaction; the binding is immutable. The private key of the beneficial address never needs to be online and is not used by the node or the Relay.

To set it up, create a new offline wallet for the beneficial address (for example, a hardware wallet or an air-gapped wallet), record the address securely, and submit the on-chain binding from your node address to the beneficial address. Then run your node with the operational (hot) key as usual.

### How it works

After you bind your node address to a beneficial address on-chain, that binding becomes the single source of truth for payouts. When emissions accrue, stake is refunded, or a Relay withdrawal is processed, the system looks up the on-chain binding and sends tokens to the beneficial address—never to the node address. The Relay independently reads the binding on-chain before sending, so a compromised UI or host cannot spoof the destination. Because the beneficial key remains offline, even if the node’s hot key is exposed, an attacker cannot change the binding or divert funds.

## Bind the Beneficial Address

Bind the beneficial address in Crynux Portal:

<https://portal.crynux.io>

Connect the node’s operational wallet to the Portal to view the current binding and set a beneficial address in the dashboard.

The binding is per Crynux L2 chain, and you may set different beneficial addresses on different chains. Stake refunds and emissions are paid to the beneficial address bound on the chain where the node initially staked. Relay withdrawals let you choose a destination chain; tokens are sent to the beneficial address bound on that destination chain.


# Assign GPU to the Node

How to run multiple nodes on a single device with multiple GPUs

If you have multiple GPUs in a single computer, you can optimize performance by starting multiple nodes on the computer and assigning each GPU to a different node.

To enable GPU assignment, use the Docker version of Crynux Node. For a guide on the basics of starting a Crynux Node as a Docker container, please refer to the tutorial below:

{% content-ref url="/pages/gGypoNA8XJ1TX4aGfQmE" %}
[Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker)
{% endcontent-ref %}

## Find the ID of the specific GPU

If you want to assign a specific GPU to a node, you must find the ID of the GPU first. This can be done using the `nvidia-smi` toolkit. Start a terminal and run the following command:

```
$ nvidia-smi
```

And you will get the output similar to the following:

<figure><img src="/files/qPIWVb5WpQBZE85ipMq5" alt=""><figcaption></figcaption></figure>

Find the ID as highlighted in the image above. In this case, we have a single GPU installed in the computer, the ID of the GPU is `0`.

## GPU assignment using Docker Compose

In the `docker-compose.yml` file, find the following section:

```
deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
```

And add another line below:

<pre><code><strong>deploy:
</strong>      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["0"]
</code></pre>

## GPU assignment using command line

The GPU id could also be given to the container in the starting command. If you are starting the container using the following command before:

```
docker run -p 7412:7412 --name crynux_node --gpus all ghcr.io/crynux-network/crynux-node:latest
```

You could change it to:

```
docker run -p 7412:7412 --name crynux_node --gpus '"device=0"' ghcr.io/crynux-network/crynux-node:latest
```

The change is on the `--gpus` argument, from `all`, which provides all the GPUs to the container, to `'"device=0"'`, which provides only the GPU with id `0`.

## Start multiple containers for each of the GPUs on the same computer

For each of the GPUs, follow the tutorial to clone the docker compose project:

{% content-ref url="/pages/gGypoNA8XJ1TX4aGfQmE" %}
[Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker)
{% endcontent-ref %}

For example, if you have 3 GPUs on the same computer, just clone the docker compose project 3 times, after renaming the folders, you have 3 working folders locally:

```
$ ls
crynux_node_docker_compose_1  crynux_node_docker_compose_2  crynux_node_docker_compose_3
```

In each of the working folders, find the `docker-compose.yml` file, and edit the content:

#### 1. Change the name, service name and the container name, so that every container is using a different one:

from:

```
name: "crynux_node"
services:
  crynux_node:
    container_name: crynux_node
```

to:

```
name: "crynux_node_2"
services:
  crynux_node_2:
    container_name: crynux_node_2
```

#### 2. Add a line to specify the GPU id as mentioned above:

<pre><code><strong>deploy:
</strong>      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["0"]
</code></pre>

#### 3. Change the exposing port. So that every container is using a different port:

from:

```
ports:
  - "127.0.0.1:7412:7412"
```

to:

```
ports:
  - "127.0.0.1:7413:7412"
```

for the second container. And use `7414` for the third one.

The complete `docker-compose.yml` files for each of the 3 containers is shown below:

#### Node 1

`crynux_node_docker_compose_1/docker-compose.yml`

```
---
version: "3.8"
name: "crynux_node"

services:
  crynux_node:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node
    restart: unless-stopped
    ports:
      - "127.0.0.1:7412:7412"
    volumes:
      - "./data:/app/data"
      - "./config:/app/config"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["0"]
```

#### Node 2

`crynux_node_docker_compose_2/docker-compose.yml`

```
---
version: "3.8"
name: "crynux_node_2"

services:
  crynux_node_2:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node_2
    restart: unless-stopped
    ports:
      - "127.0.0.1:7413:7412"
    volumes:
      - "./data:/app/data"
      - "./config:/app/config"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["1"]
```

#### Node 3

`crynux_node_docker_compose_3/docker-compose.yml`

```
---
version: "3.8"
name: "crynux_node_3"

services:
  crynux_node_3:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node_3
    restart: unless-stopped
    ports:
      - "127.0.0.1:7414:7412"
    volumes:
      - "./data:/app/data"
      - "./config:/app/config"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["2"]
```

Finally, in each of the folders, run the `docker compose up` command to start the container:

```
$ cd crynux_node_docker_compose_1
$ docker compose up -d
$ cd ../crynux_node_docker_compose_2
$ docker compose up -d
$ cd ../crynux_node_docker_compose_3
$ docker compose up -d
```


# Proxy Settings

Sometimes a proxy is required to access Huggingface and Civitai in your network environment. Crynux Node supports to specify the proxy in the config file.

## Locate the config file

{% tabs %}
{% tab title="Windows" %}
Go to the directory where you click `Crynux Node.exe`, there is a sub directory with name `config`, the config file can be found inside with name `config.yml`.
{% endtab %}

{% tab title="Mac" %}
The config folder of the Mac app locates inside your home diretory at:

`~/Library/Application\ Support/crynux.io/Crynux\ Node/`

To access this folder, open a terminal window and type in the following command:

`$ open ~/Library/Application\ Support/crynux.io/Crynux\ Node/`

And the `config.yml` is located inside under the `config` folder.
{% endtab %}

{% tab title="Docker" %}
**If you have mounted the config directory outside of the container**

find the config file `config.yml` in the mounted config directory on the host machine. Which should be `config` folder inside the project root, if you have followed the tutorial [Start a Node - Docker](/node-hosting/start-a-node/start-a-node-docker).

**If you have not mounted the config directory**

the config file can be found inside the container as `/app/config/config.yml`.
{% endtab %}

{% tab title="Linux" %}
If you downloaded the binary release version of Linux server, the config file `config.yml` can be found in the `config` folder of the project root.
{% endtab %}

{% tab title="Source Code" %}
The config file is located at `config/config.yml`, relative to the project root folder.
{% endtab %}
{% endtabs %}

## Fill in the proxy settings

Open the `config.yml` file with a text editor, and find the section below:

```
---
task_config:
  proxy:
    host: ''
    password: ''
    port: 8080
    username: ''
```

Just fill in the fields according to your proxy settings, and restart the node.

If your proxy requires authentication, fill in the `username` and `password` fields accordingly, otherwise just leave the fields empty.

If the `host` is not set, the node will try to use the proxy settings in the environment variables, which will be the value given in `HTTPS_PROXY`. No proxy will be used if this environment variable is not set.

Below is an example of using a proxy at localhost, with no proxy authentication:

```
---
task_config:
  proxy:
    host: 'http://127.0.0.1'
    password: ''
    port: 33210
    username: ''
```


# Docker Compose Options

Start the Node using Docker Compose

The Docker container of the Node could also be started using Docker Compose, for more convenient configurations.

## Start the container using Docker Compose

#### 1. Create an empty working directory

```sh
$ mkdir h_node
$ cd h_node
```

#### 2. Create a file named `docker-compose.yml` in the working directory:

```yaml
---
version: "3.8"
name: "crynux_node"

services:
  h_node:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node
    restart: unless-stopped
    ports:
      - "127.0.0.1:7412:7412"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]

```

#### 3. Start the Docker container

```sh
docker compose up -d
```

Now you should already be able to access the WebUI from the browser.

## Mount the Model Cache Folder

Since the model preloading takes a long time, often we want to persist the model cache folder outside of the Docker container so that it survives the container recreation during updates. This is easily done by mounting the data folder `/app/data` to a local folder on the host machine:

#### 1. Create an empty data folder inside the working directory

```sh
$ ls .
docker-compose.yml

$ mkdir data

$ ls .
data/ docker-compose.yml
```

#### 2. Add the mounting point in the `docker-compose.yml` file

```yaml
---
version: "3.8"
name: "crynux_node"

services:
  h_node:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node
    restart: unless-stopped
    ports:
      - "127.0.0.1:7412:7412"
    volumes:
      - "./data:/app/data"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
```

#### 3. Start the Docker container

```sh
docker compose up -d
```

## Mount the Config File

The configuration file could also be mounted to the local folder, so the config won't be overridden by the container recreating. It is also easier to edit the config file outside of the Docker container.

#### 1. Create an empty config folder inside the working directory

```sh
$ ls .
data/ docker-compose.yml

$ mkdir config

$ ls .
config/ data/ docker-compose.yml
```

#### 2. Add the mounting point in the `docker-compose.yml` file

```yaml
---
version: "3.8"
name: "crynux_node"

services:
  h_node:
    image: ghcr.io/crynux-network/crynux-node:latest
    container_name: crynux_node
    restart: unless-stopped
    ports:
      - "127.0.0.1:7412:7412"
    volumes:
      - "./data:/app/data"
      - "./config:/app/config"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
```

#### 3. Start the Docker container

```sh
docker compose up -d
```

#### 4. A config file will be created automatically after the container creation

```sh
$ ls config/
config.yml
```

For an explanation of all the config items, please refer to the [Advanced Configuration](/node-hosting/advanced-configuration).


# Advanced Configuration

After mounting the config folder to a local folder. The config file will be created inside the config folder. Here is an explanation of all the config items.

```yaml
log:
  # The directory to save the log files
  dir: "logs"
  
  # Log level could be "DEBUG", "INFO", "WARNING", "ERROR"
  level: "INFO"

ethereum:
  # The private key of the wallet
  # Must be filled if headless mode is enabled
  # If headless mode is not enabled,
  # the private key can also be filled using the WebUI.
  privkey: ""
  
  # The JSON RPC endpoint of the Blockchain node
  # Here we use the private chain for the Hydrogen Network
  provider: "https://block-node.crynux.ai/rpc"
  
  # The Blockchain params
  # Leave it as is for the private chain used in the Hydrogen Network
  chain_id: 42
  gas: 42949670
  gas_price: 1

  # The deployed addresses of the smart contracts
  contract:
    token: "0xB627D84BFB8cC311A318fEf679ee498F822A0C7C"
    node: "0x73F8eAD4d29e227958aB5F3A3e38092271500865"
    task: "0x3f4e524d5Ff53D0e98eE5A37f81f4F21551502B2"

# The directory to store the temp files related to the running task
task_dir: tasks

# The database used to store the local state data
# The data will not be large. A sqlite file is more than enough
# There is no need to mount this file to the host machine to persist it
db: sqlite+aiosqlite:///db/server.db

# The URL of the Relay
relay_url: "https://relay.h.crynux.ai"

# The directory that stores the distribution files of the WebUI
web_dist: dist

# Whether to enable the headless mode
headless: false

task_config:
  # The directory to store the temp images for a task.
  output_dir: "/app/data/images"
  
  # The directory to cache the huggingface model files
  hf_cache_dir: "/app/data/huggingface"
  
  # The directory to cache the external model files
  # Such as the LoRA models from Civitai
  external_cache_dir: "/app/data/external"
  
  # The directory to store the temp logs generated
  # by the task execution engine
  inference_logs_dir: "/app/inference-logs"
  
  # The directory that stores the source code of the task execution engine
  script_dir: "/app/stable-diffusion-task"

  # Models that will be preloaded before any task execution
  # Other models specified by the task
  # will be downloaded during the task execution
  preloaded_models:
    base:
      - id: "runwayml/stable-diffusion-v1-5"
      - id: "emilianJR/chilloutmix_NiPrunedFp32Fix"
      - id: "stabilityai/stable-diffusion-xl-base-1.0"
      - id: "stabilityai/stable-diffusion-xl-refiner-1.0"
    controlnet:
      - id: "lllyasviel/sd-controlnet-canny"
      - id: "lllyasviel/control_v11p_sd15_openpose"
      - id: "diffusers/controlnet-canny-sdxl-1.0"
    vae: []
    
  # The proxy server used when downloading models.
  proxy:
    host: "http://127.0.0.1"
    port: 33210

# If the node dies right after submitting the commitments,
# and before disclosing the result on-chain. 
# And if the data is corrupted inside the container,
# which prevents the node from starting again.
# The result from the previous task execution must be fetched from
# the logs of the dead container and filled here.
# So the node could continue with the unfinished task correctly.
last_result: ""
```


# How to Run LLM using Crynux Network

Running LLM tasks with various open-source models can be as simple as calling an OpenAI-compliant API via the Crynux Network. The example below demonstrates how to send an LLM chat completion task to the Crynux Network using the official OpenAI SDK:

{% tabs %}
{% tab title="Python" %}

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="q3hXHA_8O0LuGJ1_tou4_KamMlQqAo-aYwyAIDttdmI=", # For public demonstration only, strict rate limit applied.
    timeout=60,
    max_retries=1,
)

res = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the capital of France?",
        },
    ],
    stream=False,
    extra_body={
        "vram_limit": 24,
    }
)

print(res)
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://bridge.crynux.io/v1/llm",
  apiKey: "q3hXHA_8O0LuGJ1_tou4_KamMlQqAo-aYwyAIDttdmI=", // For public demonstration only, strict rate limit applied.
  timeout: 60000,
  maxRetries: 1,
});

async function main() {
  try {
    const chatCompletion = await client.chat.completions.create({
      model: "Qwen/Qwen2.5-7B-Instruct",
      messages: [
        {
          role: "user",
          content: "What is the capital of France?",
        },
      ],
      stream: false,
      vram_limit: 24,
    });
    console.log("Chat completion response:", chatCompletion);

    return chatCompletion;
  } catch (error) {
    console.error("Error:", error);
  }
}

main();
```

{% endtab %}
{% endtabs %}

This code is standard for invoking OpenAI models through their API. The only modification is the `base_url`, which is changed from the OpenAI URL to the official Crynux Bridge. A live version of this JavaScript code, embedded in a CodePen webpage, allows you to input arbitrary text and receive a response:

{% embed url="<https://codepen.io/Luke-Weber/pen/RNWaOgL>" %}

The API, provided by the official Crynux Bridge, supports both OpenAI-compliant `/completions` and `/chat/completions` endpoints. Features like streaming, tool-calling, and numerous other configuration options are also supported. For a comprehensive list of supported features, please refer to the[ Crynux Bridge documentation](/application-development/crynux-bridge).

## GPU VRAM Requirement

The `vram_limit` parameter specifies the minimum VRAM required to execute the task. Crynux Network uses this value to route the task to a node with sufficient GPU memory. This requirement is directly tied to the model size; for example, the 8B model used in the example runs comfortably on a 24GB card.

If this parameter is omitted, the Crynux Bridge defaults to `24` (GB). Therefore, when using a model larger than 8B that requires more memory, you must explicitly set `vram_limit` to a higher value. Failure to do so may result in the task being assigned to an insufficient node, causing a timeout or failure.

## Advanced Usage

For more advanced use cases like Tool Calling, Structured Output, and integrations with LangChain/LangGraph, please refer to the following guides:

{% content-ref url="/pages/YWBZG1xi78hEF57FbIrU" %}
[Tool Use/Function Calling](/application-development/how-to-run-llm-using-crynux-network/tool-use)
{% endcontent-ref %}

{% content-ref url="/pages/X4h8fSteUIuCspbHdMV1" %}
[Structured Output](/application-development/how-to-run-llm-using-crynux-network/structured-ouput)
{% endcontent-ref %}

{% content-ref url="/pages/d3Y2YDAR411um8SH0b5e" %}
[Vision Language Models (VLM)](/application-development/how-to-run-llm-using-crynux-network/vision-language-models)
{% endcontent-ref %}

{% content-ref url="/pages/tlRR30Y1MoplaOYmaQwF" %}
[Integration with LangChain & LangGraph](/application-development/how-to-run-llm-using-crynux-network/langchain)
{% endcontent-ref %}

{% content-ref url="/pages/aEuPw0R8uRvksFPmFXzp" %}
[Hermes Agent Integration](/application-development/how-to-run-llm-using-crynux-network/hermes-agent-integration)
{% endcontent-ref %}

The API Key in the example code is for public demonstration purposes only and has a strict rate limit, making it unsuitable for production environments. To use the Crynux Network in production, choose one of the following methods:

## Method 1: Using the Official Crynux Bridge

You can request a separate API Key with a higher quota from the Crynux Discord server. Join the server and request new keys from an admin in the "applications" channel.

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

## Method 2: Hosting Your Own Crynux Bridge

You can host your own instance of the Crynux Bridge to provide private APIs for your application. This approach gives you greater control over various system aspects, including reliability and speed-related configurations.

Starting a Crynux Bridge is as straightforward as running a Docker container. An additional requirement is a wallet funded with sufficient (test) CNX to cover the tasks you run on the network. And at this moment, you can get test CNXs for free in the [Crynux Discord](https://discord.gg/y8YKxb7uZk) as well.

Crynux Bridge is fully open-sourced on [GitHub](https://github.com/crynux-network/crynux-bridge). A step-by-step guide for starting a Crynux Bridge instance is available in the following document:

{% content-ref url="/pages/kiPKEEQwV77hmCOGd58B" %}
[Crynux Bridge](/application-development/crynux-bridge)
{% endcontent-ref %}

## Method 3: Sending Tasks Directly to the Blockchain

You can bypass the Crynux Bridge entirely and interact directly with the blockchain and Crynux Relay to send tasks. Crynux SDKs are available in various languages and can be embedded directly into your code to run LLM tasks. Please consult the Crynux SDK documentation for detailed usage instructions:

{% content-ref url="/pages/T7IKwH1gpqgUPdW6UYq0" %}
[Crynux SDK](/application-development/crynux-sdk)
{% endcontent-ref %}


# Supported Models

Crynux theoretically supports any model that can be executed by the Hugging Face `transformers` library. To use a specific model, you simply need to specify its Hugging Face model ID in the task configuration. The Crynux Nodes will then automatically fetch the model from Hugging Face and execute the task.

The primary practical limitation on the number and size of models Crynux can support is the maximum available VRAM on the nodes within the Crynux Network. Currently, the nodes with the largest VRAM capacity offer 80GB.

For example, below is a list of popular models that can be used in the Crynux Network. Each entry includes the model name and a direct link to its Hugging Face repository.

{% hint style="success" %}
If a model isn't on this list, feel free to try it out as long as you're confident it's compatible with the `transformers` library and there's sufficient VRAM available on the network.
{% endhint %}

### DeepSeek Models

| Model ID                                  | Hugging Face Link                                                                                             |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B | [deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-7B   | [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)     |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B  | [deepseek-ai/DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B)   |

### Qwen Models

| Model ID                 | Hugging Face Link                                                           |
| ------------------------ | --------------------------------------------------------------------------- |
| Qwen/Qwen3-4B            | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B)                       |
| Qwen/Qwen3-8B            | [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)                       |
| Qwen/Qwen2.5-7B          | [Qwen/Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B)                   |
| Qwen/Qwen2.5-7B-Instruct | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |

### NousResearch Models

| Model ID                           | Hugging Face Link                                                                               |
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
| NousResearch/Hermes-3-Llama-3.1-8B | [NousResearch/Hermes-3-Llama-3.1-8B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B) |
| NousResearch/Hermes-3-Llama-3.2-3B | [NousResearch/Hermes-3-Llama-3.2-3B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.2-3B) |


# Tool Use/Function Calling

Crynux Bridge supports the standard OpenAI Tool Use (Function Calling) API. This allows you to describe functions to the model, and have the model intelligently choose to output a JSON object containing arguments to call one or many of those functions.

{% hint style="warning" %}
It is important to note that **not all open-source models support tool calling**. You must choose a model that has been specifically trained or fine-tuned for this capability.

Generally, models with "Instruct" in their name (Instruction Fine-Tuned models) are more likely to support tool use. For example, if you are using the Qwen model family, the base model `Qwen/Qwen2.5-7B` might not support tool calls effectively, whereas the instruction-tuned version `Qwen/Qwen2.5-7B-Instruct` is designed to handle such tasks.

Always check the model card or documentation of the specific model you intend to use to confirm its support for function calling or tool use.
{% endhint %}

The following examples demonstrate how to use the tool calling feature with the **OpenAI SDK**, the dedicated **langchain-crynux** library, and the standard **langchain-openai** library.

{% tabs %}
{% tab title="OpenAI SDK" %}
When using the official `openai` Python SDK, you define tools as dictionaries and pass them to the `tools` parameter. The `vram_limit` is passed via `extra_body`.

```python
from openai import OpenAI
import json

client = OpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="your-api-key",
)

# 1. Define the tool
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

# 2. Call the model
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
    tool_choice="auto",
    extra_body={
        "vram_limit": 24
    }
)

response_message = completion.choices[0].message
tool_calls = response_message.tool_calls

if tool_calls:
    print("Tool calls detected:")
    for tool_call in tool_calls:
        print(f"Function: {tool_call.function.name}")
        print(f"Arguments: {tool_call.function.arguments}")
```

{% endtab %}

{% tab title="LangChain-Crynux" %}
The `langchain-crynux` library provides a drop-in replacement for `ChatOpenAI` optimized for Crynux. It handles `vram_limit` as a first-class parameter.

```python
from langchain_crynux import ChatCrynux
from langchain_core.tools import tool

# 1. Define the tool using the @tool decorator
@tool
def get_current_weather(location: str, unit: str = "celsius"):
    """Get the current weather in a given location"""
    # Simulate a weather API response
    return {
        "location": location,
        "temperature": "22",
        "unit": unit,
        "condition": "Sunny"
    }

# 2. Initialize the ChatCrynux model
llm = ChatCrynux(
    base_url="https://bridge.crynux.io/v1/llm",
    model="Qwen/Qwen2.5-7B-Instruct",
    vram_limit=24,           # Specify VRAM requirement directly
    api_key="your-api-key"
)

# 3. Bind the tool to the model
llm_with_tools = llm.bind_tools([get_current_weather])

# 4. Invoke the model
query = "What's the weather like in Boston today?"
response = llm_with_tools.invoke(query)

print("Tool Calls:", response.tool_calls)
```

{% endtab %}

{% tab title="LangChain-OpenAI" %}
If you prefer using the standard `langchain-openai` library, you can pass the Crynux-specific `vram_limit` parameter inside the `model_kwargs` dictionary.

```python
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool

# 1. Define the tool
@tool
def get_current_weather(location: str, unit: str = "celsius"):
    """Get the current weather in a given location"""
    return {
        "location": location,
        "temperature": "22",
        "unit": unit,
        "condition": "Sunny"
    }

# 2. Initialize ChatOpenAI with Crynux configuration
llm = ChatOpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="your-api-key",
    model="Qwen/Qwen2.5-7B-Instruct",
    # Pass Crynux parameters in model_kwargs
    model_kwargs={
        "vram_limit": 24
    }
)

# 3. Bind and invoke
llm_with_tools = llm.bind_tools([get_current_weather])
response = llm_with_tools.invoke("What's the weather like in Boston today?")

print("Tool Calls:", response.tool_calls)
```

{% endtab %}
{% endtabs %}


# Structured Output

While obtaining unstructured text responses is useful, building reliable AI applications often requires structured data (like JSON) to interface with other systems.

## The Challenge with Open Source Models

OpenAI's official API offers native "JSON Mode" and "Structured Outputs" (via `response_format`), which guarantee that the output matches a specific JSON schema.

However, **most open-source models and OpenAI-compatible APIs do not fully support these native strict modes**. If you simply ask an open-source model to "output JSON", it might:

* Add conversational text before or after the JSON.
* Make syntax errors (missing brackets, trailing commas).
* Hallucinate keys that aren't in your schema.

## The Solution: Simulating via Tool Use

Fortunately, we can reliably achieve structured output by leveraging [**Tool Use (Function Calling)**](/application-development/how-to-run-llm-using-crynux-network/tool-use).

Since models like `Qwen2.5-7B-Instruct` are fine-tuned to generate valid JSON arguments for tool calls, we can "trick" the model into generating structured data by:

1. Defining a "tool" whose parameters match our desired output schema.
2. Forcing the model to "call" this tool.
3. Parsing the arguments of the tool call as our final output.

**LangChain** makes this pattern extremely easy with the `.with_structured_output()` method. It automatically handles the schema conversion, tool binding, and output parsing for you.

## Examples

The following examples show how to extract a structured `CalendarEvent` object from natural language using the Crynux Network.

{% hint style="info" %}
**Model Selection**: Ensure you use an **Instruct** model (e.g., `Qwen/Qwen2.5-7B-Instruct`) that supports tool calling. Base models usually cannot handle this reliably.
{% endhint %}

{% tabs %}
{% tab title="LangChain-Crynux" %}
The `langchain-crynux` package supports `with_structured_output` out of the box. It defaults to using **Tool Use** (method="function\_calling") to ensure compatibility with open-source models on the Crynux Network.

```python
from langchain_crynux import ChatCrynux
from pydantic import BaseModel, Field

# 1. Define your desired output structure using Pydantic
class CalendarEvent(BaseModel):
    name: str = Field(description="The name of the event")
    date: str = Field(description="The date of the event, in YYYY-MM-DD format")
    participants: list[str] = Field(description="List of people participating")

# 2. Initialize the model
llm = ChatCrynux(
    base_url="https://bridge.crynux.io/v1/llm",
    model="Qwen/Qwen2.5-7B-Instruct",
    vram_limit=24,
    api_key="your-api-key"
)

# 3. Configure structured output
# This automatically converts the Pydantic model to a tool definition
# and configures the LLM to use it.
structured_llm = llm.with_structured_output(CalendarEvent)

# 4. Invoke with natural language
text = "Alice and Bob are going to a Science Fair on Friday, 2024-05-10."
result = structured_llm.invoke(text)

# 5. The result is an instance of your Pydantic model
print(f"Event: {result.name}")
print(f"Date: {result.date}")
print(f"Participants: {result.participants}")
# Output:
# Event: Science Fair
# Date: 2024-05-10
# Participants: ['Alice', 'Bob']
```

{% endtab %}

{% tab title="LangChain-OpenAI" %}
You can also use the standard `langchain-openai` library. Under the hood, it uses the OpenAI tool calling API provided by Crynux Bridge.

```python
from langchain_openai import ChatOpenAI
from pydantic import BaseModel, Field

# 1. Define your desired output structure
class CalendarEvent(BaseModel):
    name: str = Field(description="The name of the event")
    date: str = Field(description="The date of the event, in YYYY-MM-DD format")
    participants: list[str] = Field(description="List of people participating")

# 2. Initialize the model
llm = ChatOpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="your-api-key",
    model="Qwen/Qwen2.5-7B-Instruct",
    model_kwargs={"vram_limit": 24}
)

# 3. Configure structured output
# We explicitly set method="function_calling" to ensure it uses tool calls
# rather than trying to use native 'json_mode' which might not be supported.
structured_llm = llm.with_structured_output(CalendarEvent, method="function_calling")

# 4. Invoke
text = "Meeting with Charlie about the project launch on Oct 15th, 2024."
result = structured_llm.invoke(text)

print(result)
# Output:
# name='Project Launch Meeting' date='2024-10-15' participants=['Charlie']
```

{% endtab %}
{% endtabs %}


# Vision Language Models (VLM)

The Crynux Network supports Vision Language Models (VLM) through the same OpenAI-compatible API. You can send images along with text prompts to these models to perform tasks like image captioning, visual question answering, and more.

## Usage

To use a VLM, you need to construct the `messages` payload with both text and image content. The image should be provided as a base64-encoded string within a data URL.

{% hint style="info" %}
**Note**: Currently, the Crynux Network only supports passing images as base64-encoded data URLs (e.g., `data:image/jpeg;base64,...`). Passing images via HTTP/HTTPS URLs is not supported.
{% endhint %}

{% tabs %}
{% tab title="Python" %}

```python
import base64
from openai import OpenAI

# Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

# Path to your image
image_path = "path/to/your/image.jpg"

# Getting the base64 string
base64_image = encode_image(image_path)

client = OpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="YOUR_API_KEY", # Replace with your actual API key
    timeout=60,
    max_retries=1,
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    },
                },
            ],
        }
    ],
    max_tokens=300,
    extra_body={
        "vram_limit": 24, # Ensure the node has enough VRAM
    }
)

print(response.choices[0].message.content)
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
import fs from 'fs';
import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: "https://bridge.crynux.io/v1/llm",
  apiKey: "YOUR_API_KEY", // Replace with your actual API key
  timeout: 60000,
  maxRetries: 1,
});

// Function to encode the image
function encodeImage(imagePath) {
  const image = fs.readFileSync(imagePath);
  return Buffer.from(image).toString('base64');
}

const imagePath = "path/to/your/image.jpg";
const base64Image = encodeImage(imagePath);

async function main() {
  const response = await openai.chat.completions.create({
    model: "Qwen/Qwen2.5-VL-3B-Instruct",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What is in this image?" },
          {
            type: "image_url",
            image_url: {
              "url": `data:image/jpeg;base64,${base64Image}`,
            },
          },
        ],
      },
    ],
    max_tokens: 300,
    vram_limit: 24, // Ensure the node has enough VRAM
  });

  console.log(response.choices[0].message.content);
}

main();
```

{% endtab %}
{% endtabs %}

## VRAM Requirement

Just like with text-only models, you should specify the `vram_limit` in the `extra_body` (Python) or directly in the options (JavaScript) to ensure the task is routed to a node with sufficient GPU memory. For the models listed above, a `vram_limit` of `24` GB is generally sufficient.


# Integration with LangChain & LangGraph

The Crynux Bridge provides an OpenAI-compatible API, making it seamless to integrate with [LangChain](https://www.langchain.com/) and [LangGraph](https://langchain-ai.github.io/langgraph/). You can use Crynux Bridge API as a drop-in replacement for OpenAI API in your AI applications.

There are two ways to use Crynux with LangChain:

1. **Using `langchain-crynux`**: A dedicated package optimized for Crynux.
2. **Using `langchain-openai`**: The standard OpenAI integration package.

## Method 1: Using `langchain-crynux` (Recommended)

The `langchain-crynux` package is a drop-in replacement for `ChatOpenAI` that is specifically tuned for the Crynux Network. It provides first-class support for Crynux-specific parameters like `vram_limit`.

### Installation

```bash
pip install langchain-crynux
```

### Usage

```python
import os
from langchain_crynux import ChatCrynux

# You can set the API key in the environment variable
# os.environ["OPENAI_API_KEY"] = "your-api-key"

chat = ChatCrynux(
    base_url="https://bridge.crynux.io/v1/llm",
    model="Qwen/Qwen2.5-7B-Instruct",
    vram_limit=24,  # Specify the required VRAM in GB
    # api_key="your-api-key", # Or pass it directly
)

response = chat.invoke("Hello, introduce yourself.")
print(response.content)
```

The `vram_limit` parameter is essential for the Crynux Network to route your task to a node with sufficient GPU memory. The default is 24GB.

## Method 2: Using `langchain-openai`

Since the Crynux Bridge is fully compatible with the OpenAI API, you can also use the standard `langchain-openai` library. This is useful if you already have an existing project using LangChain's OpenAI integration.

### Installation

```bash
pip install langchain-openai
```

### Usage

To use `ChatOpenAI` with Crynux, you simply need to override the `base_url` and pass Crynux-specific parameters via `model_kwargs`.

```python
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://bridge.crynux.io/v1/llm",
    api_key="your-api-key", # Use a real key or a dummy one for local bridges
    model="Qwen/Qwen2.5-7B-Instruct",
    temperature=0.7,
    # Pass Crynux-specific parameters in model_kwargs
    model_kwargs={
        "vram_limit": 24
    }
)

messages = [
    ("system", "You are a helpful assistant."),
    ("human", "What is the capital of France?"),
]

ai_msg = llm.invoke(messages)
print(ai_msg.content)
```

## Using with LangGraph

Both methods above return a standard LangChain Runnable, which can be directly used in LangGraph workflows. Here is a simple example of a LangGraph agent using a Crynux model.

### Installation

```bash
pip install langgraph langchain-crynux
```

### Example

```python
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, END
from langchain_crynux import ChatCrynux

# 1. Define the State
class State(TypedDict):
    messages: list

# 2. Initialize the Model
model = ChatCrynux(
    base_url="https://bridge.crynux.io/v1/llm",
    model="Qwen/Qwen2.5-7B-Instruct",
    vram_limit=24,
    api_key="your-api-key"
)

# 3. Define the Nodes
def chatbot(state: State):
    return {"messages": [model.invoke(state["messages"])]}

# 4. Build the Graph
graph_builder = StateGraph(State)
graph_builder.add_node("chatbot", chatbot)
graph_builder.set_entry_point("chatbot")
graph_builder.add_edge("chatbot", END)

graph = graph_builder.compile()

# 5. Run the Graph
response = graph.invoke({"messages": [("user", "Tell me a joke about AI.")]})
print(response["messages"][-1].content)
```


# Hermes Agent Integration

This guide shows the fastest way to connect Hermes Agent to Crynux Bridge as a custom LLM provider.

Hermes Agent is an open-source, self-improving AI agent from Nous Research. It is designed as an autonomous assistant for chat, automation, tools, and coding workflows, and it supports OpenAI-compatible endpoints, so Crynux Bridge can be used directly as a custom provider backend. Learn more on the official website: <https://hermes-agent.nousresearch.com/>.

The core setup is simple:

* set Hermes provider to `custom`
* set Crynux Bridge `base_url`
* set your Crynux API token

## Before You Start

Prepare these three items before configuration:

### 1) Base URL

The standard Crynux Bridge endpoint is:

* `https://bridge.crynux.io/v1/llm`

You can also set the VRAM limit directly in the path:

* `https://bridge.crynux.io/v1/llm/24` means VRAM limit is set to `24`

`vram_limit` is a Crynux-specific routing parameter. It defines the minimum GPU VRAM requirement (in GB) for your request, so Crynux can dispatch the task to nodes with enough GPU memory. If you choose a value that is too low for your model, the task may fail or timeout.

If you do not specify a VRAM limit, the default value is `24`.

### 2) Access Token

The public demo token has strict rate limits and is not suitable for normal use.

To get a free token with better quota:

1. Join the Crynux Discord: <https://discord.gg/y8YKxb7uZk>
2. Go to the **applications** channel
3. Request a Crynux Bridge API token from an admin

### 3) Model

Crynux generally supports open-source models that are compatible with the Hugging Face `transformers` library. In practice, the main limitation is available VRAM on network nodes, so larger models require higher VRAM settings.

Hermes workflows require tool use/function calling support, so choose a model that supports tool calling. Instruction-tuned models are usually safer choices (for example, `Qwen/Qwen2.5-7B-Instruct`). For details, refer to:

* [Tool Use/Function Calling](/application-development/how-to-run-llm-using-crynux-network/tool-use)
* [Supported Models](/application-development/how-to-run-llm-using-crynux-network/supported-models)

{% tabs %}
{% tab title="Interactive Setup (hermes model)" %}
Run:

```bash
hermes model
```

Then in the menu:

1. Choose **Custom endpoint (self-hosted / VLLM / etc.)**
2. API base URL: `https://bridge.crynux.io/v1/llm`
3. API key: paste your Crynux token
4. Model name: for example `Qwen/Qwen2.5-7B-Instruct`
5. Context length: keep auto-detect, or enter a value manually if prompted

Start Hermes:

```bash
hermes
```

{% endtab %}

{% tab title="Config File Setup (Local and Docker)" %}
Use this method for both local runtime and Docker runtime.

* Local runtime files: `~/.hermes/config.yaml` and `~/.hermes/.env`
* Docker runtime files (with `-v ~/.hermes:/opt/data`): same host files

Update these exact config items in `~/.hermes/config.yaml`:

* `model.provider`: `custom`
* `model.default`: your selected model (example: `Qwen/Qwen2.5-7B-Instruct`)
* `model.base_url`: `https://bridge.crynux.io/v1/llm`
* `model.api_key`: `${CRYNUX_API_KEY}`

Optional:

* `model.context_length`: set this only if auto-detection is incorrect

#### `~/.hermes/config.yaml`

```yaml
model:
  provider: custom
  default: Qwen/Qwen2.5-7B-Instruct
  base_url: https://bridge.crynux.io/v1/llm
  api_key: ${CRYNUX_API_KEY}
```

#### `~/.hermes/.env`

```bash
CRYNUX_API_KEY=your_real_crynux_token_here
```

After saving, start Hermes:

```bash
hermes
```

Or with Docker:

```bash
docker run -it --rm \
  -v ~/.hermes:/opt/data \
  nousresearch/hermes-agent
```

{% endtab %}
{% endtabs %}


# How to Fine-tune a Stable Diffusion Model using Crynux Network

Fine-tuning Stable Diffusion models on the Crynux Network involves creating a training task and monitoring its progress. Unlike inference tasks, fine-tuning is a long-running process that requires asynchronously creating the task, tracking the task status and downloading results upon completion.

## Fine-tuning Task Execution Process

Before diving into the code examples, let's understand the complete workflow for fine-tuning a Stable Diffusion model on the Crynux Network. The process consists of four main steps:

### 1. Dataset Preparation

The first step is to prepare your training dataset. Crynux Network supports many types of dataset sources:

* **Huggingface Dataset**: You can use any dataset available on [Huggingface](https://huggingface.co/) by specifying its dataset ID (e.g., `"lambdalabs/naruto-blip-captions"`).
* **Irys Network**: The dataset can be stored on the decentralized data network of [Irys](https://irys.xyz/). And provided to the Crynux nodes as a download link.
* **Custom Dataset via URL**: Other downloadable links are also supported. The file can be compressed (ZIP, TAR, etc.) and will be automatically extracted and loaded using the Huggingface dataset library.

Your dataset should contain image-caption pairs, with images in one column and corresponding text captions in another column. The default column names are `"image"` for images and `"text"` for captions, you can also customize these in the task config.

### 2. Model Selection

Fine-tuning tasks create LoRA (Low-Rank Adaptation) models that enhance existing Stable Diffusion models. You need to specify:

* **Base Model**: Choose a pre-trained Stable Diffusion model (e.g., `"runwayml/stable-diffusion-v1-5"` or `"stabilityai/stable-diffusion-xl-base-1.0"`)
* **LoRA Parameters**: Configure how the LoRA adapter will be applied:
  * `rank`: The dimension of LoRA attention (typically 4-64)
  * `target_modules`: Which transformer modules to apply LoRA to (common choices include `["to_k", "to_q", "to_v", "to_out.0"]`)
  * `init_lora_weights`: How to initialize LoRA weights (`"gaussian"` is commonly used)

### 3. Training Parameter Configuration

Set the training hyperparameters that control the learning process:

* **Learning Rate**: The step size for gradient updates (typically 1e-4 to 1e-5)
* **Batch Size**: Number of samples processed together (usually 1-4 for fine-tuning)
* **Training Steps**: Total number of training iterations
* **Learning Rate Scheduler**: How the learning rate changes over time (e.g., `"cosine"` for gradual decay)
* **Image Resolution**: Target resolution for training images (typically 512 or 768)
* **Data Augmentation**: Whether to apply random flips, center crops, etc.

### 4. Task Execution

Once you have configured all parameters, submit the task to the Crynux Network:

1. **Create Task**: Send a POST request with your configuration to create the fine-tuning task
2. **Monitor Progress**: Poll the task status endpoint to track completion
3. **Download Results**: Once complete, download the fine-tuned LoRA model

The fine-tuned model will be returned as a ZIP file containing the LoRA weights that can be loaded into compatible Stable Diffusion inference pipelines.

## Code Examples

The example below demonstrates how to submit a fine-tuning task to the Crynux Network using HTTP requests:

{% tabs %}
{% tab title="Python" %}

```python
import time
import httpx

client = httpx.Client(
    base_url="https://bridge.crynux.io",
    timeout=180,
)

api_key = "q3hXHA_8O0LuGJ1_tou4_KamMlQqAo-aYwyAIDttdmI="  # For public demonstration only

# Fine-tuning configuration
data = {
    "model_name": "crynux-network/stable-diffusion-v1-5",
    "model_variant": "fp16",
    "dataset_url": "https://gateway.irys.xyz/GivF5FBMdJVr6xHT7hi2aE7vH55wVjrtKLpRc2E86icJ",
    "validation_num_images": 4,
    "learning_rate": 0.0001,
    "batch_size": 1,
    "num_train_steps": 100,
    "max_train_steps": 200,
    "lr_scheduler": "cosine",
    "lr_warmup_steps": 0,
    "rank": 4,
    "init_lora_weights": "gaussian",
    "target_modules": ["to_k", "to_q", "to_v", "to_out.0"],
    "center_crop": True,
    "random_flip": True,
    "mixed_precision": "fp16",
    "seed": 42,
    "timeout": 1800,
}

headers = {
    "Authorization": f"Bearer {api_key}",
}

# Step 1: Create fine-tuning task
resp = client.post(
    "/v1/images/models",
    json=data,
    headers=headers,
    timeout=180,
)
resp.raise_for_status()
res = resp.json()
task_id = res["data"]["id"]
print(f"Task ID: {task_id}")

# Step 2: Monitor task status
success = False
while True:
    resp = client.get(f"/v1/images/models/{task_id}/status")
    resp.raise_for_status()
    res = resp.json()
    status = res["data"]["status"]

    if status == "success":
        print("Task completed successfully")
        success = True
        break
    elif status == "failed":
        print("Task failed")
        success = False
        break
    elif status == "running":
        print("Task is still running...")

    time.sleep(60)  # Check status every minute

# Step 3: Download results if successful
if success:
    with client.stream(
        "GET",
        f"/v1/images/models/{task_id}/result",
        headers=headers,
        timeout=180,
    ) as resp:
        resp.raise_for_status()
        with open("finetuned_model.zip", "wb") as f:
            for chunk in resp.iter_bytes():
                f.write(chunk)
    print("Fine-tuned model downloaded as finetuned_model.zip")
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
import fetch from 'node-fetch';

const API_KEY = "q3hXHA_8O0LuGJ1_tou4_KamMlQqAo-aYwyAIDttdmI="; // For public demonstration only
const BASE_URL = "https://bridge.crynux.io";

async function finetuneStableDiffusion() {
    try {
        // Fine-tuning configuration
        const data = {
            model_name: "crynux-network/stable-diffusion-v1-5",
            model_variant: "fp16",
            dataset_url: "https://gateway.irys.xyz/GivF5FBMdJVr6xHT7hi2aE7vH55wVjrtKLpRc2E86icJ",
            validation_num_images: 4,
            learning_rate: 0.0001,
            batch_size: 1,
            num_train_steps: 100,
            max_train_steps: 200,
            lr_scheduler: "cosine",
            lr_warmup_steps: 0,
            rank: 4,
            init_lora_weights: "gaussian",
            target_modules: ["to_k", "to_q", "to_v", "to_out.0"],
            center_crop: true,
            random_flip: true,
            mixed_precision: "fp16",
            seed: 42,
            timeout: 1800,
        };

        const headers = {
            "Authorization": `Bearer ${API_KEY}`,
            "Content-Type": "application/json",
        };

        // Step 1: Create fine-tuning task
        const createResponse = await fetch(`${BASE_URL}/v1/images/models`, {
            method: "POST",
            headers: headers,
            body: JSON.stringify(data),
        });

        if (!createResponse.ok) {
            throw new Error(`Failed to create task: ${createResponse.statusText}`);
        }

        const createResult = await createResponse.json();
        const taskId = createResult.data.id;
        console.log(`Task ID: ${taskId}`);

        // Step 2: Monitor task status
        let success = false;
        while (true) {
            const statusResponse = await fetch(`${BASE_URL}/v1/images/models/${taskId}/status`);
            if (!statusResponse.ok) {
                throw new Error(`Failed to get status: ${statusResponse.statusText}`);
            }

            const statusResult = await statusResponse.json();
            const status = statusResult.data.status;

            if (status === "success") {
                console.log("Task completed successfully");
                success = true;
                break;
            } else if (status === "failed") {
                console.log("Task failed");
                success = false;
                break;
            } else if (status === "running") {
                console.log("Task is still running...");
            }

            // Wait 60 seconds before checking again
            await new Promise(resolve => setTimeout(resolve, 60000));
        }

        // Step 3: Download results if successful
        if (success) {
            const downloadResponse = await fetch(`${BASE_URL}/v1/images/models/${taskId}/result`, {
                headers: headers,
            });

            if (!downloadResponse.ok) {
                throw new Error(`Failed to download results: ${downloadResponse.statusText}`);
            }

            const fs = require('fs');
            const fileStream = fs.createWriteStream('finetuned_model.zip');
            downloadResponse.body.pipe(fileStream);

            return new Promise((resolve, reject) => {
                fileStream.on('finish', () => {
                    console.log("Fine-tuned model downloaded as finetuned_model.zip");
                    resolve();
                });
                fileStream.on('error', reject);
            });
        }
    } catch (error) {
        console.error("Error:", error);
    }
}

finetuneStableDiffusion();
```

{% endtab %}
{% endtabs %}

## Fine-tuning Process Overview

The fine-tuning process on Crynux Network consists of three main steps:

1. **Task Creation**: Submit a POST request to `/v1/images/models` with your fine-tuning configuration. This returns a task ID that you'll use to track progress.
2. **Status Monitoring**: Poll the `/v1/images/models/{task_id}/status` endpoint to check if the task has completed, failed, or is still running. Fine-tuning can take anywhere from minutes to hours depending on your configuration.
3. **Result Download**: Once the task succeeds, download the fine-tuned model using the `/v1/images/models/{task_id}/result` endpoint. The result is typically a ZIP file containing your fine-tuned model weights.

## Understanding the Task Execution Flow

When you submit a fine-tuning task, here's what happens behind the scenes:

### Task Distribution

Your fine-tuning task is distributed across multiple nodes in the Crynux Network. Each node receives the same task definition and executes it independently to ensure consensus.

### Training Execution

The training process follows these steps:

1. **Data Loading**: The dataset is downloaded and prepared according to your specifications
2. **Model Loading**: The base Stable Diffusion model is loaded with LoRA adapters applied
3. **Training Loop**: The model is trained for the specified number of steps using your hyperparameters
4. **Validation**: During training, validation images are generated to monitor progress
5. **Checkpointing**: Training checkpoints are saved periodically

### Result Generation

Upon completion, the task produces:

* **LoRA Weights**: The trained LoRA adapter weights that can be applied to the base model
* **Validation Images**: Sample images generated during training to assess model quality
* **Training Logs**: Information about the training process and metrics

### Using Your Fine-tuned Model

The downloaded ZIP file contains the LoRA weights that can be loaded into compatible Stable Diffusion pipelines. You can use these weights with the same base model to generate images with your custom style or subject matter.

## Key Configuration Parameters

The fine-tuning configuration includes various parameters that control the training process. Here are the most important parameters you'll need to configure:

### Dataset Parameters

* `dataset_url`: URL to download your custom dataset (or use `dataset_name` for Hugging Face datasets)
* `validation_num_images`: Number of validation images to generate during training

### Model Parameters

* `model_name`: The base Stable Diffusion model to fine-tune (e.g., `"runwayml/stable-diffusion-v1-5"`)
* `model_variant`: Model precision variant (`"fp16"`, `"bf16"`, or `null`)

### LoRA Parameters

* `rank`: LoRA attention dimension (typically 4-64, higher values = more capacity but larger file size)
* `target_modules`: Transformer modules to apply LoRA to (common: `["to_k", "to_q", "to_v", "to_out.0"]`)
* `init_lora_weights`: LoRA weight initialization method (`"gaussian"` recommended)

### Training Parameters

* `learning_rate`: Initial learning rate (typically 1e-4 to 1e-5)
* `batch_size`: Training batch size (usually 1-4 for fine-tuning)
* `num_train_steps`: Steps per task execution
* `max_train_steps`: Total training steps across all tasks
* `lr_scheduler`: Learning rate schedule (`"cosine"`, `"linear"`, etc.)
* `lr_warmup_steps`: Warmup steps for learning rate

### Data Processing Parameters

* `center_crop`: Whether to center crop images to resolution
* `random_flip`: Whether to randomly flip images horizontally
* `mixed_precision`: Training precision (`"fp16"` or `"bf16"`)

For a comprehensive list of all supported parameters and their detailed descriptions, please refer to the fine-tuning task's document:

{% content-ref url="/pages/kF1BIZvTRByilPJEdbWH" %}
[Fine-Tuning Task](/application-development/execute-tasks/fine-tuning-task)
{% endcontent-ref %}

## Best Practices for Fine-tuning

To achieve the best results with your fine-tuning tasks, consider these recommendations:

### Dataset Quality

* **Image Quality**: Use high-quality, consistent images (512x512 or higher resolution)
* **Caption Quality**: Write descriptive, accurate captions that capture the key features
* **Dataset Size**: Aim for 10-100 images per concept for good results
* **Diversity**: Include variations in poses, lighting, and backgrounds

### Model Selection

* **Base Model**: Choose a model that matches your target style (SD 1.5 for general use, SDXL for higher quality)
* **LoRA Rank**: Start with rank 4-8 for most use cases, increase to 16-32 for complex concepts
* **Target Modules**: Use the default `["to_k", "to_q", "to_v", "to_out.0"]` for most applications

### Training Parameters

* **Learning Rate**: Start with 1e-4, reduce to 1e-5 for sensitive concepts
* **Batch Size**: Use 1-2 for most cases to avoid memory issues
* **Training Steps**: 500-2000 steps usually sufficient, monitor validation images
* **Scheduler**: Use `"cosine"` for smooth learning rate decay

### Monitoring Progress

* **Validation Images**: Check generated validation images to assess training progress
* **Task Status**: Monitor task status regularly, especially for long-running tasks
* **Error Handling**: Implement proper error handling for failed tasks

## Get the API Key to Run Tasks

The API Key in the example code is for public demonstration purposes only and has a strict rate limit, making it unsuitable for production environments. To use the Crynux Network for fine-tuning in production, choose one of the following methods:

### Method 1: Using the Official Crynux Bridge

You can request a separate API Key with a higher quota from the Crynux Discord server. Join the server and request new keys from an admin in the "applications" channel.

{% embed url="<https://discord.gg/y8YKxb7uZk>" %}

### Method 2: Hosting Your Own Crynux Bridge

You can host your own instance of the Crynux Bridge to provide private APIs for your application. This approach gives you greater control over various system aspects, including reliability and speed-related configurations.

Starting a Crynux Bridge is as straightforward as running a Docker container. An additional requirement is a wallet funded with sufficient (test) CNX to cover the tasks you run on the network. And at this moment, you can get test CNXs for free in the [Crynux Discord](https://discord.gg/y8YKxb7uZk) as well.

Crynux Bridge is fully open-sourced on [GitHub](https://github.com/crynux-network/crynux-bridge). A step-by-step guide for starting a Crynux Bridge instance is available in the following document:

{% content-ref url="/pages/kiPKEEQwV77hmCOGd58B" %}
[Crynux Bridge](/application-development/crynux-bridge)
{% endcontent-ref %}

### Method 3: Sending Tasks Directly to the Blockchain

You can bypass the Crynux Bridge entirely and interact directly with the blockchain and Crynux Relay to send fine-tuning tasks. Crynux SDKs are available in various languages and can be embedded directly into your code to run fine-tuning tasks. Please consult the Crynux SDK documentation for detailed usage instructions:

{% content-ref url="/pages/T7IKwH1gpqgUPdW6UYq0" %}
[Crynux SDK](/application-development/crynux-sdk)
{% endcontent-ref %}


# Crynux Bridge

How to Start a Crynux Bridge Locally

Crynux Bridge is middleware that connects traditional applications to the Crynux Network. It simplifies using the Crynux Network for applications by handling all complex interactions with the Crynux Network. The application only needs to interact with the Crynux Bridge by sending task parameters and waiting for the result images or texts.

More specifically, the Crynux Bridge:

1. Manages the application wallet, signs the underlying transactions and API requests.
2. Interacts with the blockchain and Relay to execute the entire task workflow.
3. Provides simpler APIs to the application to execute tasks using only the task parameters(no blockchain transactions or signatures).

Check out this simple webpage that lets users create images from text prompts. Tasks are sent to the Crynux Bridge API, and the generated image is returned:

{% embed url="<https://codepen.io/Luke-Weber/pen/ExBqGrK>" %}

And the following webpage that implements a chatbot using the OpenAI-compliant LLM API:

{% embed url="<https://codepen.io/Luke-Weber/pen/RNWaOgL>" %}

## Features

### OpenAI-Compliant APIs for LLM text generation

The Crynux Bridge provides OpenAI-compliant LLM APIs that support both chat completions and text completions. These APIs allow you to interact with various Large Language Models (LLMs) in a conversational manner.

#### Chat Completions API

The chat completions API (`/v1/llm/chat/completions`) supports the following key parameters:

* `model`: The model to use (e.g., "Qwen/Qwen2.5-7B")
* `messages`: An array of message objects with `role` and `content`
* `temperature`: Controls randomness in the output (range: 0.0 to 2.0)
* `max_tokens`: Maximum number of tokens to generate
* `top_p`: Nucleus sampling parameter (range: 0.0 to 1.0)
* `top_k`: Top-k sampling parameter (range: 1 to Infinity)
* `min_p`: Minimum probability threshold (range: 0.0 to 1.0)
* `repetition_penalty`: Penalty for repeating tokens (range: 0.0 to 2.0)
* `frequency_penalty`: Penalty for frequent tokens (range: -2.0 to 2.0)
* `presence_penalty`: Penalty for token presence (range: -2.0 to 2.0)
* `seed`: Seed for deterministic outputs
* `n`: Number of completions to generate (default: 1)
* `stream`: Whether to stream the response (default: false)
* `stop`: Array of strings that stop generation when encountered

#### Text Completions API

The text completions API (`/v1/llm/completions`) provides a simpler interface for non-chat use cases. It's ideal for tasks like text completion, summarization, or single-turn text generation. The API supports the following key parameters:

* `model`: The model to use (e.g., "Qwen/Qwen2.5-7B")
* `prompt`: The text prompt to generate a completion for
* `temperature`: Controls randomness in the output (range: 0.0 to 2.0)
* `max_tokens`: Maximum number of tokens to generate
* `top_p`: Nucleus sampling parameter (range: 0.0 to 1.0)
* `top_k`: Top-k sampling parameter (range: 1 to Infinity)
* `min_p`: Minimum probability threshold (range: 0.0 to 1.0)
* `repetition_penalty`: Penalty for repeating tokens (range: 0.0 to 2.0)
* `frequency_penalty`: Penalty for frequent tokens (range: -2.0 to 2.0)
* `presence_penalty`: Penalty for token presence (range: -2.0 to 2.0)
* `seed`: Seed for deterministic outputs
* `n`: Number of completions to generate (default: 1)
* `stream`: Whether to stream the response (default: false)
* `stop`: Array of strings that stop generation when encountered

### Image Generation APIs

The Crynux Bridge provides an OpenAI-compatible image generation API that uses Stable Diffusion models. The API (`/v1/images`) supports the following key parameters:

* `model`: The model to use (default: "crynux-network/sdxl-turbo")
* `prompt`: Text description of the desired image
* `n`: Number of images to generate (default: 1)
* `size`: Image dimensions (default: "512x512", options: "256x256", "512x512", "1024x1024")
* `response_format`: Response format (default: "b64\_json")
* `output_format`: Image format (default: "png")

The response includes:

* `created`: Timestamp of when the image was generated
* `data`: Array of generated images, each containing:
  * `b64_json`: Base64-encoded image data
  * `url`: URL to the generated image (if response\_format is "url")
  * `revised_prompt`: The prompt after any automatic modifications
* `usage`: Token usage statistics

### Multi-user Support and Role-Based Access Control

The Crynux Bridge is designed to support multiple users, enabling seamless collaboration and improved management of access to both the image generation and LLM APIs. With Role-Based Access Control (RBAC), administrators can define specific roles with varying permissions, ensuring that each user can only access those features necessary for their tasks.

## Start a Crynux Bridge Locally

### 1. Get the Docker Compose files

The Docker Compose files are located in the `build` folder of the Crynux Bridge project:

{% embed url="<https://github.com/crynux-network/crynux-bridge/tree/main/build>" %}

Download the folder to the deployment server, or clone the whole project:

```sh
$ git clone https://github.com/crynux-network/crynux-bridge.git
$ cd build
```

### 2. Application wallet configuration

The application wallet's private key will be loaded from a file in the build folder and stored as Docker secrets. For security, this file can be deleted once the container is created. Ensure to back up the private key, as it will be required again if the container needs to be recreated.

When Crynux Bridge runs tasks, task fees are deducted from the wallet's Relay Account balance instead of being deducted directly from the wallet through on-chain transactions. The wallet is still required for Bridge startup and signing operations, but the wallet itself can have zero token balance as long as its corresponding Relay Account is funded in advance.

To deposit into the Relay Account before starting production traffic:

1. Open the [Crynux Portal](https://portal.crynux.io).
2. Connect the same wallet you will use as the Bridge application wallet.
3. Go to the wallet dashboard and locate the **Relay Account** section.
4. Click **Deposit**, enter the amount, and confirm the on-chain transaction in your wallet.

Create a file named `privkey.txt` and paste the private key into the file. The private key should be a hex string prefixed with `0x`.

```sh
# Inside the build folder

$ cat "0xabcd...23cd" >> privkey.txt
```

### 3. Database configuration

Crynux Bridge relies on a database to store data. A MySQL instance is configured in the Docker Compose file by default.

If the default configuration meets your needs, no further action is required. If you need to use another database instance, remove the service section of MySQL in the `docker-compose.yml` file, and modify `config/config.yml` to use another database instance:

```yaml
# config/config.yml

db:
  driver: "mysql"
  connection: "crynux_bridge:crynuxbridgepass@(mysql:3306)/crynux_bridge?parseTime=true"
  log:
    level: "info"
    output: "/app/data/logs/crynux_bridge_db.log"
    max_file_size: 100
    max_days: 30
    max_file_num: 5
```

### 4. Start the Docker container

In the build folder, run the following command to start the containers:

```sh
# Inside the build folder

$ docker compose up -d
```

### 5. API keys and rate limits configuration

Once the Docker container is started, find the correct IP address of the Docker container. It is either `127.0.0.1`, or an IP address on your Docker network.

Open the following URL in your web browser:

{% embed url="<https://crynux-network.github.io/crynux-bridge/examples/access-key-generation.html>" %}

You should see a webpage like this:

<figure><img src="/files/W5BNe2rtSsAoPvjDT5ri" alt=""><figcaption></figcaption></figure>

Enter your local Crynux Bridge IP address and port (default is 5028), along with the private key from step 2. Choose the role and set limits as needed. Click "Create Access Token" to generate and display the token on the page.

### 6. Use the APIs

Once you have created and configured your API key, you can start using the APIs. Here are examples for both LLM and SD APIs:

#### **Use the LLM API**

{% tabs %}
{% tab title="Python" %}

```python
import requests
import json

# API configuration
API_URL = "https://bridge.crynux.io/v1/llm/chat/completions"
API_KEY = "your-api-key-here"  # Replace with your API key

# Request headers
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# Request payload
payload = {
    "model": "Qwen/Qwen2.5-7B",
    "messages": [
        {
            "role": "user",
            "content": "What is the capital of France?"
        }
    ],
    "stream": False
}

# Make the request
response = requests.post(
    API_URL,
    headers=headers,
    json=payload,
    timeout=180
)

# Print the response
print(response.json())
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
async function getChatCompletion() {
    try {
        const API_URL = "https://bridge.crynux.io/v1/llm/chat/completions";
        const API_KEY = "your-api-key-here";  // Replace with your API key

        const response = await fetch(API_URL, {
            method: "POST",
            headers: {
                "Authorization": `Bearer ${API_KEY}`,
                "Content-Type": "application/json"
            },
            body: JSON.stringify({
                model: "Qwen/Qwen2.5-7B",
                messages: [
                    {
                        role: "user",
                        content: "What is the capital of France?"
                    }
                ],
                stream: false
            })
        });

        const data = await response.json();
        console.log(data);
    } catch (error) {
        console.error("Error:", error);
    }
}

getChatCompletion();
```

{% endtab %}
{% endtabs %}

#### **Use the image generation API**

{% tabs %}
{% tab title="Python" %}

```python
import requests
import json

# API configuration
API_URL = "https://bridge.crynux.io/v1/images"
API_KEY = "your-api-key-here"  # Replace with your API key

# Request headers
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# Request payload
payload = {
    "model": "crynux-network/sdxl-turbo",
    "prompt": "a beautiful landscape",
    "n": 1,
    "size": "512x512"
}

# Make the request
response = requests.post(
    API_URL,
    headers=headers,
    json=payload,
    timeout=180
)

# Print the response
print(response.json())
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
async function generateImage() {
    try {
        const API_URL = "https://bridge.crynux.io/v1/images";
        const API_KEY = "your-api-key-here"; // Replace with your API key

        const response = await fetch(API_URL, {
            method: "POST",
            headers: {
                "Authorization": `Bearer ${API_KEY}`,
                "Content-Type": "application/json"
            },
            body: JSON.stringify({
                model: "crynux-network/sdxl-turbo",
                prompt: "A beautiful sunset over a calm ocean",
                n: 1,
                size: "512x512"
            })
        });

        const data = await response.json();
        console.log(data);
    } catch (error) {
        console.error("Error:", error);
    }
}

generateImage();
```

{% endtab %}
{% endtabs %}

## API List

The description of the APIs can be accessed as the OpenAPI Specification on the started Crynux Bridge instance. Assume the IP address of the instance is 192.168.1.2, the JSON schema of the specification can be accessed at:

```
http://192.168.1.2/openapi.json
```

And a human readable documentation can be accessed at:

```
http://192.168.1.2/static/api_docs.html
```

As an example, the URLs of the Crynux Bridge used by the showcase applications online are:

{% embed url="<https://api_ig.crynux.ai/openapi.json>" %}

{% embed url="<https://api_ig.crynux.ai/static/api_docs.html>" %}


# Application Workflow

Use the Crynux Network as the inference API

{% hint style="info" %}
The easiest method to connect an application to the Crynux Network is to deploy a Crynux Bridge, and connect the application to the bridge using API. The Crynux Bridge will take care of the application wallet, and all the interactions with the blockchain and Relay. The tutorial can be found in the following document:

[Crynux Bridge](/application-development/crynux-bridge)

Another convenient method is to use the SDK to embed the workflow directly into the application. The details of the SDKs are explained in the following document:

[Crynux SDK](/application-development/crynux-sdk)
{% endhint %}

The application can utilize the Crynux Network as an API service. It sends inference tasks to the network and receives images or texts in return. Two types of inference tasks are supported: Stable Diffusion image generation and GPT text generation.

The application interacts with two network components: the blockchain node and the Relay. To send tasks successfully, it must have a wallet with sufficient Test CNX tokens for payment. Test CNX tokens can be acquired for free on the [Discord Server of Crynux](https://discord.gg/y8YKxb7uZk).

Reference applications are provided for both image generation and text generation tasks. The source code can be found on the GitHub.

As the first step, we will provide a high-level overview of the complete workflow, outlining the main steps involved in the process.

## Overview

The application workflow is illustrated in the graph below:

```mermaid
sequenceDiagram
    Participant A as Application
    Participant B as Blockchain
    Participant R as DA/Relay

    A ->> B: Create task
    activate B
    note over A,B: Task Criteria<br/>Task Fee
    B -->> A: Transaction confirmed
    deactivate B
    activate A
    A ->> R: Send task parameters
    activate R
    note over A,R: Task Parameters
    deactivate A
    R -->> A: Task parameters saved
    deactivate R
    B ->> A: Event: TaskSuccess
    activate A
    A ->> R: Get task result
    deactivate A
    activate R
    R -->> A: Task result
    deactivate R
```

The application initiates the workflow by calling the `CreateTask` method of the smart contract. This method receives task parameters related to the task criteria, such as the task type and VRAM requirements, which the network uses to select suitable nodes.

The application transfers the task fee to the contract address by specifying it in the transaction's `value` field. Upon task completion, tokens are sent to the nodes. If the task fails, the fee is refunded to the application's wallet.

After the transaction is confirmed on-chain, the application should then send the task parameters to the Relay.

> Selected nodes will retrieve task parameters from the Relay and then execute the tasks locally.
>
> When images or texts are generated, nodes will create proofs and send them to the blockchain. The blockchain will verify the correctness of these proofs and transfer tokens to the nodes upon successful verification.
>
> The nodes will upload the result images/texts to the Relay, which will compare the results with the on-chain proofs to verify their accuracy.

After sending the task parameters to the Relay, the application should wait for the `TaskSuccess` event from the blockchain. Once the event is received, the application can retrieve the images or texts from the Relay, marking the completion of the task workflow.

The results have already been verified by the Relay, so no further verification by the application is necessary.

For a detailed workflow involving all network participants, please refer to the task lifecycle document:

{% content-ref url="/pages/iPukPMh2AXLB0TPkE1Wt" %}
[Task Lifecycle](/system-design/task-lifecycle)
{% endcontent-ref %}

## The Reference Applications

The workflow has been fully implemented in the showcase applications: the Image Generator and the AI Chatbot. Which can be accessed at:

The Image Generator: <https://ig.crynux.io>

The AI Chatbot: <https://chat.crynux.io>

Both applications utilize the Crynux Bridge as the backend. The Crynux Bridge includes a built-in wallet to cover task fees, eliminating the need for applications to manage their own wallets. Additionally, it isolates the blockchain and Relay from the applications. This allows applications to simply submit task parameters via API and await the result without further action.

The Crynux Bridge can be used by all the applications. The source code of the Crynux Bridge can be found at:

{% embed url="<https://github.com/crynux-network/crynux-bridge>" %}

The source code of the web UI of the Image Generator:

{% embed url="<https://github.com/crynux-network/ig-web>" %}

The source code of the web UI of the AI Chatbot

{% embed url="<https://github.com/crynux-network/chat-web>" %}

## Application Workflow Step by Step

### 1. Prepare the application wallet

An Ethereum compatible wallet must be generated. Which will be used by the application to invoke the smart contracts on-chain.

Ensure the wallet has sufficient CNX tokens to cover both task and transaction fees. The application should continuously monitor the wallet balance and alert admins to replenish tokens before it drops below the required amount for upcoming tasks.

{% hint style="info" %}
In the DApp, an application wallet is not required. The DApp will create the transaction and send it to Metamask for the user to sign directly in the browser.
{% endhint %}

### 2. Create the Task on the Blockchain

#### Construct the task parameters

The task parameters are organized as a JSON string. An example of the parameters of an image generation task is given below:

```json
{
    "version": "2.0.0",
    "base_model": {
        "name": "stabilityai/sdxl-turbo"
    },
    "prompt": "best quality, ultra high res, photorealistic++++, 1girl, desert, full shot, dark stillsuit, "
              "stillsuit mask up, gloves, solo, highly detailed eyes,"
              "hyper-detailed, high quality visuals, dim Lighting, ultra-realistic, sharply focused, octane render,"
              "8k UHD",
    "negative_prompt": "no moon++, buried in sand, bare hands, figerless gloves, "
                       "blue stillsuit, barefoot, weapon, vegetation, clouds, glowing eyes++, helmet, "
                       "bare handed, no gloves, double mask, simplified, abstract, unrealistic, impressionistic, "
                       "low resolution,",
    "task_config": {
        "num_images": 9,
        "steps": 1,
        "cfg": 0
    },
    "lora": {
        "model": "https://civitai.com/api/download/models/178048"
    },
    "controlnet": {
        "model": "diffusers/controlnet-canny-sdxl-1.0",
        "image_dataurl": "data:image/png;base64,12FE1373...",
        "preprocess": {
            "method": "canny"
        },
        "weight": 70
    },
    "scheduler": {
        "method": "EulerAncestralDiscreteScheduler",
        "args": {
            "timestep_spacing": "trailing"
        }
    }
}
```

The task definition above follows the schema given in the [Stable Diffusion Task Framework](https://github.com/crynux-network/stable-diffusion-task). A wide range of common configurations are supported. The framework also provides a JSON schema to validate task parameters. More information about the framework can be found in the document below:

{% content-ref url="/pages/1DaHVxBpm1NeVj27j6if" %}
[Text-to-Image Task](/application-development/execute-tasks/text-to-image-task)
{% endcontent-ref %}

A similar framework for the GPT text generation task is also provided in the following document:

{% content-ref url="/pages/wbQGyrKGcOdpNuWNNgqU" %}
[Text-to-Text Task](/application-development/execute-tasks/text-to-text-task)
{% endcontent-ref %}

{% hint style="info" %}
The application must validate task parameters against the schema before sending them to the network, especially when the task parameters are generated by the user on the frontend.
{% endhint %}

#### Send the create task transaction to the blockchain

Once the JSON string for the task parameters is ready, the application must create and send the `CreateTask` transaction to the blockchain.

`CreateTask` method of the [Task Contract](https://github.com/crynux-network/crynux-contracts/blob/75a2f7014d9d797df9721be17161ec32c745b9dd/contracts/Task.sol#L75) has five arguments:

```solidity
function createTask(
    uint taskType,
    bytes32 taskHash,
    bytes32 dataHash,
    uint vramLimit,
    uint cap
)
```

* `taskType` is an integer that identifies the task type: 0 for SD task and 1 for GPT task.
* `taskHash` is the keccak256 hash of the JSON string of the task arguments.
* `dataHash` is reserved for the future features and is not used right now. The application could just pass 32 zero bytes to it.
* `vramLimit` indicates the minimum VRAM required to execute the task. The Crynux Network will select the capable nodes based on this value.
* `cap` indicates the task size. It is used to estimate the task execution time by the Crynux Network. It should be set to the number of images in the SD task, and 0 in a GPT task.

In addition to the arguments listed above, the task fee should be set in the `value` field of the transaction. The application is free to choose any task fee value, a higher task fee will result in a faster task execution, while lower task fee will result in longer waiting time.

The source code that implements the invocation of the `CreateTask` method in the Crynux Bridge [can be found here](https://github.com/crynux-network/crynux-bridge/blob/652ea694980da774a283782886bedaa362a53a50/blockchain/task.go#L32).

#### Wait for the transaction confirmation

After sending the transaction, the application should wait for confirmation before proceeding. The transaction might be reverted by the blockchain for various reasons. All possible reasons for a transaction being reverted can be found [in the source code](https://github.com/crynux-network/crynux-contracts/blob/43f98cc0d0b6726c54dc93103739414c6313a6c9/contracts/Task.sol#L59C21-L59C21).

If the transaction is reverted, no event will be emitted. Therefore, the creation result can only be queried using the transaction hash or the receipt provided by the blockchain when sending the `CreateTask` transaction.

### 3. Upload the Task Parameters to the Relay

Once the transaction is confirmed, the next step is to upload the task parameters JSON string to the Relay. Use the following API endpoint:

{% openapi src="<https://dy.relay.crynux.io/openapi.json>" path="/v1/inference\_tasks" method="post" %}
<https://dy.relay.crynux.io/openapi.json>
{% endopenapi %}

The complete API documentation can be found in the [OpenAPI Specifications](https://dy.relay.crynux.io/openapi.json) of the Relay server.

To upload, simply invoke the API to the Relay server. Ensure the request is signed by the application wallet before sending.

> The Relay tracks the blockchain for task creations, recording the task ID and the creator's address (application wallet) upon creation. To upload task arguments, the request must originate from the same task creator's wallet with a verified signature.

The signature is generated using ECDSA with the same curve as Ethereum, on the Keccak256 hash of a string. This string is created by including all query and body parameters (except `timestamp` and `signature`) from the request in a JSON string with keys sorted alphabetically and concatenated with the current Unix timestamp.

The reference implementation of the signing method in Crynux Bridge [can be found here](https://github.com/crynux-network/crynux-bridge/blob/main/relay/sign_data.go). The code to upload the task parameters to the Relay can also be found [in the source code](https://github.com/crynux-network/crynux-bridge/blob/652ea694980da774a283782886bedaa362a53a50/relay/inference_task.go#L41).

### 4. Wait for the Task to Finish

When the task finishes, either the `TaskSuccess` or `TaskAborted` event will be emitted. If the `TaskSuccess` event is emitted, the application can retrieve the result from the Relay. If the `TaskAborted` event is emitted, indicating a failure, the application can retry by creating a new task.

Several reasons can cause task execution failure. Task arguments might not pass node schema validation, some nodes might not run the consensus protocol correctly, or a task might take too long on a single node. The exact reason is included as an argument in the emitted event.

If a task is aborted, CNX tokens may either be returned to the application wallet or still paid to the nodes. This depends on who is at fault for the task's failure.

There are two ways the application could monitor the blockchain for relevant events.

#### Tracking new blocks and filtering the target events

The first method involves continuously tracking new blocks and filtering them for these two types of events.

To ensure reliable block tracking, the application must handle potential crashes caused by unhandled bugs. Additionally, extended downtime can result in delays when catching up with new blocks.

#### Query for the task status periodically

Another approach is to extract the task ID from the creation transaction, store it, and periodically check the blockchain for the latest task status. This method eliminates the need to track the block, but it is less efficient due to a high volume of unnecessary queries.

The Crynux Bridge uses the first method, the source code of the block synchronization [can be found here](https://github.com/crynux-network/crynux-bridge/blob/main/tasks/sync_block.go).

### 5. Fetch the result from the Relay

The final step is to retrieve the actual images or texts from the Relay. This can be accomplished by calling the Relay's API as follows:

#### Get images

The URL could be treated like an image downloading link as it returns the binary stream of the image content directly. The signature and timestamp is still required.

{% openapi src="<https://dy.relay.crynux.io/openapi.json>" path="/v1/inference\_tasks/{task\_id}/results/{image\_num}" method="get" %}
<https://dy.relay.crynux.io/openapi.json>
{% endopenapi %}

#### Get texts

The API endpoint to get text results from the Relay is the same as the endpoint above, except that the `image_num` should be set to zero.

{% hint style="info" %}
When the application accesses the above URL after the `TaskSuccess` event is received, it could keep getting `404 not found` for a short while before it gets the correct results. The reason is that the node will start to upload images/texts to the Relay only after the `TaskSuccess` event is received. So before the uploading is done, the application can not find the results on the Relay. Several times of retrying is required at this place.
{% endhint %}

When the application accesses the URL after receiving the `TaskSuccess` event, it might encounter `404 not found` errors temporarily. This occurs because the node initiates the upload of images/texts to the Relay only after the `TaskSuccess` event is triggered. Therefore, the results won't be available on the Relay until the upload is complete. Retrying the request several times may be necessary.

The source code where the Crynux Bridge downloads the images is [located here](https://github.com/crynux-network/crynux-bridge/blob/aba6390424904c14b8f8676d5559c8ec9f6da503/relay/inference_task.go#L93).


# Execute Tasks


# Text-to-Image Task

How to define a text-to-image task

The Stable Diffusion Task Framework has two components:

1. A generalized schema to define a Stable Diffusion task.
2. An execution engine that runs the task defined in the above schema.

The task definition is represented in the key-value pairs that can be transformed into, among many other formats, a JSON string, which can be validated using a JSON schema. And the validation tools exist for most of the popular programming languages.

The execution engine is integrated into the node of the Hydrogen Network, and the JSON string format of the task definition is used to send tasks in the Hydrogen Network.

The following is an intuitive look at a task definition:

```json
{
    "version": "2.0.0",
    "base_model": {
        "name": "stabilityai/sdxl-turbo"
    },
    "prompt": "best quality, ultra high res, photorealistic++++, 1girl, desert, full shot, dark stillsuit, "
              "stillsuit mask up, gloves, solo, highly detailed eyes,"
              "hyper-detailed, high quality visuals, dim Lighting, ultra-realistic, sharply focused, octane render,"
              "8k UHD",
    "negative_prompt": "no moon++, buried in sand, bare hands, figerless gloves, "
                       "blue stillsuit, barefoot, weapon, vegetation, clouds, glowing eyes++, helmet, "
                       "bare handed, no gloves, double mask, simplified, abstract, unrealistic, impressionistic, "
                       "low resolution,",
    "task_config": {
        "num_images": 9,
        "steps": 1,
        "cfg": 0
    },
    "lora": {
        "model": "https://civitai.com/api/download/models/178048"
    },
    "controlnet": {
        "model": "diffusers/controlnet-canny-sdxl-1.0",
        "image_dataurl": "data:image/png;base64,12FE1373...",
        "preprocess": {
            "method": "canny"
        },
        "weight": 70
    },
    "scheduler": {
        "method": "EulerAncestralDiscreteScheduler",
        "args": {
            "timestep_spacing": "trailing"
        }
    }
}
```

More examples of the different Stable Diffusion tasks can be found [in the GitHub repository](https://github.com/crynux-network/stable-diffusion-task/tree/main/examples).

## Acceleration of the Image Generation

### SDXL Turbo

SDXL Turbo is an adversarial time-distilled [Stable Diffusion XL](https://huggingface.co/papers/2307.01952) (SDXL) model capable of running inference in as little as 1 step. To use SDXL Turbo in your task:

#### 1. Use the SDXL Turbo model as the base model:

```
"base_model": {
    "name": "crynux-network/sdxl-turbo"
},
```

#### 2. Set the `timestep_spacing` scheduler argument:

```
"scheduler": {
    "method": "EulerAncestralDiscreteScheduler",
    "args": {
        "timestep_spacing": "trailing"
    }
}
```

#### 3. Set `cfg` to zero, and set steps to 1-4:

```
"task_config": {
    "steps": 1,
    "cfg": 0
}
```

### Latent Consistency Models (LCM)

{% hint style="danger" %}
Negative prompts won't work with LCM methods.
{% endhint %}

[Latent Consistency Models (LCMs)](https://hf.co/papers/2310.04378) enable fast high-quality image generation by directly predicting the reverse diffusion process in the latent rather than pixel space. In other words, LCMs try to predict the noiseless image from the noisy image in contrast to typical diffusion models that iteratively remove noise from the noisy image. By avoiding the iterative sampling process, LCMs are able to generate high-quality images in 2-4 steps instead of 20-30 steps.

There are two ways LCM could be used in a Stable Diffusion task: LCM and LCM-LoRA:

{% tabs %}
{% tab title="LCM" %}
**1.Load the LCM model corresponding to your base model using the `unet` argument:**

```
"base_model": {
    "name": "stabilityai/stable-diffusion-xl-base-1.0"
},
"unet": "latent-consistency/lcm-sdxl",
```

**2.Use the `LCMScheduler`:**

```
"scheduler": {
    "method": "LCMScheduler"
}
```

**3.Set `cfg` to 3-13, and set `steps` to 4:**

```
"task_config": {
    "steps": 4,
    "cfg": 5
},
```

{% endtab %}

{% tab title="LCM-LoRA" %}
**1. Load the LCM-LoRA model corresponding to your base model using `lora` argument:**

```
"base_model": {
    "name": "runwayml/stable-diffusion-v1-5"
},
"lora": {
    "model": "latent-consistency/lcm-lora-sdv1-5"
},
```

**2. Use the `LCMScheduler`:**

```
"scheduler": {
    "method": "LCMScheduler"
}
```

**2. Set the cfg to 1-2, and steps to 4:**

```
"task_config": {
    "steps": 4,
    "cfg": 1
},
```

{% endtab %}
{% endtabs %}

## Base Model

The base model could be the original Stable Diffusion models, such as the Stable Diffusion 1.5 and the Stable Diffusion XL, or a checkpoint that is fine-tuned based on the original Stable Diffusion models.

The model can be specified in two ways: a Huggingface model ID, or a file download URL.

#### Huggingface Model ID

The Huggingface model ID for the original Stable Diffusion models are listed below:

* **Stable Diffusion 1.5**

```json
{
  "base_model": "runwayml/stable-diffusion-v1-5"
}
```

* **Stable Diffusion 2.1**

```json
{
  "base_model": "stabilityai/stable-diffusion-2-1"
}
```

* **Stable Diffusion XL**

```json
{
  "base_model": "stabilityai/stable-diffusion-xl-base-1.0"
}
```

* **Custom Fine-tuned Checkpoints**

Other custom fine-tuned checkpoints based on the original SD models can also be used, for example, the [ChilloutMix](https://huggingface.co/emilianJR/chilloutmix_NiPrunedFp32Fix) model on the Huggingface:

```json
{
  "base_model": "emilianJR/chilloutmix_NiPrunedFp32Fix"
}
```

#### File Download URL

A URL can also be used as the base model. The execution engine will download the file before executing the task.

For example, if we want to use an SDXL fined-tuned checkpoint on Civitai. The webpage of the model is <https://civitai.com/models/169868/thinkdiffusionxl> and the download link of the model file can be copied from the download button on the webpage:

<https://civitai.com/api/download/models/190908>

We could use the model in the task as following:

```json
{
  "base_model": "https://civitai.com/api/download/models/190908"
}
```

{% hint style="info" %}
Only `safetensors` format is supported in the download URL.

The execution engine assumes the download URL to be a binary stream of a model file in the `safetensors` format. If other formats are used, or the content of the link is not a model file at all, the execution engine will throw an exception during the execution.
{% endhint %}

## LoRA Model

LoRA models can be specified using the same format as the base model: the Huggingface model ID or the file download URL. The weight of the LoRA model can also be set in the arguments:

```json
{
   "lora": {
     "model": "https://civitai.com/api/download/models/31284",
     "weight": 80
   }
}
```

The weight should be an integer between 1 and 100.

If the LoRA model given is not compatible with the base model, for example, a LoRA model fine-tuned on the Stable Diffusion 1.5 is used, but the base model is set to be Stable Diffusion XL, the execution engine will also throw an exception.

## Controlnet

The Controlnet section has two parts: the Controlnet model, and the preprocess method.

The Controlnet model also supports the Huggingface ID and the download URL, which is exactly the same as the LoRA model.

The control image should be a PNG image encoded in the DataURL format. The DataURL string should be filled in the `image_dataurl` field.

```json
{
   "controlnet": {
      "model": "lllyasviel/control_v11p_sd15_openpose",
      "weight": 90,
      "image_dataurl": "base64,image/png:..."
   }
}
```

#### Image Preprocessing

The image preprocessing function is implemented using the [`controlnet_aux`](https://github.com/patrickvonplaten/controlnet_aux) project. All the preprocessing methods and models in this project can be used:

```json
{
   "controlnet": {
      "model": "lllyasviel/sd-controlnet-canny",
      "weight": 90,
      "image_dataurl": "base64,image/png:...",
      "preprocess": {
         "method": "canny",
         "args": {
            "high_threshold": 200,
            "low_threshold": 100
         }
      }
   }
}
```

Here is a list of all the available preprocess methods and their arguments:

<table data-full-width="false"><thead><tr><th width="291">Method</th><th>Arguments</th></tr></thead><tbody><tr><td>canny</td><td>high_threshold, low_threshold</td></tr><tr><td>scribble_hed</td><td></td></tr><tr><td>scribble_hedsafe</td><td></td></tr><tr><td>softedge_hed</td><td></td></tr><tr><td>softedge_hedsafe</td><td></td></tr><tr><td>depth_midas</td><td></td></tr><tr><td>mlsd</td><td>thr_v, thr_d</td></tr><tr><td>openpose</td><td></td></tr><tr><td>openpose_face</td><td></td></tr><tr><td>openpose_faceonly</td><td></td></tr><tr><td>openpose_full</td><td></td></tr><tr><td>openpose_hand</td><td></td></tr><tr><td>dwpose</td><td></td></tr><tr><td>scribble_pidinet</td><td>apply_filter</td></tr><tr><td>softedge_pidinet</td><td>apply_filter</td></tr><tr><td>scribble_pidisafe</td><td>apply_filter</td></tr><tr><td>softedge_pidisafe</td><td>apply_filter</td></tr><tr><td>normal_bae</td><td></td></tr><tr><td>lineart_coarse</td><td></td></tr><tr><td>lineart_realistic</td><td></td></tr><tr><td>lineart_anime</td><td></td></tr><tr><td>depth_zoe</td><td>gamma_corrected</td></tr><tr><td>depth_leres</td><td>thr_a, thr_b</td></tr><tr><td>depth_leres++</td><td>thr_a, thr_b</td></tr><tr><td>shuffle</td><td>h, w, f</td></tr><tr><td>mediapipe_face</td><td>max_faces, min_confidence</td></tr></tbody></table>

If preprocessing is not needed, just set the value of the `controlnet` section to be null, or just delete the section from the JSON.

## Prompt

Unlike the basic SD models, the length of the prompt is not limited in this framework. The prompt and the negative prompt are specified separately:

```json
{
   "prompt": "a realistic portrait photo of a beautiful girl, blonde hair+++, smiling, facing the viewer",
   "negative_prompt": "low resolution++, bad hands"
}
```

#### Prompt Weighting

Prompt weighting is supported using the [Compel](https://github.com/damian0815/compel) library. The basic idea is to put more plus signs (`+`) to give the word more weights. More complex usages can be found in the documentation of the Compel library.

## Textual Inversion

Textual Inversion models are also supported:

```json
{
  "textual_inversion": "sd-concepts-library/cat-toy"
}
```

## VAE

The VAE model used in the Stable Diffusion pipeline can also be replaced with another one, either from the Huggingface ID, or a file download URL:

```json
{
  "vae": "stabilityai/sd-vae-ft-mse"
}
```

## SDXL Refiner

If the Stable Diffusion XL is selected as the base model in the task, the SDXL Refiner could also be used to further refine the image, which is by design of the SDXL:

```json
{
    "refiner": {
       "model": "stabilityai/stable-diffusion-xl-refiner-1.0",
       "denoising_cutoff": 80
    }
}
```

The `denoising_cutoff` is used to stop the denoising process earlier in the pipeline, when the noise level reaches the cutoff value, and leave the rest to the refiner model, which is called the [ensemble of expert denoisers](https://research.nvidia.com/labs/dir/eDiff-I/).

{% hint style="info" %}
If the Controlnet is used with the Stable Diffusion XL base model, the `denoising_cutoff` argument is not supported due to the current limitations in the [diffusers library](https://huggingface.co/docs/diffusers/index). If refiner is configured, it will be executed after the base model generation is completed, the cutoff value is ignored.
{% endhint %}

## Task Config

There are also some config options that can be tuned:

```json
{
   "task_config": {
     "image_width": 512,       // The width of the generated image
     "image_height": 512,      // The height of the generated image
     "steps": 30,              // Step to run
     "seed": 34736484,         // The seed used to initialize the random processes
     "num_images": 6,          // The number of images to generate in a single task
     "safety_checker": true,   // Filter the unsafe images
     "cfg": 5                  // Classifier-Free Guidance, how close the images should be to the prompt given
   }
}
```

{% hint style="info" %}
Hydrogen Network requires a deterministic image generation process, which means the images generated on the different nodes, given the same task definition, should be as close as possible. This is a requirement for the consensus protocol to work. The seed is left as a required argument in the task definition so that all the nodes could use the same seed to initialize their random number generators, which will hopefully produce the same random numbers across all the nodes.

Beside the seed, the Stable Diffusion Task Framework has been implemented to maximize the reproducibility, for all the components used, across the whole image generation process.
{% endhint %}


# Text-to-Text Task

How to define a text-to-text task

The GPT Task framework has two components:

1. A generalized schema to define a llm text generation task.
2. An execution engine that runs the task defined in the above schema.

The task definition is represented in the key-value pairs that can be transformed into, among many other formats, a JSON string, which can be validated using a JSON schema. And the validation tools exist for most of the popular programming languages.

The execution engine is integrated into the node of the Helium Network, and the JSON string format of the task definition is used to send tasks in the Helium Network.

## GPT Task definition

The following is an intuitive look at a task definition:

```json
{
    "model": "gpt2",
	"messages": [
		{
			"role": "user",
			"content": "I want to create a chat bot. Any suggestions?"
		}
	],
	"generation_config": {
		"max_new_tokens": 30,
		"do_sample": true,
		"num_beams": 1,
		"temperature": 1.0,
		"typical_p": 1.0,
		"top_k": 20,
		"top_p": 1.0,
		"repetition_penalty": 1.0,
		"num_return_sequences": 1
	},
	"seed": 42,
	"dtype": "auto",
	"quantize_bits": 4
}
```

More examples of the different GPT tasks can be found [in the GitHub repository](https://github.com/crynux-network/gpt-task/tree/main/examples).

### Model

The base model could be any large language model suitable for text generation task.

The model should be a Huggingface model ID.

All huggingface models list in the [huggingface models page](https://huggingface.co/models?pipeline_tag=text-generation\&sort=trending) can be used for base model.

For example:

```json
{
    "model": "mistralai/Mistral-7B-v0.1"
}
```

### Messages

Messages is a list of message objects comprising the conversation so far.

For example:

```json
{
	"messages": [
		{
			"role": "user",
			"content": "I want to create a chat bot. Any suggestions?"
		}
	]
}
```

#### Message Object

Message object has two fields: `role` and `content`.

The field `role` represents the role of message author, can be `user`, `assistant` and `system`.

The field `content` is the message content.

During execution, the messages will be formatted to a plain string using the model's chat template, and then be send to the model as input prompt. Accroding to the different message role, different tags defined by the model will be added around each message. However, some models have no chat template, in this situation all the message contents will be simply joined to a single string.

### Generation Config

Generation config is a set of parameters to control the text generation behavior of the model.

For example:

```json
{
   "generation_config": {
		"max_new_tokens": 30,
		"do_sample": true,
		"num_beams": 1,
		"temperature": 1.0,
		"typical_p": 1.0,
		"top_k": 20,
		"top_p": 1.0,
		"repetition_penalty": 1.0,
		"num_return_sequences": 1
	},
}
```

The meaning of each parameters in generation config can be found in the [huggingface generation config](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig).

#### max\_new\_tokens

The maximum numbers of tokens to generate, ignoring the number of tokens in the input prompt.

#### do\_sample

Whether or not to use sampling ; use greedy decoding otherwise.

#### num\_beams

Number of beams for beam search. 1 means no beam search.

#### temperature

The value used to modulate the next token probabilities. The higher the temperature, the flattering the next token probabilities. When the temperature equals 0, the sampling will be downgraded to greedy decoding.

#### typical\_p

Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to typical\_p or higher are kept for generation. See [this paper](https://arxiv.org/pdf/2202.00666.pdf) for more details.

#### top\_k

The number of highest probability vocabulary tokens to keep for top-k-filtering.

#### top\_p

If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top\_p or higher are kept for generation.

#### repetition\_penalty

The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.

#### num\_return\_sequences

The number of independently computed returned sequences for each element in the batch.

### Seed

The seed used to initialize the random processes.

{% hint style="info" %}
Helium Network requires a deterministic algorithm for text generation, which means the text generated on the different nodes of the same deivces, given the same task definition, should be the same. This is a requirement for the consensus protocol to work. The seed is left as a required argument in the task definition so that all the nodes could use the same seed to initialize their random number generators, which will hopefully produce the same random numbers across all the nodes.

Beside the seed, the GPT Task Framework has been implemented to maximize the reproducibility.
{% endhint %}

### Dtype

Optional. Control the data precision for the model. Can be `float16`, `bfloat16`, `float32` or `auto`. When `dtype=auto`, the parameter `dtype` will be determined by the model's config file.

### Quantize\_bits

Optional. Control the model quantization type. Can be `4` or `8`. `4` means the INT4 quantization, `8` means the INT8 quantization.

## GPT Task Response

The following is an intuitive look at a task response:

```json
{
    "model": "gpt2",
    "choices": [
        {
            "finish_reason": "length",
            "message": {
                "role": "assistant",
                "content": "\n\nI have a chat bot, called \"Eleanor\" which was developed by my team on Skype. "
                "The only thing I will say is this",
            },
            "index": 0,
        }
    ],
    "usage": {"prompt_tokens": 11, "completion_tokens": 30, "total_tokens": 41},
}
```

## Model

The model used for text generation.

## Choices

A list of choice object. The count of choices equals the the parameter `num_return_sequences` in `generation_config` of task definition.

### Choice Object

A choice object has three fields, `finish_reason`, `message` and `index`.

`finish_reason` represents the finish reason of the generated message, can be `stop` or `length`. When finish reason is `stop`, means the generated text ends with an eos token and stops naturally. When finish reason is `length`, means the generated text is truncated by the output token length limit, which defines by the `max_new_tokens` parameter in `generation_config` of task definition.

`message` is a message object which is the same with message object used in task definition. The `role` of response message will always be `assistant`.

`index` is the index of the choice object in all choices, begins from 0.

## Usage

Usage represents the token used of this text generation task. It has three fields, `prompt_tokens`, `completion_tokens` and `total_tokens`.

`prompt_tokens` means the input prompt tokens count. `completion_tokens` means the sum of all choices content tokens count. `total_tokens` is the sum of `prompt_tokens` and `completion_tokens`.


# Text-to-Music Task

How to define a text-to-music task

The Audio Task framework has two components:

1. A generalized schema to define a text-to-audio generation task.
2. An execution engine that runs the task defined in the above schema.

The task definition is represented in the key-value pairs that can be transformed into, among many other formats, a JSON string, which can be validated using a JSON schema. And the validation tools exist for most of the popular programming languages.

The execution engine is integrated into the node of the Helium Network, and the JSON string format of the task definition is used to send tasks in the Helium Network.

## Audio Task definition

The following is an intuitive look at a task definition:

```json
{
    "model": "facebook/musicgen-small",
    "prompt": "80s pop track with bassy drums and synth",
    "generation_config": {
        "max_new_tokens": 1500,
        "do_sample": true,
        "top_k": 250,
        "top_p": 0.0,
        "temperature": 1.0,
        "guidance_scale": 3.0
    },
    "seed": 42,
    "dtype": "auto",
    "quantize_bits": 8
}
```

More examples of the different audio tasks can be found [in the GitHub repository](https://github.com/crynux-network/audio-task/tree/main/examples).

### Model

The base model could be any model suitable for the [transformers.TextToAudioPipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TextToAudioPipeline).

The model should be a Huggingface model ID.

You can find the available huggingface models list in the [huggingface models page](https://huggingface.co/models?pipeline_tag=text-to-audio\&sort=trending).

For example:

```json
{
    "model": "facebook/musicgen-small"
}
```

### Prompt

Prompt is a string used to control the audio generation.

For example:

```json
{
	"prompt": "80s pop track with bassy drums and synth"
}
```

### Generation Config

Generation config is a set of parameters to control the text generation behavior of the model.

For example:

```json
{
    "generation_config": {
        "max_new_tokens": 1500,
        "do_sample": true,
        "top_k": 250,
        "top_p": 0.0,
        "temperature": 1.0,
        "guidance_scale": 3.0
    }
}
```

The meaning of each parameters in generation config can be found in the [huggingface generation config](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig).

#### max\_new\_tokens

The maximum numbers of tokens to generate. This parameter controls the max time of generated audio. The parameter `max_new_tokens` has a corresponding relationship with the max time, and this relationship is determined by the architechture of model's audio decoder.

#### do\_sample

Whether or not to use sampling ; use greedy decoding otherwise.

#### temperature

The value used to modulate the next token probabilities. The higher the temperature, the flattering the next token probabilities. When the temperature equals 0, the sampling will be downgraded to greedy decoding.

#### top\_k

The number of highest probability vocabulary tokens to keep for top-k-filtering.

#### top\_p

If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top\_p or higher are kept for generation.

#### num\_return\_sequences

The number of independently computed returned sequences for each element in the batch.

### Seed

The seed used to initialize the random processes.

{% hint style="info" %}
Helium Network requires a deterministic algorithm for text generation, which means the text generated on the different nodes of the same deivces, given the same task definition, should be the same. This is a requirement for the consensus protocol to work. The seed is left as a required argument in the task definition so that all the nodes could use the same seed to initialize their random number generators, which will hopefully produce the same random numbers across all the nodes.

Beside the seed, the GPT Task Framework has been implemented to maximize the reproducibility.
{% endhint %}

### Dtype

Optional. Control the data precision for the model. Can be `float16`, `bfloat16`, `float32` or `auto`. When `dtype=auto`, the parameter `dtype` will be determined by the model's config file.

### Quantize\_bits

Optional. Control the model quantization type. Can be `4` or `8`. `4` means the INT4 quantization, `8` means the INT8 quantization.

## Audio Task Response

The response of audio task is a tuple of generated audio waveform and its sampling rate.

The audio waveform is a `np.ndarray` of shape `(audio_length, channels)`. The sampling rate is an integer.

You can use the following code to write the generated audio waveform to file.

```python
import scipy
from audio_task.inference import run_task

# audio is the generated audio waveform, sr is the sampling rate
audio, sr = run_task(
    model="facebook/musicgen-small",
    prompt="80s pop track with bassy drums and synth",
)
scipy.io.wavfile.write("example.wav", rate=sr, data=audio)
```


# Text-to-Video Task

How to define a text-to-video task


# Fine-Tuning Task

The fine tuning task aims to fine tune a lora model for a pretrained stable diffusion model. It has two components:

1. A generalized schema to define a Fine tuning task.
2. An execution engine that runs the task defined in the above schema.

The task definition is represented in the key-value pairs that can be transformed into, among many other formats, a JSON string, which can be validated using a JSON schema. And the validation tools exist for most of the popular programming languages.

The execution engine is integrated into the node of the Hydrogen Network, and the JSON string format of the task definition is used to send tasks in the Hydrogen Network.

## Fine tuning Task definition

The following is an intuitive look at a task definition:

```json
{
    "model": {
        "name": "runwayml/stable-diffusion-v1-5",
        "revision": "main",
    },
    "dataset": {
        "name": "lambdalabs/naruto-blip-captions",
        "image_column": "image",
        "caption_column": "text",
    },
    "validation": {
        "num_images": 4,
    },
    "train_args": {
        "learning_rate": 1e-4,
        "batch_size": 1,
        "gradient_accumulation_steps": 4,
        "num_train_steps": 100,
        "max_train_steps": 15000,
        "scale_lr": true,
        "resolution": 512,
        "noise_offset": 0,
        "lr_scheduler": {
            "lr_scheduler": "cosine",
            "lr_warmup_steps": 500,
        },
        "adam_args": {
            "beta1": 0.9,
            "beta2": 0.999,
            "weight_decay": 0.01,
            "epsilon": 1e-8
        }
    },
    "lora": {
        "rank": 4,
        "init_lora_weights": "gaussian",
        "target_modules": ["to_k", "to_q", "to_v", "to_out.0"]
    },
    "transforms": {
        "center_crop": true,
        "random_flip": true,
    },
    "dataloader_num_workers": 2,
    "mixed_precision": "fp16",
    "seed": 1337,
    "checkpoint": null,
    "version": "2.1.0"
}
```

Full example of the fine tuning task can be found [in the GitHub repository](https://github.com/crynux-network/stable-diffusion-task/tree/main/examples/finetune_lora.py).

### model

```json
{
    "model": {
        "name": "runwayml/stable-diffusion-v1-5",
        "variant": "fp16",
        "revision": "main",
    }
},
```

Model defines the pretrained base model for fine tuning.

The argument `name` defines the name of pretrained base model.

The mode name could be the original Stable Diffusion models, such as the Stable Diffusion 1.5 and the Stable Diffusion XL, or a checkpoint that is fine-tuned based on the original Stable Diffusion models.

The model name should be a Huggingface model ID.

The argument `variant` means the model dtype variant, can be null (no variant), fp16 (float16), bf16 (bfloat16). Default is `null`.

The argument `revision` means the model revision, can be main or a commit hash of the model repo. Default is `main`.

### dataset

```json
{
    "dataset": {
        "name": "lambdalabs/naruto-blip-captions",
        "url": "",
        "config_name": null,
        "image_column": "image",
        "caption_column": "text",
    }
}
```

Dataset defines the dataset to train on.

The `name` argument specifies the dataset to train on. You can provide either a Hugging Face dataset ID or a local file path to a dataset on your filesystem.

The `url` argument specifies the URL of a dataset file to download. The file can be compressed (in formats like .zip, .tar, .tar.gz, etc.) and will be automatically extracted. The downloaded file will then be loaded using the Hugging Face dataset library.

If both `name` and `url` are provided, only the `name` argument will be used. You must specify either `name` or `url` (but not necessarily both).

The argument `config_name` defines the config file name of the dataset, leave as null if there is only one config. Default is `null`.

The argument `image_column` defines the column of the dataset containing an image. Default is `"image"`.

The argument `caption_column` defines the column of the dataset containing a caption. Default is `"text"`.

### validation

```json
{
    "validation": {
        "prompt": null,
        "num_images": 4
    }
}
```

Validation defines the prompt for the validation inference.

The argument `prompt` defines the prompt for the validation inference. It should be a string or null. When `prompt` is null, we will random select `num_images` prompt from the dataset for inference.

### train\_args

```json
"train_args": {
    "learning_rate": 1e-4,
    "batch_size": 1,
    "gradient_accumulation_steps": 4,
    "num_train_steps": 100,
    "max_train_steps": 15000,
    "scale_lr": true,
    "resolution": 512,
    "noise_offset": 0,
    "lr_scheduler": {
        "lr_scheduler": "cosine",
        "lr_warmup_steps": 500,
    },
    "adam_args": {
        "beta1": 0.9,
        "beta2": 0.999,
        "weight_decay": 0.01,
        "epsilon": 1e-8
    }
}
```

Train args defines the arguments for training.

The argument `learning_rate` defines the initial learning rate (after potentail warmup period) to use.

The argument `batch_size` defines the batch size for training dataloader.

The argument `gradient_accumulation_steps` defines the number of updates steps to accumulate before performing a backward/update pass.

The argument `prediction_type` defines the prediction\_type that shall be used for training. Choose between 'epsilon' or 'v\_prediction' or leave `None`. If left to `None` the default prediction type of the scheduler: `noise_scheduler.config.prediction_type` is chosen.

The argument `max_grad_norm` defines the max gradient norm for clipping gradient norm in training.

Usually the training progress will take a long time to complete. We cannot run the whole training progress in one task, because each task has a max execution time limit in the crynux network, and the time limit is too short to complete the whole training progress. So we need to split the whole training progress into serveral tasks, each task runs only a few steps of the training progress and save its result. The next task will use the previous task result as its base model to continue the training. This progress will repeat until the whole training is completed. We use arguments `num_train_epochs` or `num_train_steps` to define the epochs or updates steps performed in one task, and arguments `max_train_epochs` or `max_train_steps` to define the epochs or updates steps the whole training progress takes. If `num_train_steps` and `max_train_steps` are provided, they will overrided `num_train_epochs` and `max_train_epochs`, respectively.

The argument `scale_lr` defines the whether to scale the learning rate by number of GPUs, gradient accumulation steps and batch size. Default is true.

The argument `resolution` defines the resolution for the input images. All the images in the train/validation dataset will be resize to this resolution.

The argument `noise_offset` defines the scale of noise offset.

The argument `snr_gamma` defines the SNR weighting gamma to be used if rebalancing the loss. Recommended value is 5.0. Default is null, means not to rebalance the loss.

#### lr\_scheduler

```json
"lr_scheduler": {
    "lr_scheduler": "cosine",
    "lr_warmup_steps": 500,
}
```

The argument `lr_scheduler` defines the learning rate scheduler type to use. Choose between \["linear", "cosine", "cosine\_with\_restarts", "polynomial", "constant", "constant\_with\_warmup"].

The argument `lr_warmup_steps` defines the number of steps for the warmup in the lr scheduler.

#### adam\_args

```json
"adam_args": {
    "beta1": 0.9,
    "beta2": 0.999,
    "weight_decay": 0.01,
    "epsilon": 1e-8
}
```

The argument `adam_args` defines parameters for the Adam optimizer.

The argument `beta1` defines the beta1 parameter for the Adam optimizer.

The argument `beta2` defines the beta2 parameter for the Adam optimizer.

The argument `weight_decay` defines the weight decay to use.

The argument `epsilon` defines the epsilon value for the Adam optimizer.

### lora

```json
"lora": {
    "rank": 4,
    "init_lora_weights": "gaussian",
    "target_modules": ["to_k", "to_q", "to_v", "to_out.0"]
}
```

Lora defines arguments for the lora layers.

The argument `rank` defines the dimension of the LoRA attention.

The argument `init_lora_weights` defines how to initialize the weights of the adapter layers. Can be a boolean or choose between \["gaussian", "loftq"]. Passing True (default) results in the default initialization from the reference implementation from Microsoft. Passing ‘gaussian’ results in Gaussian initialization scaled by the LoRA rank for linear and layers. Setting the initialization to False leads to completely random initialization and is discouraged. Pass 'loftq' to use LoftQ initialization.

The argument `target_modules` defines the names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed.

### transforms

```json
"transforms": {
    "center_crop": true,
    "random_flip": true,
}
```

Transforms defines the tranform operations applied to the image before training.

The argument `center_crop` defines whether to center crop the input images to the resolution. If not set, the images will be randomly cropped. The images will be resized to the resolution first before cropping.

The argument `random_flip` defines whether to randomly flip images horizontally.

### dataloader\_num\_workers

The argument `dataloader_num_workers` defines the number of subprocesses to use for data loading. 0 means that the data will be loaded in the main process.

### mixed\_precision

The argument `mixed_precision` defines whether to use mixed precision in training. Choose between fp16 and bf16 (bfloat16). Bf16 requires PyTorch >=1.10 and an Nvidia Ampere GPU. No means to disable the mixed precision.

### seed

The argument `seed` defines the seed used to initialize the random processes.

{% hint style="info" %}
Helium Network requires a deterministic algorithm for text generation, which means the text generated on the different nodes of the same deivces, given the same task definition, should be the same. This is a requirement for the consensus protocol to work. The seed is left as a required argument in the task definition so that all the nodes could use the same seed to initialize their random number generators, which will hopefully produce the same random numbers across all the nodes.

Beside the seed, the GPT Task Framework has been implemented to maximize the reproducibility.
{% endhint %}

### checkpoint

The argument `checkpoint` defines whether this task should be resumed from a previous checkpoint. It should be a directory containing the checkpoint files in your local file system. If the task is executed in the Hydrogen Network, this parameter will be injected automatically if the checkpoint is provided.

## Fine tuning Task Response

The fine tuning task response are two directories `checkpoint` and `validation`, stores the checkpoint files and validation result images, respectively.

The checkpoint files can be used as the final lora weights, or as the checkpoint the next task to be resumed from. The validation result images can be used to check the model quality.

When executing the fine tuning task, you need to pass an argument `output_dir` to specify where the `checkpoint` and `validation` directories will be stored. If the task is executed in the Hydrogen Network, the `output_dir` parameter will be injected automatically.


# API Specification of the Relay

The OpenAPI specification of the Relay

## Resources

The JSON schema of the OpenAPI Specification of the Relay can be found at:

{% embed url="<https://dy.relay.crynux.ai/openapi.json>" %}

The rendered document of the specification can be accessed at:

{% embed url="<https://dy.relay.crynux.ai/static/api_docs.html>" %}

## API List

### Task Related APIs

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/inference\_tasks" method="post" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/inference\_tasks/{task\_id}" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/inference\_tasks/{task\_id}/results" method="post" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/inference\_tasks/{task\_id}/results/{image\_num}" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

### Network Stats Related APIs

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/network" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/network/nodes/data" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/network/nodes/number" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/network/tasks/number" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}

### Other APIs

{% openapi src="<https://dy.relay.crynux.ai/openapi.json>" path="/v1/now" method="get" %}
<https://dy.relay.crynux.ai/openapi.json>
{% endopenapi %}


# Crynux SDK

SDKs to bootstrap the application development

To speed up the integration of the Crynux Network into the applications. SDKs in several commonly used languages have been provided. The whole workflow to send the tasks and get the results are well encapsulated to be invoked easily.

### Crynux SDK in Python

The source code of the SDK in Python can be found on the GitHub:

{% embed url="<https://github.com/crynux-network/crynux-sdk-python>" %}

The example usages are provided in the README of the project.

### Crynux SDK in Go

The SDK in Go will be released shortly.

### Crynux SDK in JavaScript

The source code of the SDK in JavaScript can be found on the GitHub:

{% embed url="<https://github.com/crynux-network/crynux-sdk-js>" %}

The example usages are provided in the README of the project.


# Multi-chain Architecture

Crynux Multi-chain Architecture

The Crynux Network is built on a multi-chain architecture, operating across multiple EVM-compatible blockchains. It currently supports Base and Near, with future plans to expand the ecosystem to more blockchains.

The architecture has three layers:

* **L0 — Ethereum**: the canonical CNX ERC20 token lives on Ethereum, together with the Emission contract that releases new CNX according to the emission schedule. All CNX in circulation originates from this layer.
* **L1 — Base and Near**: CNX is bridged from Ethereum to each supported L1 network, where it becomes the CNX token on that network. On Base, this is done through the standard Optimism token bridge; on Near, through the Rainbow Bridge.
* **L2 — Crynux Blockchains**: on each L1, Crynux runs a dedicated L2 blockchain. CNX on the L1 is bridged to the L2, where it becomes the native gas token of the chain, similar to how ETH works on Ethereum mainnet. `Crynux on Base` is launched on top of Base using Arbitrum Orbit, and `Crynux on Near` will be launched on top of Near as a Virtual Chain by Aurora.

```mermaid
flowchart BT
  CRYNUX_BASE["Crynux on Base (L2)<br/>(Arbitrum Orbit Chain)<br/>Native Token"] <-- Standard Arbitrum Token Bridge --> BASE["Base (L1)<br/>OptimismMintableERC20"]
  CRYNUX_NEAR["Crynux on Near (L2)<br/>(Virtual Chain by Aurora)<br/>Native Token"] <-- Token Bridge --> NEAR["Near (L1)<br/>NEP-141"]
  BASE <-- Standard Optimism Token Bridge --> ETH["Ethereum (L0)<br/>ERC20 + Emission"]
  NEAR <-- Rainbow Bridge --> ETH
```

You can use the Crynux Portal at [portal.crynux.io](https://portal.crynux.io) to add networks easily: open the site, connect your wallet, choose the network you want, and the portal will automatically add the corresponding network to MetaMask.

## Ethereum (L0)

Ethereum is the top layer of the architecture. It hosts the canonical CNX ERC20 token, which defines the total CNX supply across all networks, and the Emission contract, which releases new CNX to the network participants according to the emission schedule. CNX on every other network is a bridged representation of the canonical token on Ethereum.

<table><thead><tr><th width="190.800048828125">Contract</th><th>Address</th></tr></thead><tbody><tr><td>Crynux Token</td><td><a href="https://etherscan.io/token/0xa97998Bf97f5A6A96393b85B4e02A0440AE220F2">0xa97998Bf97f5A6A96393b85B4e02A0440AE220F2</a></td></tr><tr><td>Emission Contract</td><td><a href="https://etherscan.io/address/0x72666bA4dE68bB46b4dE59641af99346318016DD">0x72666bA4dE68bB46b4dE59641af99346318016DD</a></td></tr></tbody></table>

{% hint style="warning" %}
Crynux Portal does NOT support direct deposits and withdrawals to Ethereum Network. To move CNX between L1s and Ethereum, use their standard ERC20 bridge contracts.
{% endhint %}

## Crynux Relay

Payments in Crynux go through the Crynux Relay: users deposit CNX into their Relay account, spend and earn inside the Relay, and withdraw back on-chain anytime. As a result, the Relay always holds pooled user funds on-chain.

The Relay also makes cross-chain transfers easier for users: since deposits and withdrawals are supported on multiple networks, a user can deposit CNX on one network and withdraw it on another, without interacting with the underlying bridges directly. To serve withdrawals on every supported network, the Relay maintains a reserve of CNX on each of them, so its system wallets also hold the tokens backing these cross-chain reserves.

[Crynux Portal](https://portal.crynux.io) is the web frontend of the Relay. Through the Portal, users can access the Relay's cross-chain features: it supports direct deposits and withdrawals on Base and `Crynux on Base`, and can also be used to transfer CNX between Base and `Crynux on Base` without directly interacting with the native bridge contracts.

```mermaid
flowchart BT
  RELAY(("Crynux Relay")) <-- Deposit/Withdraw --> BASE["Base (L1)"]
  RELAY <-- Deposit/Withdraw --> CRYNUX_BASE["Crynux on Base (L2)"]
  RELAY <-. Coming Soon .-> NEAR["Near (L1)"]
  RELAY <-. Coming Soon .-> CRYNUX_NEAR["Crynux on Near (L2)"]
```

### System Wallets

The Relay uses a set of system wallets on-chain:

<table><thead><tr><th width="199.5999755859375">Wallet</th><th>Purpose</th></tr></thead><tbody><tr><td>Relay Deposit Address</td><td>Pooled user deposits. This is where users send CNX to top up their Relay accounts. Its balance is simply the aggregate deposits of all Relay users.</td></tr><tr><td>Relay Hot Wallet</td><td>Pays out Relay withdrawals. Its key lives on a server, and since any online key is a potential attack target, the hot wallet only keeps a small working balance.</td></tr><tr><td>Relay Cold Wallet</td><td>The Relay's cold-storage reserve. The key is kept fully offline. When the hot wallet runs low, it is manually refilled from cold storage. Even if the withdrawal server were fully compromised, only the small hot-wallet float would be at risk — never the main reserve.</td></tr></tbody></table>

The Relay deposit address and system wallets exist on every L1 and L2 network that supports Relay deposits and withdrawals. The addresses on each network are listed in the blockchain sections below.

## Crynux Blockchains

{% tabs %}
{% tab title="Base" %}

### Base (L1)

Base is an Ethereum Layer 2 chain using Optimism. In the Crynux architecture, Base serves as an L1 network. The Relay uses the same set of wallet addresses on Base as on Ethereum.

<table><thead><tr><th width="200.2000732421875">Item</th><th>Address</th></tr></thead><tbody><tr><td>Crynux Token CA</td><td><a href="https://basescan.org/token/0x9557DD9E241bc9636732623B672B4090AF519396">0x9557DD9E241bc9636732623B672B4090AF519396</a></td></tr><tr><td>Relay Deposit Address</td><td><a href="https://basescan.org/address/0x95dAd4af9aCaDEaf1704d3C980e7f571A9c5C5a0">0x95dAd4af9aCaDEaf1704d3C980e7f571A9c5C5a0</a></td></tr><tr><td>Relay Hot Wallet</td><td><a href="https://basescan.org/address/0x2Dc0538727d569cD40f7a2FcfD2749A3D62f44d9">0x2Dc0538727d569cD40f7a2FcfD2749A3D62f44d9</a></td></tr><tr><td>Relay Cold Wallet</td><td><a href="https://basescan.org/address/0x552A7D01C9e854244cC04Fd3e6C47f9036132f74">0x552A7D01C9e854244cC04Fd3e6C47f9036132f74</a></td></tr></tbody></table>

### Crynux on Base (L2)

`Crynux on Base` is an Arbitrum Orbit chain launched on top of Base. It uses CNX as its native gas token.

<table><thead><tr><th width="200.199951171875">Item</th><th>Value</th></tr></thead><tbody><tr><td>JSON RPC</td><td>https://json-rpc.base.crynux.io</td></tr><tr><td>Chain ID</td><td>18896214</td></tr><tr><td>Token Symbol</td><td>CNX</td></tr><tr><td>Decimal</td><td>18</td></tr><tr><td>Block Explorer</td><td><a href="https://cnxscan.base.crynux.io/">https://cnxscan.base.crynux.io</a></td></tr></tbody></table>

The Crynux Relay uses the following wallet addresses on `Crynux on Base`:

<table><thead><tr><th width="200.2000732421875">Wallet</th><th>Address</th></tr></thead><tbody><tr><td>Relay Deposit Address</td><td><a href="https://cnxscan.base.crynux.io/address/0x95dAd4af9aCaDEaf1704d3C980e7f571A9c5C5a0">0x95dAd4af9aCaDEaf1704d3C980e7f571A9c5C5a0</a></td></tr><tr><td>Relay Hot Wallet</td><td><a href="https://cnxscan.base.crynux.io/address/0x2Dc0538727d569cD40f7a2FcfD2749A3D62f44d9">0x2Dc0538727d569cD40f7a2FcfD2749A3D62f44d9</a></td></tr><tr><td>Relay Cold Wallet</td><td><a href="https://cnxscan.base.crynux.io/address/0x552A7D01C9e854244cC04Fd3e6C47f9036132f74">0x552A7D01C9e854244cC04Fd3e6C47f9036132f74</a></td></tr></tbody></table>
{% endtab %}

{% tab title="Near" %}
{% hint style="info" %}
Coming soon. The Near network is still being deployed and will be available shortly.
{% endhint %}

### Near (L1)

The Crynux Token on Near is bridged from the canonical CNX ERC20 token on Ethereum through the Rainbow Bridge.

### Crynux on Near (L2)

`Crynux on Near` will be launched on top of Near as a Virtual Chain by Aurora, using CNX as its native gas token.
{% endtab %}
{% endtabs %}


# Token Flow

This page explains, in simple terms, where your tokens are, when they move, and where you can see them. There are four main places/accounts involved:

{% hint style="success" %}
The [Crynux Portal](https://portal.crynux.io) now shows all token locations linked to your node wallet—Node Wallet, Beneficial Wallet, Stake Locked, and Relay Account—so you can review your entire token distribution in one place.
{% endhint %}

| Place             | Description                                                                       | Visible in on-chain wallet? | Where to view                                                            | Network check                                    |
| ----------------- | --------------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------ |
| Node Wallet       | Your on-chain node wallet (e.g., MetaMask).                                       | Yes (except staked portion) | [Crynux Portal](https://portal.crynux.io), MetaMask (node address)       | Check both L2 blockchains (switch networks)      |
| Beneficial Wallet | On-chain beneficial wallet (if configured) that receives refunds and withdrawals. | Yes (when used)             | [Crynux Portal](https://portal.crynux.io), MetaMask (beneficial address) | Check both L2 blockchains (switch networks)      |
| Stake Locked      | Tokens locked as stake after you start the node.                                  | No                          | [Crynux Portal](https://portal.crynux.io), Node WebUI                    | Check both L2 blockchains (switch node versions) |
| Relay Account     | Task fees credited while your node executes tasks. Stored in the Crynux Relay.    | No                          | [Crynux Portal](https://portal.crynux.io), Node WebUI                    | -                                                |

## Token Flowchart

```mermaid
flowchart TD
  NW["Node Wallet"]
  BW["Beneficial Wallet"]
  SL["Stake Locked"]
  RA["Relay Account"]

  NW -->|"Stake at start"| SL
  SL -->|"Refund on stop (no Beneficial)"| NW
  SL -->|"Refund on stop (with Beneficial)"| BW
  RA -->|"Withdraw (no Beneficial)"| NW
  RA -->|"Withdraw (with Beneficial)"| BW
```

## Token Movement

### Node Wallet

* When you start the node, the required stake is deducted from your On-chain Node Wallet.
* The deducted amount becomes Stake Locked. It will not show in MetaMask (or other wallets) because it is locked. You can see the locked amount in the [Crynux Portal](https://portal.crynux.io) and the Node WebUI (see Stake Locked below).
* When you stop the node and there is no Beneficial Address set, the stake refund goes back to the Node Wallet and will be visible there.
* When withdrawing task fees in the Portal and there is no Beneficial Address set, withdrawals go to the Node Wallet.

### Stake Locked

* What it is: the portion of tokens deducted from the Node Wallet at start and locked as stake by the node.
* Visibility: not visible in on-chain wallet balances; visible in the Node WebUI as “CNX Staked”, and in the [Crynux Portal](https://portal.crynux.io).
* Lifecycle: created when the node starts; released when the node stops.
  * If a Beneficial Address is configured, the released stake is refunded to the Beneficial Wallet.
  * Otherwise, the released stake is refunded to the Node Wallet.
* You cannot transfer Stake Locked directly; it becomes spendable only after it is refunded on stop.

### Beneficial Wallet

* If a Beneficial Address is configured for the Node Wallet, the stake refund after stopping the node is sent to the Beneficial Wallet. It will not appear in the Node Wallet; check the wallet that controls the Beneficial Address.
* If a Beneficial Address is configured, Portal withdrawals of task fees are sent to the Beneficial Wallet.
* Always verify balances using the wallet that holds the Beneficial Address.

### Relay Account

* Task fees earned by your node are credited to the Relay Account, which is recorded in the Crynux Relay.
* This balance is not reflected in the on-chain balance of your wallet. You can view it in the [Crynux Portal](https://portal.crynux.io) and in the Node WebUI.
* How to view: import or select your Node Wallet address in MetaMask, open the Crynux Portal, connect with the Node Wallet, and check the Dashboard. You can also check the Node WebUI.
* How to withdraw: in the Portal, use Withdraw to move funds from the Relay Account to an on-chain address.
  * If a Beneficial Address is configured, withdrawals go to the Beneficial Wallet.
  * Otherwise, withdrawals go to the Node Wallet.


# FAQ

## Node Starting Questions

<details>

<summary>Where is the faucet? / Where to get the test CNX tokens?</summary>

The test CNX tokens can be acquired using the slash command in the Discord of Crynux, follow the tutorial below:

[Get the Test CNX Tokens](broken://pages/7BF1BqejWnxoI8XfziZA)

</details>

<details>

<summary>Can I start multiple node instances on a single GPU?</summary>

**TLDR: you may get even less rewards by starting multiple nodes on a single device**

No one can stop you doing that.

If your GPU is powerful enough, the bottleneck becomes the consensus process (you will be waiting for other nodes to submit results), in such cases you could start multiple nodes to fully utilize the power of the GPU.

However, if your nodes are executing too many tasks simultaneously, the task execution will become slower (due to the bottleneck on GPU or network bandwidth). And if you are slower than the other 2 nodes in a task,

* You will get a smaller portion from the task fee.
* Your chance of receiving tasks will decrease, and you will get less tasks.
* Your node could be kicking out of the network. It is not a slashing though, the staked tokens are still safe.

The details can be found in the doc:

[Quality of Service (QoS)](/system-design/quality-of-service-qos)

Meanwhile, we are developing the new feature to support the concurrent task execution on powerful GPUs and multiple GPUs, which will fully utilize the local capabilities.

</details>

<details>

<summary>Can I start a node on multiple GPUs?</summary>

No. The node can execute one task on one GPU at the same time. If you have Multiple GPUs, you can start multiple nodes on the device, and assign each GPU to a different node. The tutorial can be found at:

[Assign GPU to the Node](/node-hosting/assign-gpu-to-the-node)

</details>

<details>

<summary>Can I use the same wallet on multiple node instances?</summary>

No you can't do it.

The same wallet can only get one task from the network at the same time. If multiple nodes are started with the same wallet, they will be executing the same task at the same time, and the nodes who submit the result later will just fail.

After the hot/cold wallet architecture is implemented, [as described in this doc](/node-hosting/private-key-security), it can also be used to easily collect funds from multiple nodes to a single cold wallet.

</details>

<details>

<summary>Can I use AMD Radeon cards to run a node?</summary>

Nope. The AMD GPUs are not supported at this moment. Only Nvidia GPU and Apple M1/M2/M3 are supported.

We will add support for AMD GPUs in a future release.

</details>

<details>

<summary>Can I start a node without GPU?</summary>

No. GPU is required to execute the AI tasks from the applications, which is the fundamental requirement of a Crynux Node.

</details>

<details>

<summary>Can I start a node on VPS?</summary>

If you mean VPS without GPUs, the answer is no. GPU is required to execute the AI tasks from the applications, which is the fundamental requirement of a Crynux Node.

</details>

## Node is not Working as Expected

<details>

<summary>Node manager init error: Failed to download models due to network issue</summary>

#### If you are using the Windows binary release

please find the log file according to this document:

[Locate the Error Message](/troubleshooting/locate-the-error-message)

If there are error messages similar to:

```
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\...\\crynux-node-helium-v2.0.7-windows-x64\\crynux-node-helium-v2.0.7-windows-x64\\data\\huggingface\\models--stabilityai--stable-diffusion-xl-base-1.0\\snapshots\\462165984030d82259a11f4367a4eed129e94a7b\\unet\\diffusion_pytorch_model.fp16.safetensors'
```

It is due to the long path limitation on Windows. Please enable the long path support according to this guide, and then restart the computer:

**Enable Long Path Support on Windows**

Open the Windows Registry Editor by pressing `Win + R` and typing `regedit`. Navigate to `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem`. Find the `LongPathsEnabled` DWORD (create it if not exist) and set its value to `1`.

#### Otherwise

Make sure you could connect to Huggingface on the device running the node. If you are using a proxy, please provide the proxy config to the node according to the doc:

[Proxy Settings](/node-hosting/proxy-settings)

</details>

<details>

<summary>Node manager init error: The initial inference task exceeded the timeout limit(5 min)</summary>

Your computer is too slow to run a Crynux Node. If the time required for your node to finish a task exceeds the timeout period, other nodes will abort the task since they do not want to waste more time on the waiting. And your node will get no reward at all.

Besides, more timeout on the tasks will decrease the QoS score of your node, which will eventually cause your node being kicked out of the network.

Please use a more powerful device to run the node instead. To understand the details, please refer to:

[Quality of Service (QoS)](/system-design/quality-of-service-qos)

</details>

<details>

<summary>The node status shows <code>Stopped</code> after running for a while</summary>

If there is no other error messages shown, the node is probably kicked out of the network due to frequent timeout on tasks.

* You may be running more nodes than your GPU could handle
* Your device may not be powerful enough to run a node

If the node has a slow GPU, or poor network, the task submission will be slow. If the time required to finish a task exceeds the timeout period, other nodes will abort the task since they do not want to waste more time on the waiting.

More timeout on the tasks will decrease the QoS score of the timeout node, which will eventually cause the node being kicked out of the network. It is not a slashing though, the staked tokens are still safe. The details can be found in the doc:

[Quality of Service (QoS)](/system-design/quality-of-service-qos)

</details>

<details>

<summary>Failed to execute script 'main' ... 5 validation errors for Config</summary>

If the following popup shows when starting the node on Windows:

<img src="/files/3gmLvmuFVUEguJojkuPW" alt="" data-size="original">

Please check your anti-virus software for deletion or quarantine of the files of the Node. The config file might have be deleted.

</details>


# Locate the Error Message

Find out what exactly goes wrong

To identify the cause of the problem, refer to the log file for the detailed error message and full stack trace. If seeking community help, providing these details initially can save a lot of time.

## Locate the log file

{% tabs %}
{% tab title="Windows" %}
Go to the directory where you click `Crynux Node.exe`, there is a sub directory with name `data`, and inside `data` folder there is a folder with name `logs`, all the log files can be found inside.
{% endtab %}

{% tab title="Mac" %}
The log files of the Mac app locates inside your home folder at:

`~/Library/Application\ Support/crynux.io/Crynux\ Node/`

To access this folder, open a terminal window and type in the following command:

`$ open ~/Library/Application\ Support/crynux.io/Crynux\ Node/`

And the log files are located inside under the `logs` folder.
{% endtab %}

{% tab title="Docker" %}
**Find the logs in the container output**

Find the container name of the Crynux Node:

```bash
$ docker ps
```

The output should be similar to:

```
CONTAINER ID   IMAGE                                COMMAND              CREATED          STATUS          PORTS                      NAMES
77e559a0d707   ghcr.io/crynux-network/crynux-node:2.0.4  "bash start.sh run"  33 minutes ago   Up 32 minutes   127.0.0.1:7412->7412/tcp   ecstatic_chatterjee
```

In this case, the container name is `ecstatic_chatterjee`.

In a terminal, type in the following command:

```bash
$ docker logs {container_name}
```

If you want to save the logs to a file, use the following command:

```bash
$ docker logs {container_name} >> crynux.log
```

**Find the log file inside the container**

The log file can also be found under `/app/logs` inside the container.
{% endtab %}

{% tab title="Linux" %}
If you downloaded the binary release version of Linux server, the log files can be found in the `logs` folder of the project root.
{% endtab %}

{% tab title="Source Code" %}
The log file is located at `logs/crynux-server.log`, relative to the project root folder.
{% endtab %}
{% endtabs %}

There are several log files inside the `logs` folder. The content of each file is described below:

* `crynux-server.log`: Node manager related logs.
* `crynux-worker.log`: Task executor related logs.
* `crynux_worker_inference.log`: Task execution logs.
* `crynux_worker_prefetch.log`: Model downloading logs.
* `main.log`: GUI related logs. Not available on Docker versions.

Most of the error messages could be identified in the first two log files: `crynux-server.log` and `crynux-worker.log`.

## Locate the error message

Open the log file in a text editor. Navigate to the time where you encountered the error, and find the lines with `[ERROR]`, which is usually the error message. And there will be a stack trace around the error message. **If you are asking for help, remember to provide the full stack trace from the first line to the last**.

Here is an example of a log file with error message and the stack trace:

```
[2024-05-15 18:08:27] [INFO    ] crynux_worker.prefetch: Start worker process: worker, data/huggingface, data/external
[2024-05-15 18:08:27] [INFO    ] crynux_worker.prefetch: Start prefetching models
[2024-05-15 18:08:35] [ERROR   ] crynux_server.node_manager.node_manager: Node manager init error: init task cancelled
Traceback (most recent call last):
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 454, in _run
    async with create_task_group() as init_tg:
  File "anyio\_backends\_asyncio.py", line 597, in __aexit__
  File "anyio\_backends\_asyncio.py", line 668, in task_done
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 262, in _init
    async for attemp in AsyncRetrying(
  File "tenacity\_asyncio.py", line 71, in __anext__
  File "tenacity\__init__.py", line 314, in iter
  File "concurrent\futures\_base.py", line 449, in result
  File "concurrent\futures\_base.py", line 401, in __get_result
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 269, in _init
    await to_thread.run_sync(
  File "anyio\to_thread.py", line 33, in run_sync
  File "anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
asyncio.exceptions.CancelledError
[2024-05-15 18:08:35] [INFO    ] crynux_server.node_manager.state_manager: Node status is NodeStatus.Stopped, cannot leave the network automatically
```

In this case, the error message is:

```
crynux_server.node_manager.node_manager: Node manager init error: init task cancelled
```

And the full stack trace is:

```
Traceback (most recent call last):
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 454, in _run
    async with create_task_group() as init_tg:
  File "anyio\_backends\_asyncio.py", line 597, in __aexit__
  File "anyio\_backends\_asyncio.py", line 668, in task_done
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 262, in _init
    async for attemp in AsyncRetrying(
  File "tenacity\_asyncio.py", line 71, in __anext__
  File "tenacity\__init__.py", line 314, in iter
  File "concurrent\futures\_base.py", line 449, in result
  File "concurrent\futures\_base.py", line 401, in __get_result
  File "D:\Crynux Node\_internal\crynux_server\node_manager\node_manager.py", line 269, in _init
    await to_thread.run_sync(
  File "anyio\to_thread.py", line 33, in run_sync
  File "anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
asyncio.exceptions.CancelledError
```


# Exceptions in WebUI


# Privacy Policy

How Crynux Network handles user data

\[**Last Updated: 2025-03-20]**

This document outlines how Crynux Network handle data within the decentralized AI infrastructure that provides inference and training services for LLM (Large Language Models), Stable Diffusion, and other AI models.

Crynux Network is designed as a decentralized infrastructure with permissionless nodes. This policy explains how your data is handled within this architecture.

### 1. Who's Data is Involved

The Crynux Network involves two types of participants whose data may be processed:

#### 1.1 Applications

Applications are services, platforms, or tools that utilize the Crynux Network for AI processing. When applications use the network:

* Their task inputs (prompts, images, etc.) are temporarily processed by the network
* They receive task outputs from the network
* Basic task statistics (completion, success/failure) are recorded for network operation
* No application identity information is collected beyond the wallet addresses

#### 1.2 Nodes

Nodes are providers of computational resources that execute AI tasks within the network. For node operators:

* Node performance metrics are collected (task completion, success rates)
* Node earnings and economic activity are recorded
* All node data is associated only with blockchain wallet addresses
* The GPU model is recorded for task distribution
* No personal identifiers, geographical information, or other system details are collected

#### 1.3 Types of Data Processed

In summary, the Crynux Network processes the following types of data:

* **Task Inputs:** Text prompts, images, or other inputs that applications send to the network for processing
* **Task Outputs:** Generated images, text responses, or other outputs created by the network
* **Network Statistics:** Aggregated data about tasks, success rates, task numbers, and node earnings. These statistics are only associated with blockchain wallet addresses and contain no personally identifiable information such as IP addresses, location, country, or time zone

The network does NOT collect or store:

* User personal information (names, email addresses, etc.)
* IP addresses or location information from applications
* IP addresses or location information from nodes
* Geographical data (country, time zone, etc.)

### 2. Where the Data is Processed

The Crynux Network processes data across different components of its architecture:

#### 2.1 For Applications:

* Application data (prompts, images, etc.) is first sent to the Relay component
* The Relay then distributes this data to the selected Nodes for processing
* Results are returned from the Nodes to the Relay, and then back to the Application

#### 2.2 For Nodes:

* Node performance statistics and metrics are collected and stored by the Relay
* Node earnings and economic activity are recorded by the Relay

#### 2.3 Blockchain Data:

* Both Applications and Nodes have certain public data recorded on the blockchain
* This includes wallet addresses, task hashes, consensus data, and transaction information
* Node specifications (such as GPU model) are publicly recorded on the blockchain

### 3. How the Network Handle Your Data

The Crynux Network operates with the following data handling principles:

* **Temporary Storage:** Task inputs and outputs are stored only on the Relay during task execution
* **Automatic Deletion:** All data is deleted from the Relay after task completion
* **Decentralized Processing:** Tasks are distributed to permissionless Nodes for execution
* **Node Data Cleanup:** Temporary task data is deleted from the Nodes after task execution is complete
* **Limited Analytics Collection:** The Relay collects network statistics such as total tasks, success rates, task numbers, and node earnings. These statistics are only associated with blockchain wallet addresses and contain no personally identifiable information such as IP addresses, location, country, or time zone

### 4. Data Storage Limitations

Our official implementation of the Crynux Node is designed to process task data without persistent storage. However, as a decentralized and permissionless network, we cannot technically prevent third-party node implementations from storing data processed during task execution.

Applications using the Crynux Network should be aware that while we provide guidelines and implementations that respect privacy, we cannot guarantee the behavior of all nodes in the network.

### 5. Blockchain Data

The Crynux Network uses blockchain technology to coordinate tasks and execute the consensus protocol. Information recorded on the blockchain is public and immutable, but is limited to:

* Task identifiers (hashes)
* Node participation information
* Consensus-related data (such as p-hash of images)

The actual content of tasks (prompts, images, etc.) is not stored on the blockchain.

### 6. Changes to This Policy

We may update this Privacy Policy from time to time. We will notify users of any changes by updating the "Last Updated" date at the top of this policy.


