Skip to main content

Akash Development Environment

Follow these sequential steps to build a local Akash development environment.

Overview and Requirements​

Overview​

This page covers setting up development environment for both node and provider repositories. The provider repo elected as placeholder for all the scripts as it depends on the node repo. Should you already know what this guide is all about - feel free to explore examples.

Requirements​

Golang​

Go must be installed on the machine used to initiate the code used in this guide. Both projects - Akash Node and Provider - are keeping up-to-date with major version on development branches. Both repositories are using the latest version of the Go, however only minor that has to always match.

Docker Engine​

Ensure that Docker Desktop/Engine has been installed on machine that the development environment will be launched from.

Direnv Use​

Install Direnv if Necessary​

Direnv is used for the install process. Ensure that you have Direnv install these instructions.

Configure Environment for Direnv Use​
  • Edit the ZSH shell profile with visual editor.
vi .zshrc
  • Add the following line to the profile.
eval "$(direnv hook zsh)"

Code​

In the example use within this guide, repositories will be located in ~/go/src/github.com/akash-network. Create directory if it does not already exist via:

mkdir -p ~/go/src/github.com/akash-network

Clone Akash Node and Provider Repositories​

NOTE - all commands in the remainder of this guide assume a current directory of ~/go/src/github.com/akash-networkunless stated otherwise.

cd ~/go/src/github.com/akash-network 
git clone https://github.com/akash-network/node.git
git clone https://github.com/akash-network/provider.git

Install Tools​

Run following script to install all system-wide tools. Currently supported host platforms.

  • MacOS
  • Debian based OS PRs with another hosts are welcome
  • Windows is not supported
cd ~/go/src/github.com/akash-network
./provider/script/install_dev_dependencies.sh

Development Environment General Behavior​

All examples are located within _run directory. Commands are implemented as make targets.

There are three ways we use to set up the Kubernetes cluster.

  • kind
  • minukube
  • ssh

Both kind and minikube are e2e, i.e. the configuration is capable of spinning up cluster and the local host, whereas ssh expects cluster to be configured before use.

Runbook Overview​

There are four configuration variants, each presented as directory within _run.

  • kube - uses kind to set up local cluster. It is widely used by e2e testing of the provider. Provider and the node run as host services. All operators run as kubernetes deployments.
  • single - uses kind to set up local cluster. Main difference is both node and provider (and all operators) are running within k8s cluster as deployments. (at some point we will merge single with kube and call it kind)
  • minikube - not in use for now
  • ssh - expects cluster to be up and running. mainly used to test sophisticated features like GPU or IP leases

The only difference between environments above is how they set up. Once running, all commands are the same.

This guide covers the two most frequently used development paths: Local Kind Cluster (kube) and Remote SSH Cluster (ssh). Running through either runbook requires multiple terminals. Each command is marked t1-t3 to indicate a suggested terminal number.

Parameters​

Parameters for use within the Runbooks detailed later in this guide.

NameDefault valueEffective on target(s)
SKIP_BUILDfalse
DSEQ1
  • deployment-*
  • lease-*
  • bid-*
  • send-manifest
OSEQ1
  • deployment-*
  • lease-*
  • bid-*
  • send-manifest
GSEQ1
  • deployment-*
  • lease-*
  • bid-*
  • send-manifest
KUSTOMIZE_INSTALLS

Depends on runbook.
Refer to each runbook's Makefile to see default value.

  • kustomize-init
  • kustomize-templates
  • kustomize-set-images
  • kustomize-configure-services
  • kustomize-deploy-services

Use Runbook - Local Kind Cluster (kube)​

Overview​

The local Kind cluster runbook creates a complete development environment on your local machine using Docker and Kind (Kubernetes in Docker). This is the most common development setup and requires no external resources.

Cleanup and Restart​

If at any point something goes wrong and you need to start from the beginning:

cd provider/_run/kube
make kube-cluster-delete
make clean
make init

Runbook Steps​

NOTE - this runbook requires three simultaneous terminals

For the purpose of documentation clarity we will refer to these terminal sessions as:

  • terminal1
  • terminal2
  • terminal3

STEP 1 - Open Runbook​

NOTE - run the commands in this step on terminal1, terminal2, and terminal3

Run this step on all three terminal sessions to ensure we are in the correct directory for later steps.

cd ~/go/src/github.com/akash-network/provider/_run/kube

STEP 2 - Create and Provision Local Kind Kubernetes Cluster​

NOTE - run this command in this step on terminal1 only

NOTE - this step may take several minutes to complete

make kube-cluster-setup
Possible Timed Out Waiting for the Condition Error​

If the following error is encountered when running make kube-cluster-setup:

Waiting for deployment "ingress-nginx-controller" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ingress-nginx-controller" rollout to finish: 0 of 1 updated replicas are available...
error: timed out waiting for the condition
make: *** [../common-kube.mk:120: kube-setup-ingress-default] Error 1

This is an indication that the Kubernetes ingress-controller did not initialize within the default timeout period. In such cases, re-execute make kube-cluster-setup with a custom timeout period such as the example below. This step is NOT necessary if make kube-cluster-setup completed on first run with no errors encountered.

cd provider/_run/kube
make kube-cluster-delete
make clean
make init
KUBE_ROLLOUT_TIMEOUT=300 make kube-cluster-setup
Goreleaser Issue​

If build fails due to:

Unable to find image 'ghcr.io/goreleaser/goreleaser-cross:v' locally

This is likely due to GOVERSION_SEMVER environment variable not set properly. Can use following as temp workaround and to proceed:

# Set the correct version based on your Go installation - use your locally installed Go version and replace below
export GOVERSION_SEMVER=v1.24.2

# Verify it's set correctly
echo "GOVERSION_SEMVER is now: $GOVERSION_SEMVER"

# Now retry your make command
KUBE_ROLLOUT_TIMEOUT=300 make kube-cluster-setup

STEP 3 - Start Akash Node​

NOTE - run this command in this step on terminal2 only

make node-run

STEP 4 - Create an Akash Provider​

NOTE - run this command in this step on terminal1 only

make provider-create
Note on Keys​

Each configuration creates four keys: The keys are assigned to the targets and under normal circumstances there is no need to alter it. However, it can be done with setting KEY_NAME:

# create provider from **provider** key
make provider-create

# create provider from custom key
KEY_NAME=other make provider-create

STEP 5 - Start the Akash Provider​

NOTE - run this command in this step on terminal3 only

make provider-run

STEP 6 - Create and Verify Test Deployment​

NOTE - run the commands in this step on terminal1 only

Create the Deployment​
  • Take note of the deplpyment ID (DSEQ) generated for use in subsequent steps
make deployment-create
Query Deployments​
make query-deployments
Query Orders​
  • Steps ensure that an order is created for the deployment after a short period of time
make query-orders
Query Bids​
  • Step ensures the Provider services daemon bids on the test deployment
make query-bids

STEP 7 - Test Lease Creation for the Test Deployment​

NOTE - run the commands in this step on terminal1 only

Create Lease​
make lease-create
Query Lease​
make query-leases
Ensure Provider Received Lease Create Message​
  • Should see "pending" inventory in the provider status and for the test deployment
make provider-status

STEP 8 - Send Manifest​

NOTE - run the commands in this step on terminal1 only

Send the Manifest to the Provider​
make send-manifest
Check Status of Deployment​
make provider-lease-status
Ping the Deplpyment to Ensure Liveness​
 make provider-lease-ping

STEP 9 - Verify Service Status​

NOTE - run the commands in this step on terminal1 only

Query Lease Status​
make provider-lease-status
Fetch Pod Logs​
  • Note that this will fetch the logs for all pods in the Kubernetes cluster. Filter/search for the test deployment's ID (DSEQ) for related activities.
make provider-lease-logs

Use Runbook - Remote SSH Cluster (ssh)​

Overview​

The SSH runbook connects to a pre-existing remote Kubernetes cluster via SSH and kubeconfig access. This approach is primarily used for testing sophisticated features like GPU workloads or IP leases that may require specific hardware or network configurations.

Important: In the SSH development environment, the Akash node and provider services run locally on your machine, while the remote Kubernetes cluster is used for hosting deployments and workloads. This hybrid approach allows you to develop and debug Akash services locally while leveraging remote cluster resources for actual workload execution.

Prerequisites​

Before starting this runbook, you must have:

  1. A running Kubernetes cluster accessible via SSH
  2. SSH key-based authentication configured for the cluster nodes
  3. Kubectl access to the cluster from your local machine
  4. Root or sudo access on the cluster nodes for container image management

Remote Cluster Setup​

STEP 1 - Configure SSH Access​

Set up passwordless SSH access to your remote cluster:

# Edit SSH config to specify key and user
vim ~/.ssh/config

# Add entry for your cluster
Host <YOUR_CLUSTER_IP>
    IdentityFile ~/.ssh/your-key
    User root
    IdentitiesOnly yes

# Test SSH access
ssh <YOUR_CLUSTER_IP> "kubectl get nodes"

STEP 2 - Prepare Remote Cluster​

Ensure your remote cluster has the required container management tools:

# SSH into your remote cluster node
ssh root@<YOUR_CLUSTER_IP>

# Install required packages
apt-get update
apt-get install -y uidmap slirp4netns

# Install nerdctl for container image management
curl -sSL https://github.com/containerd/nerdctl/releases/download/v1.7.2/nerdctl-1.7.2-linux-amd64.tar.gz | tar Cxzv /usr/local/bin/

# Verify nerdctl works with your cluster's containerd
nerdctl version

# Label the node for ingress controller scheduling (required for single-node clusters)
kubectl get nodes  # Note the node name
kubectl label node <NODE_NAME> ingress-ready=true

# Verify the label was added
kubectl get nodes --show-labels | grep ingress-ready

STEP 3 - Configure Local Kubeconfig Access​

Set up kubectl access to your remote cluster:

# SSH into your remote cluster
ssh root@<YOUR_CLUSTER_IP>

# First, update the API server to advertise the external IP
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

# Find this line:
# - --advertise-address=<INTERNAL_IP>
# Change it to:
# - --advertise-address=<YOUR_CLUSTER_IP>

# Wait for the API server pod to restart (check with: kubectl get pods -n kube-system)
# This may take a minute or two

# Once restarted, create a kubeconfig with external IP
kubectl config view --raw > /tmp/external-kubeconfig.yaml

# Update the server URL to use external IP
sed -i 's|https://127\.0\.0\.1:6443|https://<YOUR_CLUSTER_IP>:6443|' /tmp/external-kubeconfig.yaml

# Example:
# sed -i 's|https://127\.0\.0\.1:6443|https://43.57.209.6:6443|' /tmp/external-kubeconfig.yaml

# Check the cluster name in the kubeconfig
grep "name:" /tmp/external-kubeconfig.yaml

# Configure to skip TLS verification using the correct cluster name (typically cluster.local)
kubectl config set-cluster cluster.local --insecure-skip-tls-verify=true --kubeconfig /tmp/external-kubeconfig.yaml

# Verify the configuration works on the remote host
KUBECONFIG=/tmp/external-kubeconfig.yaml kubectl get nodes

# Exit back to your local machine
exit

Now copy the working kubeconfig to your local machine:

# Copy the working kubeconfig from remote cluster
scp -i ~/.ssh/your-key root@<YOUR_CLUSTER_IP>:/tmp/external-kubeconfig.yaml ~/.kube/remote-cluster-config

# Set KUBECONFIG environment variable
export KUBECONFIG=~/.kube/remote-cluster-config

# Test cluster access from local machine
kubectl get nodes

Local Environment Setup​

STEP 4 - Navigate to SSH Directory and Configure Environment​

cd ~/go/src/github.com/akash-network/provider/_run/ssh

Configure the required environment variables:

# Edit .envrc
vim .envrc

# Ensure it contains:
source_up .envrc

dotenv_if_exists dev.env

AP_RUN_NAME=$(basename "$(pwd)")
AP_RUN_DIR="${DEVCACHE_RUN}/${AP_RUN_NAME}"

export AKASH_HOME="${AP_RUN_DIR}/.akash"
export AKASH_KUBECONFIG=$KUBECONFIG
export AP_KUBECONFIG=$KUBECONFIG
export AP_RUN_NAME
export AP_RUN_DIR
export KUBE_SSH_NODES="root@<YOUR_CLUSTER_IP>"

# Reload direnv
direnv reload

# Verify required variables are set
echo "AP_RUN_NAME: $AP_RUN_NAME"
echo "KUBE_SSH_NODES: $KUBE_SSH_NODES"

STEP 5 - Address Known Issues​

Fix Goreleaser Version Issue​
# Set the correct goreleaser version based on your Go installation
export GOVERSION_SEMVER=v1.24.2
Fix Rootless Containerd Issue​

The setup script may fail when trying to install rootless containerd as root. Fix this by modifying the setup script:

# Create backup and modify setup script
cp ~/go/src/github.com/akash-network/provider/script/setup-kube.sh ~/go/src/github.com/akash-network/provider/script/setup-kube.sh.backup

# Comment out the problematic rootless setup line
sed -i.bak "s/ssh -n \"\$node\" 'containerd-rootless-setuptool.sh install'/# ssh -n \"\$node\" 'containerd-rootless-setuptool.sh install' # Commented out - using system containerd/" ~/go/src/github.com/akash-network/provider/script/setup-kube.sh
Fix Missing Makefile Target​

The SSH Makefile is missing required targets. Add them using these simple commands:

# Add the missing kube-deployments-rollout target to SSH Makefile
echo -e "\n.PHONY: kube-deployments-rollout\nkube-deployments-rollout:" >> ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile

# Add the missing kube-setup-ssh target to SSH Makefile
echo -e "\n.PHONY: kube-setup-ssh\nkube-setup-ssh:" >> ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile
Fix Provider Kubeconfig​

The provider service needs to use the correct kubeconfig for the remote cluster:

# Add kubeconfig flag to provider-run command
sed -i '' 's/--cluster-k8s \\/--cluster-k8s \\\
        --kubeconfig "$(AKASH_KUBECONFIG)" \\/' ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile
Fix GPU Target Missing Steps​

The SSH Makefile's GPU target is missing critical setup steps that are present in the standard setup. This fix ensures the GPU path includes namespace creation, ingress controller setup, and other required components:

# Add missing setup steps to GPU target
sed -i '' 's/kube-cluster-setup-gpu: init \\/kube-cluster-setup-gpu: init \\\
    $(KUBE_CREATE) \\\
    kube-cluster-check-info \\\
    kube-setup-ingress \\/' ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile

# Add helm charts installation to GPU target
sed -i '' 's/kube-deployments-rollout \\/kube-deployments-rollout \\\
    kube-install-helm-charts \\/' ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile

# Fix GPU target to use correct SSH setup target instead of kind target
sed -i '' 's/kind-setup-$(KIND_NAME)/kube-setup-$(AP_RUN_NAME)/' ~/go/src/github.com/akash-network/provider/_run/ssh/Makefile

Cluster Initialization​

STEP 6 - Initialize Environment​

# Initialize the Akash environment
make init

STEP 7 - Set Up Remote Cluster​

# Set required environment variables
export GOVERSION_SEMVER=v1.24.2
export KUBE_SSH_NODES="root@<YOUR_CLUSTER_IP>"

# Set up the cluster (may require extended timeout for ingress controller)
KUBE_ROLLOUT_TIMEOUT=300 make kube-cluster-setup

GPU Support​

The SSH runbook supports both standard and GPU workloads:

Standard Setup​

# Use the default target for non-GPU workloads
KUBE_ROLLOUT_TIMEOUT=300 make kube-cluster-setup

GPU Setup​

# Use the GPU-specific target for GPU workloads
KUBE_ROLLOUT_TIMEOUT=300 make kube-cluster-setup-gpu

Note: GPU support requires appropriate GPU drivers and container runtime configuration on your remote cluster nodes.

Provider Operations​

Once the cluster setup is complete, you can proceed with the standard Akash provider workflow:

STEP 8 - Start Akash Node​

NOTE - run this command in terminal2

First, ensure the environment is properly loaded:

# Navigate to SSH directory and set kubeconfig
cd ~/go/src/github.com/akash-network/provider/_run/ssh
export KUBECONFIG=~/.kube/remote-cluster-config
direnv reload

# Verify variables are set
echo "KUBECONFIG: $KUBECONFIG"
echo "AKASH_KUBECONFIG: $AKASH_KUBECONFIG"
echo "AP_RUN_NAME: $AP_RUN_NAME"

# Start the node
make node-run

STEP 9 - Create and Run Provider​

NOTE - run these commands in separate terminals

# Terminal 1: Create provider (ensure environment is loaded first)
cd ~/go/src/github.com/akash-network/provider/_run/ssh
export KUBECONFIG=~/.kube/remote-cluster-config
direnv reload

# Verify variables are set
echo "KUBECONFIG: $KUBECONFIG"
echo "AKASH_KUBECONFIG: $AKASH_KUBECONFIG"

make provider-create

# Terminal 3: Run provider (ensure environment is loaded first)
cd ~/go/src/github.com/akash-network/provider/_run/ssh
export KUBECONFIG=~/.kube/remote-cluster-config
direnv reload

# Verify variables are set
echo "KUBECONFIG: $KUBECONFIG"
echo "AKASH_KUBECONFIG: $AKASH_KUBECONFIG"

# Start the provider
make provider-run

STEP 10 - Test Deployment Workflow​

NOTE - run these commands in terminal1

# Create test deployment
make deployment-create

# Query and verify
make query-deployments
make query-orders
make query-bids

# Create lease
make lease-create
make query-leases

# Send manifest and verify
make send-manifest
make provider-lease-status
make provider-lease-ping

Troubleshooting​

Common Issues​

  1. SSH Permission Denied: Ensure SSH keys are properly configured and SSH agent is running
  2. Kubeconfig Access Issues: Verify external IP access and firewall rules for port 6443
  3. Container Image Upload Failures: Ensure nerdctl is properly installed on remote nodes
  4. Timeout Errors: Use KUBE_ROLLOUT_TIMEOUT=300 for slower network connections

To reset the environment:

make init

Key Differences from Local Kind Cluster​

  • No cluster creation: Uses existing remote cluster
  • Dual access method: Requires both SSH (for node operations) and kubeconfig (for Kubernetes API) access
  • Manual prerequisite setup: Remote cluster must be prepared with required tools
  • Extended timeouts: Network latency may require longer timeout values
  • Security considerations: Production clusters should use proper TLS certificates instead of skipping verification