Unstable Program Load Order Can Stall Node Bootup
During node bootup, programs are loaded in order of block height. However, within a single block, the load order of multiple programs is not stable. This instability can cause loading failures and stall node bootup.
/// Initializes the VM from storage.
#[inline]
pub fn from(store: ConsensusStore<N, C>) -> Result<Self> {
[...]
// Retrieve the list of deployment transaction IDs and their associated block heights.
let deployment_ids = transaction_store.deployment_transaction_ids().collect::<Vec<_>>();
let mut deployment_ids = cfg_into_iter!(deployment_ids)
.map(|transaction_id| {
// Retrieve the height.
let height =
match block_store.find_block_hash(&transaction_id)?.map(|hash| block_store.get_block_height(&hash))
{
Some(Ok(Some(height))) => height,
_ => {
bail!("Block height for deployment transaction '{transaction_id}' is not found in storage.")
}
};
Ok((transaction_id, height))
})
.collect::<Result<Vec<_>>>()?;
// Sort the deployment transaction IDs by their block heights.
deployment_ids.sort_unstable_by(|(_, a), (_, b)| a.cmp(b));
// Load the deployments in order of their block heights.
const PARALLELIZATION_FACTOR: usize = 256;
for (i, chunk) in deployment_ids.chunks(PARALLELIZATION_FACTOR).enumerate() {
// Load the deployments.
let deployments = cfg_iter!(chunk)
.map(|(transaction_id, _)| {
// Retrieve the deployment from the transaction ID.
match transaction_store.get_deployment(transaction_id)? {
Some(deployment) => Ok(deployment),
None => bail!("Deployment transaction '{transaction_id}' is not found in storage."),
}
})
.collect::<Result<Vec<_>>>()?;
// Add the deployments to the process.
// Note: This iterator must be serial, to ensure deployments are loaded in the order of their dependencies.
deployments.iter().try_for_each(|deployment| process.load_deployment(deployment))?;
}
[...]
}
SnarkVM enforces restrictions on finalize_cost and number_of_calls for programs, which are checked during program initialization. If Program B imports and calls Program A, an upgrade to Program A may cause Program B’s functions to exceed these restrictions. This is not an issue during deployment execution, since Program B is not re-checked after Program A is upgraded. However, during node bootup, every program is re-checked, and because the program load order within a block is not stable, Program B may be loaded after Program A’s upgrade. This can trigger a restriction check failure and prevent the node from booting.
Example sequence:
- Block 1: Deploy Program A.
- Block 2: Deploy Program B (which imports A) and upgrade Program A, increasing its calls or finalize instructions.
- During node bootup, when loading programs in Block 2, if Program B is loaded after Program A’s upgrade, the restriction check fails and node bootup is stalled.
Recommendation
It is recommended to load programs in the order of their transaction index within each block during node bootup to ensure a stable and deterministic load order.
Client Response
The client fixed this by sorting the deployment transaction according to the block height and transaction index.
let mut deployment_ids = cfg_into_iter!(deployment_ids)
.map(|transaction_id| {
// Retrieve the block hash for the deployment transaction ID.
let Some(hash) = block_store.find_block_hash(&transaction_id)? else {
bail!("Deployment transaction '{transaction_id}' is not found in storage.")
};
// Retrieve the height.
let Some(height) = block_store.get_block_height(&hash)? else {
bail!("Block height for deployment transaction '{transaction_id}' is not found in storage.")
};
// Get the corresponding block's transactions.
let Some(transactions) = block_store.get_block_transactions(&hash)? else {
bail!("Transactions for deployment transaction '{hash}' is not found in storage.")
};
// Find the index of the deployment transaction ID in the block's transactions.
let Some(index) = transactions.transactions().get_index_of(transaction_id.deref()) else {
bail!("Transaction for deployment transaction '{transaction_id}' is not found in storage.")
};
Ok((transaction_id, (height, index)))
})
.collect::<Result<Vec<_>>>()?;