2023.1.26 Update
- Fix broken links
In this article, we are going to talk about the Cosmos blockchain upgrade. On the Cosmos-SDK documentation, this process is called ‘In-Place Store Migrations’
Although there are some documents related to the In-Place Store Migrations, I feel that those are quite high-level explanations, and the knowledge is scattered. As a blockchain engineer in the team, I want to understand the details of the upgrade process because error while upgrading the chain is critical to the network as well as to the community. So I want to make sure how Cosmos-SDK-based blockchain can be upgraded in detail and what components work together for the blockchain to be upgraded.
For understanding In-Place Store Migrations of Cosmos-SDK-based blockchain, we need to see Cosmovisor and some Cosmos-SDK modules.
This article contains the following contents:
- ‘Upgrade Overview’. We are going to see how the chain can be upgraded in an abstract view.
- ‘Terms’: We are going to describe the words that show up frequently in the post.
- ‘Upgrade From ‘Cosmovisor’ Point of View’. Lots of components from different layers work together for upgrading the blockchain successfully, in this section we are going to see the upgrade from the Cosmovisor point of view.
- ‘Upgrade From ‘Application’ Point of View’.we are going to see the upgrade from the Application point of view.
- ‘Component Details’. We are going to deep-dive into each component related to the chain upgrade and how they are related to each other. Such as
x/upgrade
,x/gov
,ModuleManager
,Configurator
. - ‘Things to do for the upgrade as a developer’. Before going to the details of each component, we’re going to list what we need to write code from the developer’s perspective.
- ‘Demo’. We are going to see a simple demo of the Cosmos-SDK-based chain upgrade example with its sample script.
Terms
- Cosmovisor: Cosmovisor is a daemon that runs the application as its subprocess. And also a file watcher that polls and reads the file of
$DAEMON_HOME/data/upgrade-info.json
.upgrade-info.json
file has the information of the upgrade name, and download URLs of the binary. - UpgradeHandler: UpgradeHandler defines what modules need to be updated for the version. If the modules that need to be updated are not managed on the application such as
x/bank
on the Cosmos-SDK, that modules upgrade logic could be written in the UpgradeHandler. UpgradeHandler is written on the App level, not on the module level. UpgradeHandler must be registered tox/upgrade
module. - MigrationHandler: MigrationHandler defines how the module should be upgraded. MigrationHandler can contain the logic of how the parameters are updated, and how the state of the old scheme should be migrated into the new version. MigrationHandler must be registered to
Configurator
.
Comsos Blockchain Upgrade From ‘Cosmovisor’ Point of View
- Run application (blockchain) by Cosmovisor.
- If we need to upgrade the blockchain, build a new application binary that contains a MigrationHandler and UpgradeHandler. Then place that binary into the directory where Cosmosvisor can lookup.
- Submit SoftwareUpgrade proposal transaction to the
x/gov
module with the deposit. And make it pass with the vote. SoftwareUpgrade proposal contains the name of the upgrade, and the block height we want to upgrade. Each node that receives a proposal setPlan
data structure insidex/upgrade
module store - Every block is created and received by the node,
x/upgrade
compares the block height with the height ofPlan
we stored before. If blockchain reaches that height it writesPlan
data as a file into$NODE_HOME/data/upgrade-info.json
and killed itself bypanic()
. - Cosmovisor finds the application process is killed and based on the
upgrade-info.json
data it updates the symbolic link to lookup the proper directory which contains the updated binary. After that, Cosmovisor re-runs the application binary on the symbolic link. It automatically runs the new version application. - As a new version of the application has the migration handler, it runs the migration handlers first. And if it was successful, the application runs normally as before.
We can describe the abstract workflow of the upgrading procedure in the diagrams below:



upgrade-info.json
and killed itself

Comsos Blockchain Upgrade From ‘Application’ Point of View
Let’s take a look from the Application point of view. There’re lots of components that work together to make In-Store Migration correctly.
- The client can send the SoftwareUpgrade proposal to the
x/gov
module MsgServer. After the proposal is passed, the content of the proposal is routed to thex/upgrade
module SoftwareUpgradeHandler and is saved asPlan
data in thex/upgrade
module state. - Every time the block is created,
x/upgrade
‘s BeginBlocker checks whether it’s time to upgrade by block height. If it’s time, the old version application is panic and the new version of the application is run by Cosmovisor. - After the new version of the application is run,
x/upgrade
module runs the registered UpgradeHandler and UpgradeHandler runs the ModuleManagerRunMigrations(…)
method which calls each module’s MigrationHandler if needed. So for the module to be updated successfully, MigrationHandler must be registered on the new version of the application.




Component Details
Now, let’s take a look at each component in detail.
Cosmovisor
For Cosmovisor to work correctly, we or the node operator need to setup the folder structure. The whole folder structure would be as follows:
. # $DAEMON_HOME
├── config
├── cosmovisor
│ ├── current # symbolic link
│ ├── genesis
│ │ └── bin
│ │ └── simappd # $DAEMON_NAME
│ └── upgrades
│ └── v2.0.0 # upgrade name
│ └── bin
│ └── simappd # $DAEMON_NAME
└── data
cosmovisor
is the directory that Cosmovisor actually uses for upgrading the application. And the others like config
, data
is for the node as we know.
Let’s look at the cosmovisor
directory. current
is a symbolic link Cosmovisor generates so we don’t need to create it manually. genesis
the directory contains the first version of the application binary. Cosmovisor looks up $DAEMON_HOME/data/upgrade-info.json
when it starts and if the file does not exist, create current
with the reference of genesis
. In other words, when Cosmovisor runs up the first time, it runs the binary and points to the genesis
directory.
upgrades
directory contains various versions of binaries. You can find the v2.0.0
directory under the upgrades
. v2.0.0
must match the upgrade name when we submit the SoftwareUpgrade proposal to the x/gov
module.
As we’ve seen in the ‘Upgrade Overview’ section, before the old version of the application killed itself, it writes upgrade-info.json
with the information of the new upgrade name. And that name is determined by the contents of the SoftwareUpgrade proposal we submitted before.
And Cosmovisor reads the upgrade-info.json
and finds which version of the application should be run. Cosmovisor will update the current
symbolic link with the directory name of the upgrade name (e.g. v2.0.0
). As always, Cosmovisor finds the binary to run in the current
symbolic link, but at this time, it points to the new version of the application.
You can find the official docs here.
x/gov
What x/gov
mainly done for the upgrade is to receive the SoftwareUpgrade proposal and after the proposal is passed, send proposal content to the x/upgrade
module to make it do subsequent jobs for the upgrade.


The client can send the software upgrade proposal to the x/gov
module message server. Proposal structures look as follows:
// SoftwareUpgradeProposal is a gov Content type for initiating a software
// upgrade.
message SoftwareUpgradeProposal {
string title = 1;
string description = 2;
Plan plan = 3 [(gogoproto.nullable) = false];
}
// Plan specifies information about a planned upgrade and when it should occur.
message Plan {
string name = 1;
...
int64 height = 3;
string info = 4;
}
https://github.com/cosmos/cosmos-sdk/blob/main/proto/cosmos/upgrade/v1beta1/upgrade.proto#L12-L56
The proposal contains the title and description of the upgrade and the ‘Plan
’ structure. The Plan
contains important data related to the upgrade. name
must match with the directory name that Cosmovisor watches under the $DAEMON_HOME/cosmovisor/upgrades/
. For example, if the name of the upgrade is v2.0.0
, the application binary v2.0.0
must be in the $DAEMON_HOME/cosmovisor/upgrades/v2.0.0/bin/
.
As x/gov
module receives the proposal through a message server, it waits for the proposal to be passed through voting.
// EndBlocker called every block, process inflation, update validator set.
func EndBlocker(ctx sdk.Context, keeper keeper.Keeper) {
keeper.IterateActiveProposalsQueue(ctx, ctx.BlockHeader().Time, func(proposal types.Proposal) bool {
passes, burnDeposits, tallyResults := keeper.Tally(ctx, proposal)
...
if passes {
handler := keeper.Router().GetRoute(proposal.ProposalRoute())
cacheCtx, writeCache := ctx.CacheContext()
err := handler(cacheCtx, proposal.GetContent())
if err == nil {
...
// write state to the underlying multi-store
writeCache()
}
...
}
...
}
}
https://github.com/cosmos/cosmos-sdk/blob/main/x/gov/abci.go#L15-L130
If it passes, the proposal router routes the content of the SoftwareUpgrade proposal to the x/upgrade
module. The proposal router knows where this proposal should be routed. We usually setup a proposal router in app/app.go
before injecting x/gov
module into the application.
Now the proposal is passed to the x/upgrade
module. x/upgrade
module gets the Plan
data and saves it in its state.
func NewSimApp(...) *SimApp {
...
// register the proposal types
govRouter := govtypes.NewRouter()
govRouter.AddRoute(govtypes.RouterKey, govtypes.ProposalHandler).
AddRoute(upgradetypes.RouterKey, upgrade.NewSoftwareUpgradeProposalHandler(app.UpgradeKeeper))
govKeeper := govkeeper.NewKeeper(
appCodec, keys[govtypes.StoreKey], app.GetSubspace(govtypes.ModuleName), app.AccountKeeper, app.BankKeeper,
&stakingKeeper, govRouter,
)
...
}
x/upgrade
x/gov
module helps x/upgrade
to know the Plan
of software upgrade: when (block height) and what (the name of version). But still don’t know how to upgrade for each module. An application consists of modules. Each module functionality makes up the whole functionalities of the application. Upgrading an application (i.e. chain) usually means upgrading some modules. There might be some bugs, improvements, or adjustments to parameters.
So we need to define UpgradeHandler for the version we planned to upgrade. By registering the UpgradeHandler, x/upgrade
modules know how to upgrade for that version. UpgradeHandler signature looks as follows:
// UpgradeHandler specifies the type of function that is called when an upgrade
// is applied.
//
// `fromVM` is a VersionMap of moduleName to fromVersion (unit64), where
// fromVersion denotes the version from which we should migrate the module, the
// target version being the module's latest version in the return VersionMap,
// let's call it `toVM`.
//
// `fromVM` is retrieved from x/upgrade's store, whereas `toVM` is chosen
// arbitrarily by the app developer (and persisted to x/upgrade's store right
// after the upgrade handler runs). In general, `toVM` should map all modules
// to their latest ConsensusVersion so that x/upgrade can track each module's
// latest ConsensusVersion; `fromVM` can be left as-is, but can also be
// modified inside the upgrade handler, e.g. to skip running InitGenesis or
// migrations for certain modules when calling the `module.Manager#RunMigrations`
// function.
//
// Please also refer to docs/core/upgrade.md for more information.
type UpgradeHandler func(ctx sdk.Context, plan Plan, fromVM module.VersionMap) (module.VersionMap, error)
https://github.com/cosmos/cosmos-sdk/blob/main/x/upgrade/types/handler.go#L26
// VersionMap is a map of moduleName -> version
type VersionMap map[string]uint64
We’ve seen Plan
structure already. VersionMap is a map with the key of moduleName and value with the ConsensusVersion. ConsensusVersion is the module scoped version. So each module has a different ConsensusVersion. If the application needs to update x/foo
module only, we need to increase the ConsensusVersion of x/foo
module.
x/upgrade
manage the state of VersionMap in addition to Plan
. Every time the upgrade happens, x/upgrade
module injects the current VersionMap into the UpgradeHandler.
Then how UpgradeHandler for a specific version is invoked?
For the old version of the application, it just panic and self-killed its process. Every time the block is created x/upgrade
module checks the Plan
whether it’s time to run the upgrade procedure. It is normal that the old version application to panic and be killed when reaches the target block height because it has not UpgradeHandler. The old version application just writes the Plan data into the $DAEMON_HOME/data/upgrade-info.json
file and exits the process when reaches the block height. So don’t panic about the panic message of the application.
As we talked about before, Comovisor watches the $DAEMON_HOME/data/upgrade-info.json
file and updates the symbolic link to reference the new version of binary. And run the new version of the application. So UpgradeHandler should be defined in the new version of the application code, not in the old version.
Before the new version application is completely bootstrapped. it runs its UpgradeHandler. x/upgrade
module finds the upgradeHandlers
by the upgrade name (as I said before upgrade name is important) and then runs it. After that update the VersionMap with the return value of UpgradeHandler.
func (am AppModule) BeginBlock(ctx sdk.Context, req abci.RequestBeginBlock) {
BeginBlocker(am.keeper, ctx, req)
}
func BeginBlocker(k keeper.Keeper, ctx sdk.Context, _ abci.RequestBeginBlock) {
plan, found := k.GetUpgradePlan(ctx)
...
if !found {
return
}
// Prepare shutdown if we don't have an upgrade handler for this upgrade name (meaning this software is out of date)
if !k.HasHandler(plan.Name) {
// Write the upgrade info to disk. The UpgradeStoreLoader uses this info to perform or skip
// store migrations.
err := k.DumpUpgradeInfoWithInfoToDisk(ctx.BlockHeight(), plan.Name, plan.Info)
if err != nil {
panic(fmt.Errorf("unable to write upgrade info to filesystem: %s", err.Error()))
}
upgradeMsg := BuildUpgradeNeededMsg(plan)
panic(upgradeMsg)
}
// To make sure clear upgrade is executed at the same block
if plan.ShouldExecute(ctx) {
...
k.ApplyUpgrade(ctx, plan)
return
}
}
// ApplyUpgrade will execute the handler associated with the Plan and mark the plan as done.
func (k Keeper) ApplyUpgrade(ctx sdk.Context, plan types.Plan) {
handler := k.upgradeHandlers[plan.Name]
if handler == nil {
panic("ApplyUpgrade should never be called without first checking HasHandler")
}
updatedVM, err := handler(ctx, plan, k.GetModuleVersionMap(ctx))
if err != nil {
panic(err)
}
k.SetModuleVersionMap(ctx, updatedVM)
}
https://github.com/cosmos/cosmos-sdk/blob/55054282d2/x/upgrade/abci.go#L23-L95
UpgradeHandler
Now, let’s look at how UpgradeHandler can be implemented. There are various ways to structure the UpgradeHandler but I think Evmos way is neat.
Before the code, we should think about the upgrade scenarios: when we need the upgrade.
- We need to upgrade the module we depend on, the module we depend on means the module we depend on such as
x/bank
on Cosmos-SDK orx/evm
on Ethermint. - We need to upgrade the custom module, custom module means the module we developed for the application.
- We need to add a new module.
The ways to achieve each goal are slightly different.
Let’s look at the package structure first:
. # application project root
├── app
│ ├── upgrades
│ │ ├── v2
│ │ │ ├── constants.go
│ │ │ └── upgrades.go
│ │ └── ...
│ ├── app.go
│ └── upgrades.go
└── ...
Under the app
directory there’s upgrades/
directory and under the upgrades/
there are upgrading codes for the specific version. It is ok not to match the name of the directory for the specific version with the upgrade name we proposed before.
app/upgrades.go
file contains the aggregate of upgrades/
package codes and app/app.go
uses the app/upgrades.go
code.
The abstract format of UpgradeHandler and setup codes are as follows:
package v2 // constants.go
const (
UpgradeName = "v2.0.0"
...
)
app/upgrades/v2/constants.go
contains the value of such: upgrade name, Testnet/Mainnet upgrade block height. For the upgrade name, it must match the name we submitted SoftwareUpgrade proposal before.
package v2 // upgrades.go
func CreateUpgradeHandler(
mm *module.Manager,
configurator module.Configurator,
...
) upgradetypes.UpgradeHandler {
return func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
...
return mm.RunMigrations(ctx, configurator, vm)
}
}
app/upgrades/v2/upgrades.go
defines the UpgradeHandler which runs when Cosmovisor executes the new version of the application binary. And UpgradeHandler must end with return mm.RunMigrations(ctx, configurator, vm)
. Inside ModuleManager, it triggers each module’s MigrationHandler. We’ll see the process later.
package app // upgrades.go
func (app *App) setupUpgradeHandlers(
configurator module.Configurator,
...
) {
app.UpgradeKeeper.SetUpgradeHandler(
v2.UpgradeName,
v2.CreateUpgradeHandler(app.mm, configurator, ...),
)
...
upgradeInfo, err := app.UpgradeKeeper.ReadUpgradeInfoFromDisk()
if err != nil {
panic(fmt.Errorf("failed to read upgrade info from disk: %w", err))
}
if app.UpgradeKeeper.IsSkipHeight(upgradeInfo.Height) {
return
}
var storeUpgrades *storetypes.StoreUpgrades
switch upgradeInfo.Name {
case v2.UpgradeName:
...
}
if storeUpgrades != nil {
app.SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height, storeUpgrades))
}
}
On the app/upgrades.go
, we need to register UpgradeHandler to x/upgrade
module. x/upgrade
module tries to find the UpgradeHandler for the ‘upgrade name’ we saw it several times and if it exists, run it in the BeginBlock steps.
And if we want to add new modules, instead of updating existing modules, we need to call app.UpgradeStoreLoader
. This also will be explained below.
package app // app.go
func NewSimApp(...) *SimApp {
// setupUpgradeHandlers should be called before `LoadLatestVersion()`
// because StoreLoad is sealed after that
app.setupUpgradeHandlers(configurator, app.BankKeeper)
if loadLatest {
if err := app.LoadLatestVersion(); err != nil {
tmos.Exit(err.Error())
}
}
...
return app
}
On the app/app.go
file, we need to call the method setupUpgradeHandlers
we defined previously in the constructor of the application. This method must be called before app.LoadLatestVersion()
, because that call sealed the options and parameters of the app and we cannot modify those anymore.








Scenario#1: Upgrading Existing Module
Let’s think about the first scenario: upgrading existing modules.
You may have noticed that we cannot update the codes of existing modules. For example, x/bank
in Comos-SDK we cannot update its codes directly. So the extent we can update is parameters and states (not schema). For example, we can update the sendEnabled
parameters for specific denom or add the denom metadata using x/bank
SetDenomMetaData()
interface. we cannot update the internal state schema of denom metadata.
For updating the existing module, we need to inject related keeper into the CreateUpgradeHandler()
. Let’s say we want to add x/bank
denom metadata. We need x/bank
keeper to update its state. So the code would be as follows:
package v2 // upgrades.go
var DenomMetadata = banktypes.Metadata{...}
func CreateUpgradeHandler(
mm *module.Manager,
configurator module.Configurator,
bk bankkeeper.Keeper,
) upgradetypes.UpgradeHandler {
return func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
UpdateDenomMetadata(ctx, bk)
return mm.RunMigrations(ctx, configurator, vm)
}
}
func UpdateDenomMetadata(ctx sdk.Context, bk bankkeeper.Keeper) {
bk.SetDenomMetaData(ctx, DenomMetadata)
}
Scenario#2: Upgrade Custom Module
Second scenario: upgrading custom modules.
In this case, we can update the code easily. In other words, in addition to updating parameters and data, we can update the state schema. But for upgrading the schema, we need to do additional work for the modules that migrate the schema.
- Backup old version state schema into the other directory. Most projects save the old version schema into
x/<module>/migrations/<version>/
directory. Although the application is a new version, we still need the old version of the state schema. Because we need to migrate the old version states into the new version schema. - Write migration handler for the specific version.
Below is a workflow for upgrading the custom modules.
Before generating code with the new version of Protobuf copy the current Protobuf generated files into the x/<module>/migrations/<version>/
directory. For example, if x/foo
module we developed needs to update its state schema, first place current state schema code files like params.pb.go
into the x/foo/migrations/v1/types/params.pb.go
. Then generate codes with the new Protobuf files which will be placed in x/foo/types/
directory.
If we need to migrate the old state into a new state with a different schema. We need to write the migration handler and register it to the configurator. These migration handler codes usually put into x/<module>/keepers/migrations.go
file and concrete logic of migration is placed under the x/<module>/migrations/<new-version>/migrations.go
.
For example, if we are going to migrate x/foo
module from version 1 to 2, place Migrator into x/foo/keeper/migrations.go
and write concrete MigrationHandler codes into the x/foo/migrations/v2/migrations.go
as below:
x
├── foo
│ ├── keepers
│ │ └── migrations.go
│ ├── migrations
│ │ ├── v1
│ │ │ ├── tx.pb.go
│ │ │ └── ...
│ │ └── v2
│ │ └── migrations.go
│ └── module.go
└── ...
In this example code, x/foo
module updates its parameter and state schema. Migrator uses Keeper to get the state of x/foo. Concrete update logic is defined in x/foo/migrations/v2/migrations.go
package keeper // migrations.go
type Migrator struct {
keeper Keeper
}
func NewMigrator(keeper Keeper) Migrator {
return Migrator{keeper: keeper}
}
func (m Migrator) Migrate1to2(ctx sdk.Context) error {
if err := v2.UpdateParams(ctx, &m.keeper.paramstore); err != nil {
return err
}
return v2.MigrateStore(ctx, m.keeper.storeKey, m.keeper.cdc)
}
x/foo/migrations/v2
package uses both x/foo/types
and x/foo/migrations/v1
packages for migrating state into a new version. For updating the parameter, it simply gets the reference of Subspace of x/foo
module and set the value of the parameter key.
In the case of MigrateStore()
, for example, we can implement migration logic as below: first retrieve the key, value of state from the store and save it with the new schema and delete the old state. This is one of the ways to implement it.
package v2 // migrations.go
func UpdateParams(ctx sdk.Context, paramStore *paramtypes.Subspace) error {
...
paramStore.Set(ctx, types.KeyValue, "foobar")
return nil
}
func MigrateStore(ctx sdk.Context, storeKey storetypes.StoreKey, cdc codec.BinaryCodec) error {
store := ctx.KVStore(storeKey)
return migrateValuesWithPrefix(store, cdc)
}
func migrateValuesWithPrefix(store sdk.KVStore, cdc codec.BinaryCodec) error {
oldStoreIter := store.Iterator(nil, nil)
for ; oldStoreIter.Valid(); oldStoreIter.Next() {
oldKey := oldStoreIter.Key()
oldVal := store.Get(oldKey)
newKey, newVal := migrateValue(cdc, oldKey, oldVal)
store.Set(newKey, newVal)
store.Delete(oldKey) // Delete old key, value
}
return nil
}
func migrateValue(cdc codec.BinaryCodec, oldKey []byte, oldVal []byte) (newKey []byte, newVal []byte) {
valWithMemo := types.ValueWithMemo{
Value: string(oldVal),
Memo: "HELLO", // Default memo as 'HELLO'
}
newKey = types.GetValueKey(string(oldKey))
newVal = cdc.MustMarshal(&valWithMemo)
return
}
Finally, register the migration handler on the Configurator using RegisterMigration(moduleName string, forVersion uint64, handler MigrationHandler) error
method. Once the module registers the migration handler, ModuleManager runs the MigrationHandler using Configurator inside of UpgradeHandler.
package foo
// RegisterServices registers a GRPC query service to respond to the
// module-specific GRPC queries.
func (am AppModule) RegisterServices(cfg module.Configurator) {
...
migrator := keeper.NewMigrator(am.keeper)
// register v1 -> v2 migration
if err := cfg.RegisterMigration(types.ModuleName, 1, migrator.Migrate1to2); err != nil {
panic(fmt.Errorf("failed to migrate %s to v2: %w", types.ModuleName, err))
}
}
// ConsensusVersion implements ConsensusVersion.
func (AppModule) ConsensusVersion() uint64 {
return 2
}
You can find that MigrationHandler is registered via cfg.RegisterMigration()
with the forVersion
of 1. And ConsensusVersion()
method of x/foo
module returns 2.
VersionMap version of this module works as fromVersion
and return value of ConsensusVersion()
works as toVersion
when running loop of upgrading module inside ModuleManager
. Range for fromVersion
as inclusive, toVersion
as exclusive. And inside each loop, ModuleManger
tries to find the MigrationHandler for the forVersion
. For example, if the current version of foo in the VersionMap is 1 and the return value of ConsensusVersion()
of foo is 3. ModuleManager
tries to find MigrationHandlers for versions 1 and 2.
In the code above, because the current version in the VersionMap is 1 and the return value of ConsensusVersion()
is 2. We only need the MigrationHandler for the forVersion
of 1.
func (m Manager) RunMigrations(ctx sdk.Context, cfg Configurator, fromVM VersionMap) (VersionMap, error) {
...
updatedVM := VersionMap{}
for _, moduleName := range modules {
module := m.Modules[moduleName]
fromVersion, exists := fromVM[moduleName]
toVersion := module.ConsensusVersion()
if exists {
err := c.runModuleMigrations(ctx, moduleName, fromVersion, toVersion)
if err != nil {
return nil, err
}
}
...
updatedVM[moduleName] = toVersion
}
return updatedVM, nil
}
https://github.com/cosmos/cosmos-sdk/blob/55054282d2/types/module/module.go#L371-L462
Scenario#3: Adding New Module
Third scenario: adding a new module to our application.
In this case, we don’t need to write the migration handler as Scenario#2. Because we still don’t have states for the new module. Instead, we need to update the store for our new module. This goal is achieved by calling UpgradeStoreLoader
with the StoreUpgrades
.
package app // upgrades.go
func (app *App) setupUpgradeHandlers(
configurator module.Configurator,
...
) {
app.UpgradeKeeper.SetUpgradeHandler(
v2.UpgradeName,
v2.CreateUpgradeHandler(app.mm, configurator, ...),
)
...
var storeUpgrades *storetypes.StoreUpgrades
switch upgradeInfo.Name {
case v2.UpgradeName:
storeUpgrades = &storetypes.StoreUpgrades{
Added: []string{newmoduletypes.StoreKey},
}
}
if storeUpgrades != nil {
app.SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height, storeUpgrades))
}
}
In the code above, we’ve specified newmoduletypes.StoreKey
the Added
field inside StoreUpgrades
struct. And that values are passed into UpgradeStoreLoader
. By doing this, we can allocate a store for the x/newmodule
in the multi-store of application. And make x/newmodule
is correctly added to the app. ‘Correctly added’ means the basic things we need to do to add a module to the application such as adding Subspace of the module, adding a module to the ModuleManager, etc.
Things to do for the upgrade as a developer
Now from the application developer perspective, what codes do we need to write for upgrading the chain successfully? Let’s list TODOs from the high level.
TODO#1: Write UpgradeHandler
First, we need to write UpgradeHandler and register it to x/upgrade
module. If we want to update modules we do not manage such as x/bank
that Cosmos-SDK provides, we should inject x/bank
module into the UpgradeHandler. And we need to call ModuleManager’s RunMigration method.
TODO#2: Write MigrationHandler
If we want to upgrade the existing module that we manage, we need to place the old version of types into the other directory, update proto files for the new version then write the migration logic from the old states into the new scheme of state. After writing MigrationHandler, we need to register it via Configurator.
ConsensusVersion()
of the module that needs to be updated
TODO#3: Update ConsensusVersion simply puts the module scoped version. If we update the ConsensusVersion for the module, ModuleManager detects it and runs the MigrationHandler for the version. And that will be the handler we wrote in TODO#2.
Those TODOs might be slightly different by the upgrade scenarios:
- We need to upgrade the module we depend on, the module we depend on means the module we depend on such as
x/bank
on Cosmos-SDK orx/evm
on Ethermint. - We need to upgrade the custom module, custom module means the module we developed for the application.
- We need to add a new module.
Keep in mind that UpgradeHandler and MigrationHandler should be written in the new version of the application code, not in the old version.




Cosmos Blockchain Upgrade Demo
This is a simple blockchain upgrade demo. The script used in the demo is explained below. You can find the module state schema is changed after the upgrade. The code of migration is similar to the code in the ‘Component Details’ section.
Sample Scripts for Testing Upgrade on the Local
Before running the test, we need to download ignite, Cosmovisor binary.
setup-node.sh
From the terminal, if we run the setup-node.sh
, we build the application binary of versions v1 and v2. And copy those into Cosmovisor temporary directory. After initializing the data for the node with ignite, copy Cosmovisor temporary directory into the $DAEMON_HOME
. Finally, run the application via cosmovisor
command.
#!/bin/bash
export DAEMON_NAME=simappd
export SIMAPPD_V1_HOME=/path/to/simappd-v1
export SIMAPPD_V2_HOME=/path/to/simappd-v2
export DAEMON_HOME=/path/to/simappd-home
# Tmp cosmovisor directory. In this directory, we are going to
# setup Cosmovisor directory structures and copy this directory
# into $DAEMON_HOME
export TMP_COSMOVISOR_DIR=/path/to/tmp-cosmovisor
# update if needed
export DAEMON_ALLOW_DOWNLOAD_BINARIES=false
export DAEMON_RESTART_AFTER_UPGRADE=true
# setup cored binaries
cd $SIMAPPD_V1_HOME
make build
cp ./build/simappd $TMP_COSMOVISOR_DIR/genesis/bin/
cd $SIMAPPD_V2_HOME
make build
cp ./build/simappd $TMP_COSMOVISOR_DIR/upgrades/v2.0.0/bin/
# setup cored data
cd $SIMAPPD_V1_HOME
ignite chain init --home $DAEMON_HOME
cp -R $TMP_COSMOVISOR_DIR $DAEMON_HOME/
cosmovisor run start --home $DAEMON_HOME
upgrade-proposal.sh
After running setup-node.sh
script successfully, open another terminal window, then run the upgrade-proposal.sh
to send the SoftwareUpgrade proposal to upgrade to v2.0.0
on the 40th block.
#!/bin/bash
export SIMAPPD_V1_HOME=/path/to/simappd-v1
export SIMAPP_V1_BIN=$SIMAPPD_V1_HOME/cosmovisor/genesis/bin/cored
export DAEMON_HOME=/path/to/simappd-home
# Send SoftwareUpgrade proposal - Upgrade Name: v2.0.0
$SIMAPP_V1_BIN tx gov submit-proposal software-upgrade v2.0.0 --title v2.0.0 --description v2.0.0 --upgrade-height 40 --from validator1 --yes --home $DAEMON_HOME --chain-id app_9000-1
# Deposit for the proposal - Proposal ID: 1
$SIMAPP_V1_BIN tx gov deposit 1 10000000atoken --from validator1 --yes --home $DAEMON_HOME --chain-id app_9000-1
# Vote for the proposal
$SIMAPP_V1_BIN tx gov vote 1 yes --from validator1 --yes --home $DAEMON_HOME --chain-id app_9000-1
After blockchain has reached to 40th block, the v1 application panics and Cosmovisor re-run the application with the version of v2.
What’s Left/Next?
We’ve seen several scenarios to update/add modules on the application. But there are still left some edge cases. For example, we should handle the case when the UpgradeHandler not working correctly, so as MigrationHandler, how should we roll back temporarily, etc.
In this article, we’ve mainly seen the chain upgrade process from the technical part. But we need to address more about governance such as how the network going to propose the version upgrade to the community and vote.
For those who are curious about chain upgrades in the technical view, hope this post helps you understand more in detail.