Distributed Block Storage Design

Design framework for long term distributed storage of block chain.

Summary:

Persistent and highly available storage is key to reliability any block chain. In our proof of stake model master nodes have highest stake in network, making them obvious participants to store block chains. However in order to reduce storage requirements and reduce failure rates we have designed highly reliable distributed storage design over untrusted network.

Design Goals :

  • Blocks must be randomly distribution across Masters Nodes

  • Have enough replication and fault tolerance so prevent any data loss or data corruption

  • Enable quick integration of new masters to network.

  • Protect against bad actors and enable audit.

Terms:

MN : Master Node

Master ID : unique identifier for master node

RF : Replication factor (number of copies of block across network)

POOL : abstract concept to randomly but uniformly distribute blocks over network

NODE_IDX : Index of Master node within a pool

Above Fig is simple representation of how randomly yet efficiently we store blocks across various master nodes. Storage layer takes few Config inputs such as Master Set (Total num of masters in VK), Replication Factor

Algorithm works simply as bellow:

random_distributed_write (Master_set, RF, quorum)
Pre Conditions given : Given n Master node, r replication, quorum
Post Outcome : store r random copies across master node such that
- blocks are uniformly distributed
- blocks are geographically distributed
- There is enough replication to ensure data availability
- storage paterns can mutate over time
if master_set <= quorum
always write block locally
get master_id(key) -> from vkbook
total pools -> div(master set / replication factor)
node idx -> index of node in pool
evaluate_local_wr -> blocknum mod node idx
if current node == node idx
write -> true
update index table
else
update index table

Distributed persistent Write Work Flow

Block Input to Storage layer:

struct BlockData {
blockHash @0 :Text;
blockNum @1 :UInt32;
blockOwners @2 :List(Text);
prevBlockHash @3 :Text;
subBlocks @4 :List(SB.SubBlock);
}
struct SubBlock {
merkleRoot @0 :Text;
signatures @1 :List(Data);
merkleLeaves @2 :List(Text);
subBlockIdx @3 :UInt8;
inputHash @4 :Text;
transactions @5 :List(T.TransactionData);
}
struct TransactionData {
transaction @0 :Data;
status @1: Text;
state @2: Text;
contractType @3: UInt16;
}

Block Example

{
"_id" : ObjectId("5c3ca3b6538cda79d8a8c120"),
"blockHash" : "845ea9659cc7a0c9cabe42d8914b675f10686bdfdcca96e471ebf0c0e5bdb300",
"blockNum" : 9,
"blockOwners" : [ ],
"prevBlockHash" : "ca26a999d58dc984c20d58c9320fc06a614de8e71da96c8c6be797d0607e51b8",
"subBlocks" : [
{
"merkleRoot" : "7d2ac625766cc450cc10b8172ee48f88cbd85ebff5b468a072e2eb09df8e926b",
"signatures" : [
BinData(0,"eyJzaWduYXR1cmUiOiAiYTJmNzY2MWE1NjQ5YjkwMjgwNDk5NDFhOGZkMzdlNWJhZTE4ZDljOWNlZjZlM2ZmMjFhYzAwYTUwM2ZjZGJlM2U3YzkyZmNjYmVhN2I2ODZmOWM1NGQxMDFlNWRlNDVkN2Y3ZjEwMWFlMTBkMGM1NjU4MTQ3ZTZiNmZiOGQzMDkiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDEiLCAic2VuZGVyIjogIjE4MjZlMDAwNDE5N2Q3YmEwODU1MTc1MDUwMmM0ZjZmMjg2OWU4NjBlNjJkNDQzMTJhYTI0MDJhNWFmOGRlZDkifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiMTg5M2M3MDUxMjAxY2Q4Yjk5MjMzZTVkZWI0NDVhNzMyYWJmOWM2ZTk1Y2FkNDljMWFiODMzMWNlYWE2ZDc3MDJlYjE0OGVhOTM5OTRiMmVlMzc0YzdkMTQ4MzRiODNmMjI3ZWI3MTczOWU2NTY1OTBiMTQ5NTg4ZTVlNTQ3MDMiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDEiLCAic2VuZGVyIjogImNlNTkzMDYwMDgwZTQ3OTNjMjZhOTI2MGQ3YWU3YjZmZGFkZWYwMWFjOWFmZDkxMmRhZGQ2MGE3OWM1MmZjODAifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiYjg5MTEwODEwOWVhNTM2MzJmNzAyZmIwOWZmNjgwZGUwOGYxOTNlMzJkNTIwZjQzZGVlYTZiOTdjNmJmMDJjYjdmZGQ2M2M3N2U5YTA1MmRlNjY5MjU1ZDJkNjFmYjEwN2M4NTNmMGMzODNlMDkxOTQ5YjYwMGM4ODgyZDhhMGMiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDEiLCAic2VuZGVyIjogImRkNGU2YzkyNzZiNWExYzczNmQ5OTg4NTI5Nzc0OTg2MjcyYjk0MWE0YjZjMmZkZWQ2MGNkZGIxYmRjOWYwNGMifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiZjBkYjgzMDBjYmE3MTRhNzM5MDMxYzg2NDNhNTE0ODZmOGI5OGYyZmRkYWI4NjAxZTY2YTE2YTkwYjJhYTM4ZWM1ZjllYTM0OGRhOTQ3OGQwMTc0N2ZjYTEyMTIxZWUwYTI0ZTNlZGQ2YjEzOWI0OGRhZDM4YTcwNDIwZjFhMDIiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDEiLCAic2VuZGVyIjogImYyMWVlZDI1ZDUzZTk0NzM1ZDJjNTYzNjAxODFiMzQwNTA0MWEwNTFhMjUxZTU3OTk2M2ZkNDY0NjI0N2M2MzQifQ==")
],
"merkleLeaves" : [ ],
"subBlockIdx" : 0,
"inputHash" : "7d2ac625766cc450cc10b8172ee48f88cbd85ebff5b468a072e2eb09df8e926b",
"transactions" : [ ]
},
{
"merkleRoot" : "14b4aaa456e168b711ed113ee302303c3237bed1924ae4a430bd3a5743469639",
"signatures" : [
BinData(0,"eyJzaWduYXR1cmUiOiAiODA0NDA0MDY3N2M0Y2Y3OTI1ODg1MDI4OTdiZmZkOWNkYzQ2NmNiMmNkODA5MzlkNDBiZjliMDk2OGIyNzFmOTE1ZjE3NjJhZDJmNWI2MTM1OTE5NDQ3NjMxN2E0YzNkZjAwNjU5ZTYzODdjYTA2NzJlZDgzODAxNDJiNTg2MGQiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogIjE4MjZlMDAwNDE5N2Q3YmEwODU1MTc1MDUwMmM0ZjZmMjg2OWU4NjBlNjJkNDQzMTJhYTI0MDJhNWFmOGRlZDkifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiOGVhMDA3YjE4YWI3OGUyODBmN2I4OTJhYjRmZGI0MWRmODhkYzIzY2EyOTQwODJiZDlmM2M0MGUxNmU0NTY0NzgzZTlmZDM5YjU4OTNhYTdhZGJiODEwMDQ3MWExNzdlOTg0OTRmZTI0OTM4ZmU1N2Q0MmY4NDFmYzJmMTdlMGYiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImRkNGU2YzkyNzZiNWExYzczNmQ5OTg4NTI5Nzc0OTg2MjcyYjk0MWE0YjZjMmZkZWQ2MGNkZGIxYmRjOWYwNGMifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiZjkyYTZjNGNlMTgxODQ2ZDlkODY1OGIxNjVlZGFlNTgxNTRhM2E3NzQ4NDE2ZGQ5MTAxYjM3OGU3ZTE4YjY4ODRiYTcyODY3Mzg0YzlhNTQwNDY3NzA0ZDAyMjQ2ZjQ2MjhmZDRkOWY0NDhkNWRjNTYxMjg4NGNiNjVkZjkzMDciLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImNlNTkzMDYwMDgwZTQ3OTNjMjZhOTI2MGQ3YWU3YjZmZGFkZWYwMWFjOWFmZDkxMmRhZGQ2MGE3OWM1MmZjODAifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiZmZmMmM3MmQ3M2M1OTlmMWUwM2EwOTYzOWI2YjI5OTQwZWE3ZDQ2NzdhNzM2MmZlOGU5MDNlN2JlYjhmOTBhMzkwYmQ4Nzc2YmQzMWQwMTZkMTg2MzkwYmM5Mjc2OGZlY2ZmMDE4ZTk3ODE3NjdiNjQ0MDc0NmQwYzc2MmU0MGQiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImYyMWVlZDI1ZDUzZTk0NzM1ZDJjNTYzNjAxODFiMzQwNTA0MWEwNTFhMjUxZTU3OTk2M2ZkNDY0NjI0N2M2MzQifQ==")
],
"merkleLeaves" : [ ],
"subBlockIdx" : 1,
"inputHash" : "14b4aaa456e168b711ed113ee302303c3237bed1924ae4a430bd3a5743469639",
"transactions" : [ ]
},
{
"merkleRoot" : "eecf2c7f95c3ee74628a503a1fefe4b78c3d4983b3394ab985e8679a672d1dcb",
"signatures" : [
BinData(0,"eyJzaWduYXR1cmUiOiAiMzJkZWZhYTRmNzJjZDIyZjhiOTk3MDBlMGVhNGQ5YzdjMTg2ZjE2ZTUyMjAwZTlmNTc0ZDJhNWRiNWYxMmFlMTY5MDAwYzY5OTc0Mjk4MzdmYzE2ZTk1MzEyMzQ1M2M1ZmEyNzg4M2E3ZDljMzVmYWZhN2QwMmM1Nzg3MzYwMDkiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImNlNTkzMDYwMDgwZTQ3OTNjMjZhOTI2MGQ3YWU3YjZmZGFkZWYwMWFjOWFmZDkxMmRhZGQ2MGE3OWM1MmZjODAifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiN2Y1MjQzZThhMTQzOGFiYmM5OTUxNTYzNzYyMjM4NTQ0NzA0YzhkYjcxYjE2ZDg4ZmI2MWEzNjdlYjZjYTU3M2MwMjA2YmI2Mzc1ZjU4Y2RlMDA0MmUyNTFlNWMwN2NmYTA2Y2RlODQ5YWMxMWU4MjAwNzM3YmM4ZjZlNjg1MGIiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogIjE4MjZlMDAwNDE5N2Q3YmEwODU1MTc1MDUwMmM0ZjZmMjg2OWU4NjBlNjJkNDQzMTJhYTI0MDJhNWFmOGRlZDkifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiYTk5NjYxYTMzYWI5MDQzODljZGIxNWRlZDkyMDVkZmM4NDUzOWI2MDRlYTFlY2JmYTc2NGFlZjE5ZmFhNDNhY2YyOTcxOGVmZjY1MTRkZDlhOTc3NmMyNGMyOWU3ZjhiNjY1OWYzNTdiZTY5MjU4NmU5Yzk4NGY2ZDY1ZGZiMDgiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImRkNGU2YzkyNzZiNWExYzczNmQ5OTg4NTI5Nzc0OTg2MjcyYjk0MWE0YjZjMmZkZWQ2MGNkZGIxYmRjOWYwNGMifQ=="),
BinData(0,"eyJzaWduYXR1cmUiOiAiMWU3NTQ3M2JmZWUzNzkwNGZlMDFkMmQ3Y2Y4YjlmZWFiOTQ3YzFmMTcyNjAwM2U2YzZlNGQ1ODZmNmQ1YmQ0ODAwNDFiMjYzOGNkODAyYTVlOWFjMjUzMWVkN2RmYTQ4MzU1N2NmYTNiMmMxNTU2OGJjNDdiZjkyZDM2YWU0MDUiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIiLCAic2VuZGVyIjogImYyMWVlZDI1ZDUzZTk0NzM1ZDJjNTYzNjAxODFiMzQwNTA0MWEwNTFhMjUxZTU3OTk2M2ZkNDY0NjI0N2M2MzQifQ==")
],
"merkleLeaves" : [ ],
"subBlockIdx" : 2,
"inputHash" : "eecf2c7f95c3ee74628a503a1fefe4b78c3d4983b3394ab985e8679a672d1dcb",
"transactions" : [ ]
},
{
"merkleRoot" : "65e427dbc5685e012659cb44fe4350bbe03bb160c8735b362705052000f81dae",
"signatures" : [
BinData(0,"eyJzaWduYXR1cmUiOiAiMjAyMzM5ZTgzMDM3ZDMyYWUzMTczY2MxNjc2N2Y0NjZiZjdlMGQ4YzdiOTYxMDcwZTZlMWEyMDUwNGY2Nzk3NjhjNmYwMzNkOThlYmE2NDJjZTZmNWRmYWUwYzcyYjljMjYzNmMyMzEwMmNlNDc0NmJhZWVjMGJmY2M4YmZmMDciLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIuNDY5MjU3IiwgInNlbmRlciI6ICJjZTU5MzA2MDA4MGU0NzkzYzI2YTkyNjBkN2FlN2I2ZmRhZGVmMDFhYzlhZmQ5MTJkYWRkNjBhNzljNTJmYzgwIn0="),
BinData(0,"eyJzaWduYXR1cmUiOiAiYjAyNzhiZWZiMTc2NjhjNGIxMjFmMGY1Nzk0Zjc5NjY2MjkyZmNlNmRiNjdmM2RlZGFlMmQxMzEzZDc0ODU2ODE0YjJlNmI1MDJiMzg1ZTI0NmQ0MWQ0MTM5MWI3Mzg4MTUyODk0MTg3ZTQ3MTU0OGU0ODUzZGViZTE2NDliMDgiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIuNDcxODA4NyIsICJzZW5kZXIiOiAiZGQ0ZTZjOTI3NmI1YTFjNzM2ZDk5ODg1Mjk3NzQ5ODYyNzJiOTQxYTRiNmMyZmRlZDYwY2RkYjFiZGM5ZjA0YyJ9"),
BinData(0,"eyJzaWduYXR1cmUiOiAiZmQ1NjQzNmJjYjliYmFmYzYyYjA2YjE4Zjc3MDMxNzcxZjE2Y2IwYzljNGZiMGFjOGUwZmJlNGM4MTBkOWFjNjU0YzZkZWEyZGEzOWUwYjcxODlhZmY0ZmQ5MTA0NTZjYjFhOTM1ZGQyODgzN2QwMGM0ZjhhYmYyMjNmMTRiMDQiLCAidGltZXN0YW1wIjogIjE1NDc0Nzc5NDIuNDcxOTQ5MyIsICJzZW5kZXIiOiAiMTgyNmUwMDA0MTk3ZDdiYTA4NTUxNzUwNTAyYzRmNmYyODY5ZTg2MGU2MmQ0NDMxMmFhMjQwMmE1YWY4ZGVkOSJ9")
],
"merkleLeaves" : [
"65e427dbc5685e012659cb44fe4350bbe03bb160c8735b362705052000f81dae"
],
"subBlockIdx" : 3,
"inputHash" : "41991848a31cd3c1f940dd7943aea547f1630e93fd26222828b1491de806c6ac",
"transactions" : [
{
"transaction" : BinData(0,"EFhAAlEEAQIxXYIPAAAxRQIBMQECBP8wZDA4ZDY4YxNlZDY2NzQyODU0OTdiYjczZTFkOGFkNGU3MmJmMGJmMTAyMmJhMmMwODRjZjZkZmUzNGRmMDY5Y2JmZjA0YjNlOTYyMWNmYTk4MzNiOGY5YzI5ZDRiODE2Yzk2MWFmY2JlNDI4MGRiNTA4ODViOTk2NmJjM2Y4MGE3NzU4NzgzNDVhNjc0YTQ4MzAzNDQ2Njc3MDY1NjQ1NRA9UAEFA7gLMX0KAjE1CgQRIUoRJUpAAREBJ0EIAhGFGlGIAgERgTpRsAIB/2N1cnJlbmN5AAAA/3RyYW5zZmVyAAAA/2ExMDM3MTU5DzE0YTdhYWU4ZGQ4ZmRkYmE5NDVhYjYzYTE2OWRmZTZlMzdmNzliNGE1OGJjZjg1YmZkNjgxNjk0QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQgAA/2ExMDM3MTU5BzE0YTdhYWU4ZGQ4ZmRkYmE5NDVhYjYzYTE2OWRmZTZlMzdmNzliNGE1OGJjZjg1YmZkNjgxNjk0AAADdG8/YW1vdW50AQ0AADEBCgL/ZWQxZGE3MTMHYTZlNTZjM2I2NmM0NmYxNmYxOWI4ZGUyMmYyYmQxODVmMmM2MjlhOTIwNjY5MmEwNDBhN2IzOTIAAAEKAAARAUJ/MTAwMDAwMA=="),
"status" : "FAIL -- 'NoneType' object has no attribute 'working_db'",
"state" : "",
"contractType" : 37924
}
]
}
]
}