fix: recover from panics in LoadSectorStates #1115

frrist · 2023-01-19T17:44:34Z

related to Lotus daemon being killed and causing websocket errors lotus#10050

- related to filecoin-project/lotus#10050

birdychang

Thanks!

davidgasquez · 2023-01-23T16:59:56Z

I'm having some issues with this PR. This is the lily.log.

{"level":"info","ts":"2023-01-22T12:48:26.438Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"verified_registry_verif
ier","status":"OK","duration":9.576490392}
{"level":"error","ts":"2023-01-22T12:48:26.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01944708","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01944708.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"miner_sector_post","height":"2399050","reporter":"walk_1674391696","d
uration":9.904742493}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"miner_sector_post","sta
tus":"OK","duration":9.904740094}
{"level":"error","ts":"2023-01-22T12:48:27.338Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:27.395Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915730","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01915730.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.544Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01283930","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01283930.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.591Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01931245","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01931245.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.827Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f014409","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014409.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change typ
e modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"data_cap_balance","height":"2399050","reporter":"walk_1674391696","du
ration":10.973147105}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"data_cap_balance","stat
us":"OK","duration":10.973145203}
{"level":"error","ts":"2023-01-22T12:48:28.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01907880","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01907880.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:28.565Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0396473","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0396473.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01915033.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.410Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0126037","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0126037.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.493Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01983523.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.319Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0215497","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0215497.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.414Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01964132","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01964132.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01172955.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.794Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0883206","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0883206.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.813Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f010088","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f010088.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change type
 modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.342Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157027","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157027.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01790067.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.683Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157018","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157018.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.867Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01154901.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.016Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01858258","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01858258.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0413563.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: context canceled"}

The daemon is kept running but the walk is not completing. The list command outputs this:

[
        {
                "ID": 1,
                "Name": "walk_1674391696",
                "Type": "walk",
                "Error": "",
                "Tasks": [
                        "block_header",
                        "block_parent",
                        "drand_block_entrie",
                        "data_cap_balance",
                        "miner_beneficiary",
                        "miner_sector_deal",
                        "miner_sector_infos_v7",
                        "miner_sector_infos",
                        "miner_sector_post",
                        "miner_pre_commit_info",
                        "miner_sector_event",
                        "miner_current_deadline_info",
                        "miner_fee_debt",
                        "miner_locked_fund",
                        "miner_info",
                        "market_deal_proposal",
                        "market_deal_state",
                        "message",
                        "block_message",
                        "receipt",
                        "message_gas_economy",
                        "parsed_message",
                        "internal_messages",
                        "internal_parsed_messages",
                        "vm_messages",
                        "multisig_transaction",
                        "chain_power",
                        "power_actor_claim",
                        "chain_reward",
                        "actor",
                        "actor_state",
                        "id_addresses",
                        "derived_gas_outputs",
                        "chain_economics",
                        "chain_consensus",
                        "multisig_approvals",
                        "verified_registry_verifier",
                        "verified_registry_verified_client"
                ],
                "Running": true,
                "RestartOnFailure": false,
                "RestartOnCompletion": false,
                "RestartDelay": 0,
                "Params": {
                        "maxHeight": "2399050",
                        "minHeight": "2396162",
                        "storage": "CSV",
                        "window": "0s"
                },
                "StartedAt": "2023-01-22T12:48:16.859521259Z",
                "EndedAt": "0001-01-01T00:00:00Z"
        }
]

frrist · 2023-01-23T20:51:41Z

@davidgasquez is it failing for all epochs or just some?

davidgasquez · 2023-01-23T21:16:41Z

I think is just some (will check tomorrow when I'm back). That said, it happens in most of them and should be easy to reproduce in any recent partial archive.

davidgasquez · 2023-01-24T15:38:32Z

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"[email protected]/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"[email protected]/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

davidgasquez · 2023-01-24T15:38:35Z

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"[email protected]/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"[email protected]/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

- prevents panic

davidgasquez · 2023-01-25T08:59:33Z

Tested things locally and seems its working fine. I've got some errors but I think they're unrelated with this PR:

$ cat lily.log | grep "error" | jq .
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebarb7znoskcwvl6up2avtcu44keoi2c7cfiat7knyvgwqy3iahak"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecn3bzbox5h4x3viqwtgwvsz6glfmgm4tkxq2bvqdgddzzu3gx3ws"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebdm3jeqsoa74de7qthlxvcjjkvp62yxri7auv7ggwyemfllwntgi"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebrgzpswiwdqs5wvmw6zfrsvrdgeqpeozsbkhwuwgsswcfsdrdxto"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea7lqx55xxzdsp7use33hgeeh33tyry4m6zg3dsin6lznopk4yq3w"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebby6ag7qk743wti5bp6azdcffptrw5ws3u5gwa5c6kiepjt5ayqs"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecifg623mszaimdfbooxpj54etykqjy7sjec3xd2ycoflqn5akke2"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea4624zeqqzdrdwoe33ejxxz32zq5fr5ysmbuu3vofskgmegkx3qc"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceaoez3byuzqkblaa2gfj4zi2ubh74zj7m7jipkatbpjzoxdofmqvu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebmd7l7vevx56qkh2ehnugkwtmkysf4ryx3si4xwknanneqiybmhu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceabtnahyz77s6kzhccls6mjfv43y4snibrn4zrbhfirltwvh6wa3u"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacedjqujbo32cqfkdw3zvhqlswo4lhom4b5v6jnvd54oy3aps5jznle"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea5wktkeap5ur4czmrbx4xpwp4edfihrxm2n25adnclo3v6nakl4g"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceddzlla5vs4jxknncyoexprtm2hcn3xz37mrkfzjxtc2yiiepmd66"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebnisrg7tjtzd5rwnakvwabhjg7j7m4rgoseae3fopl3cjjsgxbvk"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecqx2oxwfbq2iwmkk4l2xvc6kajc6lndly57b6p7vxkojhzrcwgfa"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacechwhzxskk5cw4qvbaxcyolku6tu6q6prag4kqzlccbzmu7rmvkci"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceciazen2ggajc573g256ill54lk2jf6b6o6jvig2ezxbncseord22"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceb4v2dpsfp7skdoscjdcfzaxc5isxtz43doan4xowqrkmurcwpm44"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.392Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebwmgedad75xmpjbpdfkjndeppcoed3b6rjmmyta7dnkaet6g5732"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecm5axfyvqzehha6sp3sme6ripsgo7t6bchgt3ulybai3vz4afsik"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecrfir5yaawy7sdgmksusi7woxhheqmtej7x7hiz3e5mxccwk3j7u"}

Will send some jobs and keep an eye on things.

davidgasquez · 2023-01-25T17:06:22Z

Heads up that I just got the first websocket issue. Seems it might be still happening. I can't dig much for now though. Will leave a proper report when I'm back next week.

hsanjuan · 2023-01-26T09:28:45Z

@frrist do you have any ideas about why it might be panicking in the first place?

frrist · 2023-01-26T18:00:10Z

@hsanjuan current theory is filecoin-project/filet#22 (comment)

hsanjuan · 2023-01-26T20:30:13Z

The "maybe there are blocks missing in a datastore" doesn't fit the randomness that seems to affect this issue.

frrist · 2023-02-03T00:40:52Z

@davidgasquez can you point me to the github issue for the websocket bug?

davidgasquez · 2023-02-03T10:11:28Z

@davidgasquez can you point me to the github issue for the websocket bug?

There is one higher level in Filet and another in Lotus.

birdychang · 2023-03-27T08:51:01Z

@frrist should we merge?

fix: recover from panics in LoadSectorStates

7f15b8e

- related to filecoin-project/lotus#10050

frrist requested review from birdychang, davidgasquez and kasteph January 19, 2023 17:44

birdychang approved these changes Jan 19, 2023

View reviewed changes

fix: initalize SectorStates structure

2d6b073

- prevents panic

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: recover from panics in LoadSectorStates #1115

fix: recover from panics in LoadSectorStates #1115

frrist commented Jan 19, 2023

birdychang left a comment

davidgasquez commented Jan 23, 2023

frrist commented Jan 23, 2023

davidgasquez commented Jan 23, 2023

davidgasquez commented Jan 24, 2023

davidgasquez commented Jan 24, 2023

davidgasquez commented Jan 25, 2023

davidgasquez commented Jan 25, 2023

hsanjuan commented Jan 26, 2023

frrist commented Jan 26, 2023

hsanjuan commented Jan 26, 2023

frrist commented Feb 3, 2023

davidgasquez commented Feb 3, 2023

birdychang commented Mar 27, 2023

fix: recover from panics in LoadSectorStates #1115

Are you sure you want to change the base?

fix: recover from panics in LoadSectorStates #1115

Conversation

frrist commented Jan 19, 2023

birdychang left a comment

Choose a reason for hiding this comment

davidgasquez commented Jan 23, 2023

frrist commented Jan 23, 2023

davidgasquez commented Jan 23, 2023

davidgasquez commented Jan 24, 2023

davidgasquez commented Jan 24, 2023

davidgasquez commented Jan 25, 2023

davidgasquez commented Jan 25, 2023

hsanjuan commented Jan 26, 2023

frrist commented Jan 26, 2023

hsanjuan commented Jan 26, 2023

frrist commented Feb 3, 2023

davidgasquez commented Feb 3, 2023

birdychang commented Mar 27, 2023