SPX x v16 Calibration Testing

[Reiers]

Hello SPX Fellows

Upgrade your Calibration SP to Lotus v1.16.0-rc1.
❗ no FVM environment needed.

The Calibration-net upgrade is at epoch 1044660, 2022-06-16T17:30:00Z.

Your lotus node will switch from legacy VM to FVM automatically upon the upgrade.

🚀 This post if you are participating in this testing round.
If you aren't able to join for reasons, please reach out to @jennijuju or me on Slack!

Task assignment

So this is a tricky one because this will need some timing:

• Perform a prove aggregation for at least 4 sectors as soon as node upgrades

• Confirm that the PCA is successful

• Perform a prove aggregation for at least 4 sector at least 20 epochs prior to the calibration upgrade
  (So it has the PCA message pending)

• Send the pending msg post upgrade, the message should fail!

• Monitor that lotus should retry, create and send a new PCA message, message should land successfully

• Check that the sectors from in message is successful at the wdPost.

This is important - please comment that all above went successfully or comment the failure.
Keep all logs!

Testing after the network upgrade

Regular Operations

Reply with:

• - 🔁 If lotus daemon stays in sync
• - 🦭If you have successfully sealed some new sectors.
• - 🤝If you have successfully made some storage deals.
• - 🔥 If you have successfully made some snap deals.
• - 🗄️ If you have successfully setup and config the Index Provider.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

Bonus testing

(You dont have to do everything on this list, but if you have the time please do some of them)

Reply with:

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

❗ Create an issue if you think there is a bug, don't forget to attach the logs!

• Save the daemon logs, miner logs, and lotus-miner info all if you want to report a bug! If the log files are large, you can upload them in SPX Calibration - v1.16.1-rc1 Logs folder and link it in your issue report/slack message/..

New feature and Improvements

• Built-in actor bundles

Read more about it here.

[jamerduhgamer]

Calibnet SP ID - t016773

• - 🔁 If lotus daemon stays in sync

• - 🦭 If you have successfully sealed some new sectors.

• - 🤝If you have successfully made some storage deals.

• - 🔥 If you have successfully made some snap deals.

• - 🗄️ If you have successfully setup and config the Index Provider.

• - 📂 If you have successfully had some retrieval deals.

• - 💻 If you passed your first WindowPoSt.

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[TippyFlitsUK]

• - 🔁 If lotus daemon stays in sync

• - 🦭If you have successfully sealed some new sectors.

• - 🤝If you have successfully made some storage deals.

• - 🔥 If you have successfully made some snap deals.

• - 🗄️ If you have successfully setup and config the Index Provider.

• - 📂 If you have successfully had some retrieval deals.

• - 💻 If you passed your first WindowPoSt.

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[tmyuu]

Calibnet SP ID - t037137

Regular Operations

• - 🔁 If lotus daemon stays in sync
• - 🦭 If you have successfully sealed some new sectors.
• - 🤝If you have successfully made some storage deals.
• - 🔥 If you have successfully made some snap deals.
• - 🗄️ If you have successfully setup and config the Index Provider.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

Bonus testing

• Make a deal and perform a paid retrieval for the data
  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>
• Invoke CompactPartitions, which has never been used on mainnet
  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

$ lotus-miner sectors terminate --really-do-it 1
$ lotus-miner sectors terminate flush
bafy2bzacebwobqm46o6v4ikuzvhc4f7577crxu5s42mektapv7u5co563v35y
$ lotus-miner proving deadline 1
Deadline Index:           1
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
Sectors:                  6
Sector Numbers:           [1 2 3 4 5 6]
Faults:                   0
Faulty Sectors:           []
$ lotus-miner sectors compact-partitions --really-do-it --deadline 1 --partitions=0 --actor t037137
compacting 1 paritions
Requested compact partitions in message bafy2bzacecaakojqsqcpfj5svjtbne5ullmvfxvlsw5xo3pgfau4745pjfyza
$ lotus-miner proving deadline 1
Deadline Index:           1
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
Sectors:                  5
Sector Numbers:           [2 3 4 5 6]
Faults:                   0
Faulty Sectors:           []

• Extend a sector expiration
  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

$ lotus-miner sectors extend --new-expiration 2616296 2
bafy2bzacecn3oysa4w7eailibgsyae5hn7xfwn4dih6qzdz4kammneop7qzey
$ lotus-miner sectors extend --new-expiration 2616296 3
bafy2bzacebvmt6apht7rl6f6f6xotaqh4x7slajmbfyan6umhabdm5whpnxik
$ lotus-miner sectors extend --new-expiration 2616296 4
bafy2bzaceb7zf2lnz474u7xjpbnwjfj2hw6uf3tqp2ua7ynf2sjnmxrbekkx4
$ lotus-miner sectors extend --new-expiration 2616296 5
bafy2bzaceajkgypy4kigr2xurm4t4lj26yzghl76ypjs7wyoskjpbc3uhda5m
$ lotus-miner sectors extend --new-expiration 2616296 6
bafy2bzaced2t3htezxtl76j7cnhrplg5wa42gwxj6ydsdjvo2sxhfuk6c4hyu
$ lotus-miner sectors list
ID  State           OnChain  Active  Expiration                    Deals 
2   Proving         YES      YES     2616296 (in 1 year 24 weeks)  CC     
3   Proving         YES      YES     2616296 (in 1 year 24 weeks)  CC     
4   Proving         YES      YES     2616296 (in 1 year 24 weeks)  CC     
5   Proving         YES      YES     2616296 (in 1 year 24 weeks)  CC     
6   Proving         YES      YES     2616296 (in 1 year 24 weeks)  CC

• Snap a deal
  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[tmyuu]

Additional testing

• Split markets

Enabled subsystems (from miner API): [Mining Sealing SectorStorage]
Enabled subsystems (from markets API): [Markets]
Chain: [sync ok] [basefee 100 aFIL]
Miner: t037137 (32 GiB sectors)

• Migrate from Lotus to Boost

VERSION:
   1.1.0-rc2+git.0f34516

[neondragon]

Calibnet t031063

• - 🔁 If lotus daemon stays in sync
• - 🦭If you have successfully sealed some new sectors.
• - 🤝If you have successfully made some storage deals.
• - 🔥 If you have successfully made some snap deals.
• - 🗄️ If you have successfully setup and config the Index Provider.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

Bonus testing

(You dont have to do everything on this list, but if you have the time please do some of them)

Reply with:

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[cryptowhizzard]

Calibnet SP ID - t037132 || Edit 25 - Jun -> Moved further testing to mainnet @ F01240

• - 🔁 If lotus daemon stays in sync

• - 🦭 If you have successfully sealed some new sectors.

• - 🤝If you have successfully made some storage deals.

• - 🔥 If you have successfully made some snap deals.

• - 🗄️ If you have successfully setup and config the Index Provider.

• - 📂 If you have successfully had some retrieval deals.

• - 💻 If you passed your first WindowPoSt.

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now --> moved to boost
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[Reiers]

• - 🔁 If lotus daemon stays in sync
• - 🦭If you have successfully sealed some new sectors.
• - 🤝If you have successfully made some storage deals.
• - 🔥 If you have successfully made some snap deals.
• - 🗄️ If you have successfully setup and config the Index Provider.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

Bonus testing

(You dont have to do everything on this list, but if you have the time please do some of them)

Reply with:

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[Reiers]

PCA was successful and Lotus did retry the message after the upgrade.

[SBudo]

t029562

• 🔁 If lotus daemon stays in sync.
• 🦭 If you have successfully sealed some new sectors.
• 🤝If you have successfully made some storage deals.
• 🔥 If you have successfully made some snap deals.
• 🗄️ If you have successfully setup and config the Index Provider.
• 📂 If you have successfully had some retrieval deals.
• 💻 If you passed your first WindowPoSt.

Bonus testing

(You don't have to do everything on this list, but if you have the time please do some of them)

• Make a deal and perform a paid retrieval for the data

  • You may need to edit the miner's config to allow for online deals. Usually located in /var/lib/
  • lotus client import <file> , then lotus client deal
  • lotus-miner storage-deals pending-publish --publish-now
  • Wait for new sector to show up in lotus-miner sectors list
  • lotus-miner sectors seal <sector num>
  • Wait for sector to enter the state SubmitPreCommitBatch
  • lotus-miner sectors batching precommit --publish-now
  • Wait for sector to enter the state SubmitCommitAggregate
  • lotus-miner sectors batching commit --publish-now
  • Wait for sector to become active.
  • Perform a PAID retrieval for the data (tests payment channels)
    • You may need to set a retrieval price on the SP sudo lotus-miner retrieval-deals set-ask --price <price>
    • lotus client retrieve --maxPrice <price> <data CID> retr.txt
    • Get the CID of the payment channel: lotus paych list
    • lotus paych settle <paych CID>
    • Wait 12 hours, then lotus paych collect <paych CID>

• Invoke CompactPartitions, which has never been used on mainnet

  • Add some large number of sectors, and then terminate (say) 50% of them.
  • Compact the partitions: lotus-miner sectors compact-partitions --really-do-it --deadline <deadline num> --partitions={0,1,2} --actor <miner addr>
  • All terminated sectors in the partitions specified should be gone. Partitions should be packed as tightly as allowed.

• Extend a sector expiration

  • sudo lotus-miner sectors extend --new-expiration <expiration epoch> <sector num>
  • lotus-miner sectors list to check if sector expiration has been updated

• Snap a deal

  • Create a CC sector on your miner
  • make a deal from your client
  • sudo lotus-miner sectors snap-up <sector num>
  • sudo lotus-miner storage-deals pending-publish --publish-now
  • sudo lotus-miner sectors seal <sector num>

[SBudo]

PCA was successful and Lotus did retry the message after the upgrade.

[SBudo]

Extend sector expiration works - lotus-miner sectors extend --new-expiration 2593088 18
Message: bafy2bzacectbvdm6ptbkp2fp4pfl3rmfbdjv2wqcl6u6n7zznfczg5uh2qneq

Renew sector works - lotus-miner sectors renew --sector-file ./renew.txt --really-do-it
Message:

Renewing 1 sectors: bafy2bzacebomzff5eyrcbsoj2ioruxonvq4u3bd4ot4lqilx4z4nsh25g2mna
1 sectors renewed

[SBudo]

Passed WindowPoSt, included sectors committed during the network upgrade

[beck-8]

With the upgrade of FVM, subcalls can no longer be queried normally.
Commands such as lotus-shed sectors termination-estimatecan --actor xx xx no longer be used normally,return nil
Also affected are lotus state call lotus state exec-trace,this result subcalls is nil

[jennijuju]

do you have LOTUS_VM_ENABLE_TRACING enabled?

[beck-8]

no

[beck-8]

do you have LOTUS_VM_ENABLE_TRACING enabled?

this is big bug,Someone has already fixed it
filecoin-project/filecoin-ffi#293

[jennijuju]

@beck-8 thanks for raising will 👁️

[benjaminh83]

t031131 (512GiB total, 64GiB miner, onboarding 0TiB/day)
t036004 (48TiB total, 32GiB miner, onboarding 15TiB/day)!

• 🔁 If lotus daemon stays in sync.
• 🦭 If you have successfully sealed some new sectors.
• 🤝If you have successfully made some storage deals.
• 🔥 If you have successfully made some snap deals.
• 🗄️ If you have successfully setup and config the Index Provider.
• 📂 If you have successfully had some retrieval deals.
• 💻 If you passed your first WindowPoSt.

[benjaminh83]

Sorry I was in a bit late on answering here, but we did jump to v16 on the mark - no issues!

[benjaminh83]

I'm not sure if we can do deals on these calib miners right now, as we are doing scale test to storage cluster solutions.

[namoo1]

Meest NAM

t037143

• 🔁 If lotus daemon stays in sync.
• []If you have successfully sealed some new sectors.
  🤝If you have successfully made some storage deals.
  🔥 If you have successfully made some snap deals.
• 🗄️ If you have successfully setup and config the Index Provider.
  📂 If you have successfully had some retrieval deals.
• 💻 If you passed your first WindowPoSt.

[namoo1]

2022-06-23T19:56:38.801 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2022-06-23T19:56:38.805 INFO storage_proofs_core::compound_proof > snark_proof:start
2022-06-23T19:56:38.914 INFO bellperson::groth16::prover > Bellperson 0.21.0 is being used!
2022-06-23T19:57:03.244 INFO bellperson::groth16::prover > synthesis time: 24.33007372s
2022-06-23T19:57:03.244 INFO bellperson::groth16::prover > starting proof timer
2022-06-24T04:57:06.009+0900    INFO    miner   miner/miner.go:479      completed mineOne       {"tookMilliseconds": 4, "forRoun
d": 1065115, "baseEpoch": 1065114, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 0, "lookbackEpochs
": 900, "networkPowerAtLookback": "3319116366610432", "minerPowerAtLookback": "103079215104", "isEligible": false, "isWinner": f
alse, "error": null}
2022-06-23T19:57:06.875 DEBUG bellperson::gpu::locks > Acquiring priority lock at "/sealworker/tmpdir/bellman.priority.lock" ...
2022-06-23T19:57:06.875 DEBUG bellperson::gpu::locks > Priority lock acquired!
2022-06-23T19:57:06.875 DEBUG bellperson::groth16::prover > get h
2022-06-23T19:57:07.175 INFO bellperson::gpu::locks > GPU is available for FFT!
2022-06-23T19:57:07.175 DEBUG bellperson::gpu::locks > Acquiring GPU lock at "/sealworker/tmpdir/bellman.gpu.lock" ...
2022-06-23T19:57:07.175 DEBUG bellperson::gpu::locks > GPU lock acquired!
2022-06-23T19:57:07.175 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-06-23T19:57:07.203 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2022-06-23T19:57:07.203 INFO bellperson::gpu::fft > FFT: Device 0: NVIDIA GeForce RTX 3080
2022-06-23T19:57:07.203 INFO bellperson::domain > GPU FFT kernel instantiated!
2022-06-23T19:57:22.821 DEBUG bellperson::gpu::locks > GPU lock released!
2022-06-23T19:57:22.821 DEBUG bellperson::groth16::prover > multiexp h
2022-06-23T19:57:22.821 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2022-06-23T19:57:22.821 DEBUG bellperson::gpu::locks > Acquiring GPU lock at "/sealworker/tmpdir/bellman.gpu.lock" ...
2022-06-23T19:57:22.821 DEBUG bellperson::groth16::prover > get l
2022-06-23T19:57:22.821 DEBUG bellperson::gpu::locks > GPU lock acquired!
2022-06-23T19:57:22.821 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-06-23T19:57:22.823 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0.2)
2022-06-23T19:57:22.823 INFO bellperson::gpu::multiexp > Multiexp: Device 0: NVIDIA GeForce RTX 3080 (Chunk-size: 2173429)
2022-06-23T19:57:22.823 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2022-06-23T19:58:45.543 DEBUG bellperson::groth16::prover > get_a b_g1 b_g2
2022-06-23T19:58:45.543 DEBUG bellperson::groth16::prover > multiexp a b_g1 b_g2
2022-06-24T04:59:06.197+0900    INFO    miner   miner/miner.go:479      completed mineOne       {"tookMilliseconds": 4, "forRoun
d": 1065119, "baseEpoch": 1065118, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 0, "lookbackEpochs
": 900, "networkPowerAtLookback": "3319116366610432", "minerPowerAtLookback": "103079215104", "isEligible": false, "isWinner": f
alse, "error": null}
2022-06-24T04:59:36.117+0900    INFO    miner   miner/miner.go:479      completed mineOne       {"tookMilliseconds": 2, "forRoun
d": 1065120, "baseEpoch": 1065119, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 0, "lookbackEpochs
": 900, "networkPowerAtLookback": "3319116366610432", "minerPowerAtLookback": "103079215104", "isEligible": false, "isWinner": f
alse, "error": null}
2022-06-24T05:00:06.007+0900    INFO    miner   miner/miner.go:479      completed mineOne       {"tookMilliseconds": 3, "forRoun
d": 1065121, "baseEpoch": 1065120, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 0, "lookbackEpochs
": 900, "networkPowerAtLookback": "3319116366610432", "minerPowerAtLookback": "103079215104", "isEligible": false, "isWinner": f
alse, "error": null}
2022-06-23T20:00:11.169 DEBUG bellperson::gpu::locks > GPU lock released!
2022-06-23T20:00:12.446 DEBUG bellperson::groth16::prover > proofs
2022-06-23T20:00:12.447 DEBUG bellperson::gpu::locks > Priority lock released!
2022-06-23T20:00:12.447 INFO bellperson::groth16::prover > prover time: 189.202415673s
2022-06-23T20:00:13.738 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-06-23T20:00:13.738 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2022-06-23T20:00:13.739 INFO filcrypto::util::types > generate_window_post: end
2022-06-24T05:00:13.739+0900    INFO    storageminer    storage/wdpost_run.go:664       computing window post   {"batch": 0, "el
apsed": 214.997079669, "skip": 0, "err": null}
2022-06-23T20:00:13.740 INFO filcrypto::util::types > verify_window_post: start
2022-06-23T20:00:13.740 INFO filecoin_proofs::api::window_post > verify_window_post:start
2022-06-23T20:00:13.740 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-ke
y
2022-06-23T20:00:13.740 INFO filecoin_proofs::caches > found params in memory cache for WINDOW_POST[34359738368]-verifying-key
2022-06-23T20:00:13.756 INFO filcrypto::util::types > verify_window_post: end
2022-06-24T05:00:13.756+0900    ERROR   storageminer    storage/wdpost_run.go:711       generated incorrect window post proof
{"post": [{"PoStProof":8,"ProofBytes":"mbwf0/PWo4jTMCSC/KEdZozJODe1alZtAz06hRDvqw2ryUVKi1ZI25yVf5RArtJXosQ1imGpQRWXpO2eUjrBOPnHM
hKNFzL6dLAeuHqa/NtraDTelojz0DtOL/AMDITVD0rA33PpHsqggNvxOLOAfhIMW383SjQbB057tMALr/3xRJJIrYRnt899wSwUfWZsmAnpm6EM4m5zazx2duSKH/fCq
BCJvUzVqre2nDgixrqHFwEMLxnGp7yDEcyaopt/"}], "error": null}
2022-06-24T05:00:13.757+0900    ERROR   storageminer    storage/wdpost_run.go:100       runPoStCycle failed: checking sectors to
 skip:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /root/lotus/storage/wdpost_run.go:607
  - checking provable sectors:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).checkSectors
        /root/lotus/storage/wdpost_run.go:231
  - context canceled
2022-06-23T20:00:13.757 INFO filcrypto::util::types > generate_fallback_sector_challenges: start
2022-06-24T05:00:13.757+0900    WARN    storageminer    storage/wdpost_changehandler.go:254     Aborted window post Proving (Dea
dline: &{CurrentEpoch:1065035 PeriodStart:1065054 Index:0 Open:1065054 Close:1065114 Challenge:1065034 FaultCutoff:1064984 WPoSt
PeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

Daemon:  1.16.0-rc2+calibnet+git.6638ed4c3+api1.5.0
Local: lotus-miner version 1.16.0-rc2+calibnet+git.6638ed4c3

[hail100]

How deal with this problem?
lotus VERSION:1.16.0-rc3+calibnet+git.824da5ea5

2022-06-24T02:34:20.781 INFO filcrypto::util::types > create_fvm_machine: start
2022-06-24T02:34:20.781 INFO filcrypto::fvm::machine > using FVM V1
2022-06-24T02:34:20.782 INFO filcrypto::util::types > create_fvm_machine: end
2022-06-24T10:34:20.782+0800    WARN    rpc     [email protected]/handler.go:279        error in RPC call to 'Filecoin.MinerCreateBlock': failed to load tipset state:
    github.com/filecoin-project/lotus/chain/consensus/filcns.(*FilecoinEC).CreateBlock
        /home/fil/git/lotus/chain/consensus/filcns/mine.go:25
  - making vm:
    github.com/filecoin-project/lotus/chain/consensus/filcns.(*TipSetExecutor).ApplyBlocks
        /home/fil/git/lotus/chain/consensus/filcns/compute_state.go:177
  - Serialization error for Cbor protocol: RequireLength { name: "tuple", expect: 1, value: 0 }
2022-06-24T02:34:21.790 INFO filcrypto::util::types > generate_winning_post_sector_challenge: start
2022-06-24T02:34:21.790 INFO filecoin_proofs::api::winning_post > generate_winning_post_sector_challenge:start
2022-06-24T02:34:21.790 INFO filecoin_proofs::api::winning_post > generate_winning_post_sector_challenge:finish
2022-06-24T02:34:21.790 INFO filcrypto::util::types > generate_winning_post_sector_challenge: end