Discussions from the Solana Validator Community on operations, upcoming changes, and best practices.

Community Validator Discussions

solana community

The Solana validator community discusses a recent validator led network outage postmortem.  Next, Solana Labs engineers answer questions about the root cause analysis of the February 25th outage.  Lastly, the group discusses new release process plans for mainnet.

Table of contents

00:00 Introduction and agenda
01:00 Community led postmortem summary
08:13 Update of ongoing root cause analysis
12:00 New plans for testing/releases


CHAT LOGS ----------------

Tim Garcia: https://hackmd.io/@KFEZk8oMTz6vBlwADz0M4A/SJ4sCr9Ai
Tim Garcia: Notes from the community lead post mortem
Rustam Aldaganov: Предлагаю посмотреть Jam-файл: https://jamboard.google.com/d/1mYuPdNz9tYA4XKKUkWkwuLNbDs8mGGT6pMvHT7hbSnY/edit?usp=meet_whiteboard.
Zantetsu: Rustam not sure what that is, it denies me access to view it
Zantetsu: Hm maybe a Russian translation fo the doc that Max posted, just guessing
Tim Garcia: Official report from Solana.com: https://solana.com/news/02-25-23-solana-mainnet-beta-outage-report
Tim Garcia: Ongoing discussion in #mb-outage-20230225
Tim Garcia: https://solana.com/news/plans-to-improve-the-network-upgrades
Kollan House: +1
Naor Shukrun: why we went with 1.14.6 as recomended before evenallowing 5% to upgrade and than 20 etc
Naor Shukrun: 1.14.16*
Thenoblest: nice plan
Overclock Validator: Yeah maybe more perfectionism is required... we were seeing regular 1 min fork/confirmation stalls on mb
Overclock Validator: Maybe some of the forking is caused by block/account cu limits not being tuned enough
MCF MCF: i came late so sry if this is redundant - but i assume you fed the "fat" block into testnet - and it didn't crash ?
Overclock Validator: Idk if the longer confirmation stalls were being looked into before
Azul Dev: lol @ solar flare
Blake Bartenbach: a SOLar flare
Naor Shukrun: so I missed it, sorry!
MCF MCF: those changes are gated tho right
Brian Long: And 1-week advance notice too.
Tim Garcia: Yes, the plan is to create a fat block in testnet and crash it.
Michael Hubbard: @Steve I haven't seen the talk about creating stake accounts on discord?
Michael Hubbard: I thought the goal these days was to primarily keep these discussions public
Max Sherwood: Lido told us to upgrade toward the end of the week
Thenoblest: What if send a "fat" block to version 1.15 in the testnet, I wonder what effect it will have.
Zantetsu: The fat block happened at a "random" time though, there is no preparing for that
Steve Czabaniuk: That happened to be a conversation that happened in person, but generally yes, we try to keep things in public domain
Blake Bartenbach: I believe the politically correct term is "thicc block"
Overclock Validator: Max what validator do yall run?
MCF MCF: @Blake lol
Max Sherwood: We run H2O Nodes
Tim Garcia: Identity transition demo: https://pumpkins-pool.gitbook.io/pumpkins-pool/
Overclock Validator: Oh nice. Glad to see you running a validator -- didn't know you were
Max Sherwood: Yeah I work full time at Ultimate.app and run the H2O Nodes validator together with a friend who is the actual IT/devops guy
Max Sherwood: We were onboarded to Lido's node operator set last year so we get stake from them
Max Sherwood: but it's a public validator
Kollan House: Thanks everyone

DISCLAIMER
The content herein is provided for educational, informational, and entertainment purposes only, and does not constitute an offer to sell or a solicitation of an offer to buy any securities, options, futures, or other derivatives related to securities in any jurisdiction, nor should not be relied upon as advice to buy, sell or hold any of the foregoing. This content is intended to be general in nature and is not specific to you, the user or anyone else. You should not make any decision, financial, investment, trading or otherwise, based on any of the information presented without undertaking independent due diligence and consultation with a professional advisor.  Solana Foundation Foundation and its agents, advisors, council members, officers and employees (the “Foundation Parties”) make no representation or warranties, expressed or implied, as to the accuracy of the information herein and expressly disclaims any and all liability that may be based on such information or any errors or omissions therein. The Foundation Parties shall have no liability whatsoever, under contract, tort, trust or otherwise, to any person arising from or related to the content or any use of the information contained herein by you or any of your representatives. All opinions expressed herein are the speakers’ own personal opinions and do not reflect the opinions of any entities.

Solana Community Validator Discussion - March 2 2023