After going live with the gridcoin.science website in support of the research blockchain, I decided to take another step to support my favourite things: Steemit, Gridcoin and science. Today, I added a new witness node to the network named gridcoin.science, embodying all three in a single endeavor to the mutual benefit of our communities.

A Little About Me

I am Tim, a biomedical engineer living in Christchurch, New Zealand. I have a Bachelor's Degree in Mechanical Engineering, and am currently 6 months from completing a PhD. This doctorate is focused on building computer models of the brain to better understand neurodegenerative diseases like Alzheimer's and Parkinson's. If you would like to learn more about me, I invite you to read my introductory post.

Some of what I get up to on Steemit:

• I am an active moderator for the Utopian.io platform, created by @elear. In this capacity I curate Steemit content to reward high quality contributions to open-source communities. The project has already released over USD$227,506 in contributor rewards, and can be explored further on the Utopian.io homepage.

• I am an active Steemit community member. My blog aims to represent, inform and progress the distributed compute communities surrounding Gridcoin and BOINC. These topics are close to my heart as a medical researcher, where my progress is often slowed by compute limitations. I aim to facilitate a cross-over between the Steemit community and the scientific community to the benefit of both.

• I have been involved with the ongoing debate and analysis around vote and flagging weights, linear vs n^2 rewards, and vote buying. As the platform moves to address these issues for the betterment of the whole, I'd like to be involved in both the search and the implementation of solutions to these challenges.

Meet Nick

Running this witness with me is my close friend Nick (@deltik), a software and systems engineer living around the world – usually in Austin, Texas, or Vienna, Austria. The time difference between him and myself ensures round-the-clock support if anything were to happen to the witness server.

Nick's professional experience began with web hosting back in 2008. Since then he has branched out into managed support, custom software solutions, and cloud virtualization. He is currently working as a converged infrastructure engineer, while contributing significant resources to open-science BOINC projects like SRBase and Cosmology@home.

Despite the ability to copy conventional digital data bit for bit, that data is easily corrupted and lost.

Nick is fascinated by blockchains because in contrast, they are ideal for preserving data integrity. One such blockchain implementation is Steem, and he believes in the potential of this backend for the social aspect rather than it merely being a transaction ledger.

For this reason, Nick has joined forces with me to support the Steem blockchain with a new witness!

Witness Tech Specs

We've implemented the witness daemon a bit differently. We wanted the foundation to be rock-solid, so we started with dedicated hardware:

CPU: Intel® Xeon® Processor D-1521 @ 2.40GHz
Memory: 64GB DDR4-2666 ECC
Disk: 2×480GB SSD and 2×2TB HDD
Bandwidth: 250Mbps
Operating System: Ubuntu 16.04 LTS

Storage

On the disks, we did a custom installation of Ubuntu 16.04 LTS on a ZFS root. There are two pools that mirror each type of disk (fastpool made from the SSDs and slowpool made from the HDDs). We chose ZFS because of its unparalleled data reliability features.

What does ZFS do for us?

• The Steem blockchain is quite compressible. Since ZFS supports native compression, we were able to achieve a 37% reduction in disk usage.
• ZFS lets us take regular snapshots, which means we can roll back the witness in case of a screw-up. Hopefully there won't be any incidents, but it's best to be prepared.
• Snapshots also allow us to send backups as a stream (zfs send/zfs receive) into flexible backup destinations. We have a 500GiB NFS backup datastore right now, which is more than enough for the witness, and we're exploring more resilient disaster recovery solutions for the future.
• The ZFS Adjustable Replacement Cache (ARC) boosts performance of the ZFS datasets, which make input-output (I/O) operations on our witness faster.
• Unlike standard RAID, ZFS is aware of the data it stores and checksums all of it. Combined with the disk mirror (analogous to RAID 1), we are even protected from silent data corruption, and regular scrubs heal any inconsistent data automatically.

Virtualization

As for the witness itself, we put it in a KVM virtual machine on this dedicated server. Here are the specs of that virtual machine:

CPU: 4 logical cores of Intel Xeon D-1521
Memory: 16GiB
Disk: 400GiB SSD-backed
Bandwidth: Unmetered
Operating System: Ubuntu 16.04 LTS

Virtualization has several advantages:

• Isolation: The witness runs in its own environment so that it can't interfere with the operational infrastructure below.
• Security: If the witness is compromised by some unforeseen vulnerability, we can stop it in its tracks, go back in time, and rectify the vulnerability before it's exploited again.
• Overhead: Virtual machines don't have the hardware overhead of dedicated servers, so starting up is much faster.
• Portability: In combination with ZFS snapshots, we've opened the possibility of migrating the witness to another physical server if we ever need to upgrade.

Why not containerization?

Although steemd can make use of operating-system-level virtualization through the use of Docker, we chose full virtualization for greater isolation and to explore how far we could go with zram.

Memory

There's a problem. As of January 2018, steemd loads over 20GiB into its memory-mapped file, shared_memory.bin, but we only gave 16GiB of RAM to the virtual machine. Doesn't that mean we run out of memory or have to do unacceptably slow paging to swap?

Actually, no. We discovered that this witness runs comfortably with the help of zram. steemd's entire memory-mapped file, no matter how large the config.ini shared-file-size is set, compresses to around 8.2GiB (as of January 2018) in zram.

Thanks to the speed of zram, there is a negligible loss in Steem witness performance.

The Steem blockchain is going to keep growing, though, which means that someday, the shared memory file will be too large to fit in the 16GiB allocated to the witness virtual machine. Fortunately, the dedicated server is fully under our control and has 64GB of RAM, so we will have room to scale vertically for a while.

Room for Improvement

If the community supports this witness, we will match the commitment by developing a high availability infrastructure.

Our goal will be to minimize downtime. There are existing ways to make witnesses more robust, of which we will make use, but we would like to go above and beyond. We are considering:

• The possibility of an active/active configuration, allowing two or more witnesses to run at the same time without stepping on each other
• Shared storage, enabling live migrations so that physical servers can be taken down while the virtual machines stay up
• Faster ways to redeploy steemd, maybe obviating the need for --replay-blockchain

Closing Remarks

If you have made it this far - thank you for reading!

To vote, visit https://steemit.com/~witnesses and add gridcoin.science to the box at the bottom of the page, click vote, and authorise using your Active Key.

Wrapping things up, I'd like to invite you to ask @deltik or me anything in the comments below. Let's get to know each other better!

---