Several grumpy opinions about remote work at Tailscale

As a "fully remote work" company, we had to make some choices about the technologies we use to work together and stay in touch.

We decided early on - about the time we realized all three cofounders live in different cities - that we were going to go all-in on remote work, at least for engineering, which for now is almost all our work. As several people have pointed out before, fully remote is generally more stable than partly remote. In a partially remote team, the remote workers seem to always end up treated as an underclass, overlooked in meetings, bypassed for promotions, fired when they eventually refuse to relocate because the remote work policy inevitably changes (hi, Yahoo!), etc.

The good news with our plan is the founders could "dogfood" a few different remote work ideas ourselves before we ever hired anyone. So we decided to try some stuff. Here's what we discovered.

Notion

We're using Notion as a team wiki and note taking app. It's ... okay. I mean, it's probably the best tool for the job, and it's great in some ways, but it's severely limited in others.

Things I like about Notion:

• Great for quick to-do lists and milestone planning
• Easy to make persistent hyperlinks between docs
• Easy to arrange docs in a hierarchy (but not as organized as good old GracefulTavi)
• Tables and Kanban board views are pretty awesome
• Just the right level of formatting. When I paste text into Notion, I never worry about it coming out in a weird font or colour.

Things that drive me crazy:

• The "show me what changed" view is nearly useless; tons of updates about tiny clutter changes, but no good way to give me a deduplicated list of all the docs that changed. Virtually any wiki's RecentChanges view is better.

• Doc comments miss the point of doc comments, by being almost invisible and creating no incentive to resolve them. (Trivia: Rumour has it that Google Docs comments also sucked until an intern showed up and made them insanely better as a 20% project.) There are so many ways that the greatness of Google Docs comments has failed to be copied by every other tool (including, for example, Google Sheets). Everything else sucks. When we want to wordsmith stuff as a team, we move it from Notion into Docs.

• Support for to-do lists and "reminders" is there, but pretty weak. For example, there's no way to make repeating reminders or get a consolidated list of to-do items across multiple pages, so people request a separate to-do list app. So far we've resisted, but we won't be able to for much longer.

• No API means you can't fix any of the limitations yourself.

Anyway, as they say, there are the tools you complain about and the tools you don't use. I've tried a heckuvalot of content managers and they've all been worse, so Notion it is. To be fair, it's a very big area and hard to please everyone. And I'm really picky. But they're so close...

Keybase to Slack

At first we tried using Keybase to manage our secret keys, and coincidentally its built-in team chat feature for our team chats. Keybase has a bit of a bad reputation because of some of their early cryptography missteps and their (very unfortunate) recent association with cryptocurrency. But whatever you think of their security or business model, their chat system is surprisingly one of the best. You can make channels and securely confirm identities without stupid QR codes; message expiration rules are clear; the notifications are A+. Among other things - and this completely dazzled me - when I read a message on any of my devices, the notification for that message disappears instantly from all my other devices! I didn't even know it was possible to auto-remove obsolete notifications, so seldom is it done.

Which, of course, led me to wonder why it isn't done. In my cynicism I'm sure I can guess why; auto-removing notifications never increases your "engagement" metric. Whereas a completely bogus chat notification from four hours ago, already dealt with four hours ago on a different computer, drives engagement every time. I respect the Keybase people for choosing the path of user happiness, except I suspect they're soon going to need paying users instead of happy users, because that's the world we live in.

However, keybase had some problems for us. First, it guzzles absolutely epic amounts of CPU and memory. If you think Slack is bloaty, Keybase outdoes it by like 2x, plus it has giant memory leaks so you have to restart it all the time. There's no web UI (they're too paranoid about security), and the android app just crashes for me on ChromeOS. In other news, I'm pretty sure I never ever want to hear about a "security and privacy" tool that includes 150MB of Electron (aka "Chromium but with the security and privacy features turned off").

Also, nobody but us uses Keybase, and it doesn't support popular cute things like Github integrations. So unfortunately, we had to give up on it and switch to Slack. Y'all know how Slack works so I hardly need to describe it, but I would summarize it as "absolutely terrible at everything except user lock-in," and here we are. There's a business lesson in there somewhere.

I eventually turned off Slack notifications entirely, after experimenting with many different variations. @here is an abomination; notifications in each "other Slack instance" need to be set separately; it spams your @#$!! phone with every single message anyone types, even while you're on your PC. Forget it, notification privileges revoked, and I've been much happier since.

Gmail

For a while, we tried to run our own email server (in the name of being free of "big tech" for our core systems) but it didn't work out. Gmail's UI gets worse every year (correlated with decreasing information density, though the causation lies elsewhere), but at least it's mostly familiar.

Interestingly, because of Notion and Slack, we hardly use email at all between us internally. It's almost exclusively used for customers and investors.

At the advice of the excellent book The Great CEO Within, I followed the instructions in Andreas Klinger's guide to Gmail Inbox Zero. His combination of Gmail configs is pure genius; it completely changed how I do email, and makes Inbox Zero easy and achievable, by separating the triage and work phases. Highly recommended. I also learned about several Gmail options I didn't know existed.

Streak CRM

We reviewed several CRM tools. The consistent advice we received was, "You'll end up on Salesforce eventually, but don't do it yet." Ok, sure, I can take advice.

Streak was appealing because I wanted something that would integrate extremely tightly with my email. Streak does what I want: I associate an email thread with a particular customer or helpdesk ticket, and then it's magically shared with all the other Streak users in your domain, and it continues sharing as new messages are sent and received, and it's 100% inside the Gmail UI. Not bad at all.

The underlying concept of Streak is what I would call "batshit insane from top to bottom." It has a tough learning curve at first, but so apparently does every CRM. It has scattered features all over that just look like extra buttons or tabs in the Gmail UI. The frustration their dev team must have endured as they implemented this, and the frustration they must continue to endure as they keep it up to date, must be nearly intolerable. But the end result is quite remarkable; these are devs who care about keyboard shortcuts, highly efficient workflows, and making short work of huge batches of emails. I'd say Google should buy them and just integrate the whole thing into standard Gmail, except then Google would kill them with love by accident, as megacorporations usually do with acquisitions, and we'd all be worse off. Oh well.

Anyway it works, I like it. And besides sales, it's quite a remarkably good support/helpdesk ticket system, which it seems to have only tangentially been designed for. Customers don't even know they're in a ticket system (is that better or worse?) but it lets us collaborate on tickets, make sure tickets don't get lost, and so on, just like a good ticketing system should. Except without having to learn yet another new UI.

(Uh, just because we have a good ticketing system doesn't mean we can actually keep up with emails some days. Sorry. We try. Life at a startup is exciting.)

Videoconferencing

In a remote company, meetings are essential. There are all kinds of subtle issues that affect the way humans interact on the call. This is the area where we experimented the most; unfortunately, although videoconferencing has come very very very far in the last 10 years, there is still no perfect answer.

Let's enumerate some imperfect answers, in vaguely chronological order:

• Webex: included only for completenes, because it's a total tire fire. Nobody who has honestly reconsidered their conferencing system in 10+ years would choose it. If you have a subscription to Webex, cancel it right now. Your employees, suppliers, and customers will hold a festival in your honour.

• Skype: still works surprisingly okay. Their original peer-to-peer network (now deleted) inspired Tailscale's VPN mesh. But Microsoft wants you to use Skype for Business instead, so it's hard to send people links to prescheduled group meetings. You can see the writing on the wall, might as well not even start.

• Skype for Business / Microsoft Teams: seems okay actually, but you have to buy into some whole Microsoft Teams Ecosystem to get started. Your meeting invitees don't have to buy in, but it sure makes it look like they do, which makes a bad first impression. Forget it.

• Hangouts: permanently deprecated but somehow still not dead. They deserve some credit for being (I think) the first videoconferencing app that didn't require a browser plugin or standalone app. To be fair, this is because they make the browser and the browser "coincidentally" now has videoconferencing APIs (webrtc) in it. Ironically however, Hangouts is one of the least effective users of webrtc and has numerous bugs.

  One of my favourite bugs that has been around for 5+ years: if you sit on a call for a few minutes waiting for someone else to arrive, there will often be a completely unreasonable amount of lag+echo when they do. This will go away if you close the window and reconnect. But don't go away too long or it'll happen to the other person! Plus, the best advice to get rid of echo is "clap your hands to retrain the echo canceller!" which, while it works, is not necessary with anybody else's tool.

  Hangouts has some good bits that make it sticky and/or popular. First, it's free! Second, it supports dial-in phone numbers for people who have technical problems. Third, it's highly integrated, not to say bundled, with Calendar, to the point where it obnoxiously auto-schedules a Hangouts meeting id even for your team lunch.

  Unfortunately, Hangouts has perenially bad lag (sometimes several seconds), huge problems with echo, a video codec that kills the battery on several popular kinds of computers, and a pretty bad screen layout algorithm that only shows the current speaker in one giant window, which makes it hard to follow the facial expressions of all your teammates.

• Whereby: formerly Appear.in. Let's be honest, Appear.in was a much better name, but apparently some other completely unrelated company in Norway had trademarked the name "Appear" (can you even trademark that? Maybe in Norway) so here we are. Anyway, someone recommended them to me a few years ago as the first pure-webrtc conferencing tool that cared about bufferbloat, and that's exactly right: no plugins required, far less bloaty javascript than Hangouts, extremely low latency. And nowadays their layout algorithm is very nice, showing everyone in the call at once and maximizing the space used on the screen for what matters: people's facial expressions.

  There are a few downsides. First, their layout algorithm is utterly useless when someone starts screen sharing; it cuts off both sides of the screen and/or makes it tiny unless you fiddle around in sub-submenus. There should be a shortcut or something. Also, they biased so far toward low latency that if your network is at all glitchy or jittery, it's almost game over; they don't recover well in situations where the only right answer is, unfortunately, to introduce enough lag to compensate for the jitter.

  They are also remarkably bad at taking your money. We finally started paying them a few weeks ago only because we felt bad for them when we started paying for Zoom (see below), which we like less. (As I write this, Tailscale still has no payment system hooked up to our web site at all, so I sympathize, empathize, hypocritize, and shake my head all at once. Sorry, Whereby; we love you, but please try not to be Dumb Like Us.)

  As a result of the glitchiness on high-jitter links and their lack of dial-in phone numbers, unfortunately we don't use Whereby for meetings with outsiders; there's too much risk of a connection failure, wasting 10-15 minutes at the start of a meeting. We do use it for internal team meetings (where it's worth investing in one-time setup to have a long-term great experience).

  Another weird limitation is that Whereby meeting rooms really act like "rooms"; there isn't a separate meeting id per meeting. That makes it hard to do simple things like a series of half-hour meetings; the next people wander into a meeting with the previous people. Awkward.

• Zoom: Ok, I'm gonna be straight with you, Zoom is obviously what you need to use for your meetings - everybody knows it - and it's also quite bad. Some of the badness comes from what used to be their technical advantage: reputedly, they made a special video codec/plugin that would subtly speed up and slow down the audio and video streams to compensate for network jitter. This is kind of a long story, but I think the idea is rather than just freezing the video when packets got lost and retransmitted, they could slow down the stream during the missing segment, then speed it up again once it resumed, and you wouldn't realize there was a glitch. Pretty clever actually. In fact, I'm guessing this is why they're called "Zoom." Clever, right?

  Except, well, it doesn't seem to work. I still get plenty of audio/video glitches, albeit non-fatal ones. What's worse, to make their fancy codec work, they have to have a native plugin or app, because webrtc can't do it. That means the first time you call into Zoom, there's this installation process that installs what turned out to be an accidental spy cam app on your Macbook. Whoops. I mean, they've fixed it now, so that's nice.

  The latency is also not great (wasn't lower latency the whole point of the fancy algorithm?). The screen layout algorithm is abysmal, giving you the choice between "one giant screen with one person talking and no indicator when new people join the call" or "each person in a tiny little box and most of the screen pixels are wasted."

  But here's the thing: invite links work. Their plugin installer is transparent enough that pretty much everyone succeeds at it (quite a feat!), and for everyone else, they have dial-in phone numbers. They have integrations with everything, including Calendly and Google Calendar and Slack. They sell hardware "Zoom rooms" that let you have futuristic videoconferencing rooms that used to cost 10x as much. (Hangouts tried to do this too. It was going pretty well before they lost focus.) Most of all though, Zoom just works. It doesn't work well, which is sad. But it works. I've never had the "10 minutes futzing with the videoconference connection" at the start of a Zoom call. And they absolutely figured out how to extract big money from you as soon as you start liking it, so they have a business model ensuring they'll stick around. So yeah, this is what we use for calls with people outside our team, although the high latency and bad video quality continue to make me sad.

• Honourable mention: FaceTime: The audio/video quality and latency in FaceTime is absolutely, positively, the undeniable best of everything we tested, in both good network conditions (where Whereby is comparable) and bad conditions (where everything else is worse). This probably has to do with them hiring some of the best network and codec people in the world to work on it. (eg. I saw a talk by Stuart Cheshire about how ECN contributes to this.) Unfortunately, the usual Apple limitations make it essentially unusable for anything except calling your parents: it only works on Apple devices. There are no meeting URLs to put in a calendar. It only works on Apple devices. You can't pre-schedule multi-way calls. People can phone you whenever they want (I hate that). And oh, did I mention, it only works on Apple devices.

Short answer: we use Whereby for most internal meetings, and Zoom for externally-facing meetings. We would prefer to use Whereby for everthing, if it gets a bit better.

Videoconferencing hardware

As a fully remote company, we don't have "meeting rooms," so Zoom Rooms are not a thing that makes sense for us. Which is fine, because despite what you might guess, the latency is not better with Zoom hardware than with a general purpose computing device.

We tested a few different setups looking for a good combination of latency, video quality, and reliability. It was definitely not as cool as any of Dan Luu's latency tests, but this is the apenwarr blog, not the danluu blog, and you get what you pay for. Sorry.

What we learned was:

• PCs of any sort (Linux, macOS, ChromeOS, Windows) all have higher latency than dedicated iPhone or iPad devices. (We didn't bother testing Android video latency because, well, let's be honest, it's not going to be an improvement.)

• Most (but not all) front-facing iPad cameras are not great. They're okay, but not great, especially in low light. If you have a lot of meetings, a bit better video quality is nice to have. The very latest 2019 iPad Pro has a pretty great front-facing camera that works in low light (the best kind of light), so that's what I use now.

• Older iPhones (like my aging iPhone 6S) go into CPU throttling with some video codecs, notably Whereby's, so the video quality starts off good but degrades after a few minutes when it gets hot. My new iPad Pro does not have this problem. I think my coworker has an iPhone X and also did not report it.

• You absolutely should use some sort of "personal microphone" whenever you do a call. The state of echo cancellation is pretty good now (except in Hangouts), but there's nothing a single screen-mounted microphone can do about ambiant noise. The single best favour you can do for your call partners is to use a personal microphone. Airpods include two personal microphones that work great.

• Airpods (when connected to iPhones or iPads) have very low latency, not detectable by humans. They're not any worse than a wired microphone, which is a pretty good technical achievement (one of the goals of Bluetooth 4.x I gather... but goals don't always translate into reality). Because of this ultra-low latency, they do occasionally glitch out when there's a 2.4 GHz noise burst, but it's brief and generally worth the tradeoff just to not have your head wired into your computer.

• Warning: Airpods (and all bluetooth devices) have higher and highly variable latency depending what you connect them to. macOS is definitely not perfect about latency (and tends to have worse videoconferencing performance overall, for whatever reason, than an iPad). Windows bluetooth varies from great to absymally terrible, depending mostly on the driver but also the phase of the moon. Linux bluetooth is hahahahaha sorry I forgot what I was going to say.

• I mounted my iPad above my monitor, at the so-called "selfie angle," using a $40 iPad mount I bought from Amazon. Mounting it this way has two advantages: I look slightly up at the person I'm talking to rather than down, and when I type notes into my computer, it doesn't look like I'm off to the side. This is aside from the separate benefit of using an iPad for calls: I can have the call visible at all times, without obscuring my computer desktop.

We didn't get all fancy with green screens and pro-quality microphones and all that stuff that other people talk about. Maybe it would be better, I don't know, but it definitely sounds like too much work to dump on every employee.

Short answer: iPad + Airpods + Whereby is a really great combination in 2020. And it also works well with Zoom, which is good because you're stuck with it.