Of Burn Out And Boiled Frogs

I can't speak for everyone, but as for my own experience, burn out can be tough to self diagnose because it happens slowly, by degrees.

It's a state of mind where pushing yourself even a little bit, even to do the things you'd normally do as a matter of course, feels incredibly difficult. It's that feeling of knowing you need to do something but being unable to actually complete the thoughts or actions necessary to actually get it done.

Often, at least for me, it's a result of having to perform the same set of tasks over and over again in a stressful situation.

Human brains simply weren't designed to be subjected to a constant diet of stress combined with cognitively challenging tasks and an overall lack of novelty or variation.

Understanding this can go a long way for preventing it, but even so, especially in the technology industry, it can happen to anyone.

Changing things up with your work load can sometimes help, but other times the source of the stress or conditions around your work mean that even varying your worrk may not be enough.

There's a really superlative book on this topic I can strongly recommend - Why Zebras Don't Get Ulcers.

How Does One Un-Boil a Frog?

There are a few things I have found to be helpful through the years in recovering from burn out.

Do Something Fun & DIFFERENT. Pick Up a New Hobby!

As a part of recovering from my most recent bout of burn out I've started learning how to play the piano/keyboard. A key aspect in this being effective is that it's something VERY different from the activity that got you here. For me, learning an entirely new skill that has nothing to do with computers has been satisfying and fulfilling.

Tech projects can be good too provided they're well and truly disconnected from the work that's burning you out and don't end up being a source of stress themselves!

Why not go for some long walks/hikes in the woods? Go for a nice bicycle ride? Finding fun activities that are also exercise can be good on many levels!

Give Yourself a Break

As professionals we put a ton of pressure on ourselves. Everybody wants to be a superstar and nobody likes to think about losing, even when losing a tiny and actually almost meaningless skirmish can feel like you've lost the war.

It can be difficult to zoom out and gain that perspective in the moment, but it's important to try and figure it out. Having solid support systems can be very helpful here, whether it's your spouse, a good friend, or even a therapist. Speaking of...

Get Help If You Need It

Many of us were raised to think of therapy as a thing for "crazy people" - that nebulous other that SURELY isn't us! We're stronger than that and more self sufficient.

Except, when push comes to shove, we're all human. We all need mechanisms in our lives to help provide balance and an outside perspective to keep us grounded and help avoid boiled frog syndrome.

I myself use BetterHelp and have found it incredibly useful.

Gratitude

This may sound a bit wu or perhaps irrelevant to some, but I have recently found it to be a powerful tool for shaping the way I think about things in a positive direction.

Taking a moment to zoom out from whatever I'm thinking or feeling and appreciate the fact that I live a life of such incredible privilege helps me gain some much needed perspective.

Clearly I'm not the only one, I found this Youtube video on the topic really helpful.

The Ego Trap

There's something else I want to talk about. This may not apply to you, but if it does, maybe I can help.

Many of us who work in technology take a lot of pride in our work. We may even, if we're lucky enough to go to work for what some might consider an 'elite' company, attain positions with a fair bit of cachet in the industry.

This can create yet another variety of boiled frog - the ego trap. Where your very identity becomes bound up in your job. It's not just "I work for BigCo" - it's "I AM BigCo".

I can't imagine that anyone who's fallen for this would even admit to it in the moment. It's the kind of clarity many of us only get when our situation changes out from under us and we're forced to confront an unpleasant reality.

Remember, no matter how excited you are to be working for any particular company at a particular job, at the end of the day it's JUST a job.

Cliches about health being the only thing that matters aside, this situation can create some incredibly intense tunnel vision:

"I WILL find a way to maintain the status quo!".

Thing is, maybe you Can't and moreover maybe you shouldn't! For your own sake and perhaps for that of your employer.

Sometimes, the only winning move is embracing your limitations and accepting that you can only do so much so fast.

I don't mean to say that one should just meekly accept defeat, but sometimes winning is having the wisdom to tactically retreat and formulate a better plan for success :)

Skills can be built upon and improved, and taking ownership of that process and the will to make it happen can itself be incredibly empowering!

Over the last few years I've been feeling like living exclusively in my Python programming happy place is something I can't afford to do.

Setting The Scene

It's almost impossible to ignore the rise of JavaScript in our industry. What started out as an extension project for Mozilla has grown into an ecosystem with more tentacles than an H.P. Lovecraft novel.

So while there are aspects of the language that I find very uncomfortable to work with and that seem to be a bad fit for my brain, I feel compelled to give JavaScript a fair shake and make a point of using it "in anger" in an actual project that I can ultimately get something out of beyond just learning and keep using and maintaining over time.

For the last few years I've been running my blog on Pelican a beautiful Python based static site generator which has one of the most elegant and easy to understand and modify code bases of any open source project I've ever used.

However, Pelican's minimalist DIY spirit was forcing me to come to terms with an ugly reality I was loathe to admit: I know absolutely nothing about modern web development!

Sure, I've been able to cobble together simple HTML pages for years. Nothing difficult or involved there, but there is a very long way between being able to add some HTML tags to a text file and producing a website that people will actually want to visit in 2022.

O.K. This smells like an opportunity to lean into the discomfort and grow some new skills so I can hang out a shingle worthy of my brand in full tilt 2022 adaptive web style.

Enter The Great Gatsby(JS)

On the face of it, there's a lot to like about GatsbyJS. Here are a few things I found particularly attractive going in:

• Layout will adaptively resize itself depending on the format and characteristics of the viewer's browser
• Sites written with GatsbyJS degrade gracefully and work fine even when people have JS turned off
• Site components communicate using an GraphQL back-end

So, I dove in head first and started working through the tutorial which, helpfully enough, shows the developer how to build a rather nice, feature-ful blog complete with rich tags support and an elegant navigation system.

And I've gotta say, the tutorial was a really good experience! Want to build an index page? Great! Write some JSX that queries the GraphQL back-end using a fairly elegant syntax that embeds the query right into the page. The query returns your post list which your page then renders. Hey this is pretty cool!

Here's what that first page looks like so you can get the flavor:

import * as React from 'react'
import { Link, useStaticQuery, graphql } from 'gatsby'
import {
  container,
  heading,
  navLinks,
  navLinkItem,
  navLinkText
} from './layout.module.css'

const Layout = ({ pageTitle, children }) => {
  const data = useStaticQuery(graphql`
    query {
      site {
        siteMetadata {
          title
        }
      }
    }
  `)

  return (
    <div className={container}>
      <title>{pageTitle} | {data.site.siteMetadata.title}</title>
      <header>{data.site.siteMetadata.title}</header>
      <nav>
        <ul className={navLinks}>
          <li className={navLinkItem}>
            <Link to="/" className={navLinkText}>
              Home
            </Link>
          </li>
          <li className={navLinkItem}>
            <Link to="/about" className={navLinkText}>
              About
            </Link>
          </li>
        </ul>
      </nav>
      <main>
        <h1 className={heading}>{pageTitle}</h1>
        {children}
      </main>
    </div>
  )
}

export default Layout

A couple of weeks of evening spare time (probably a couple of days of wall-clock time - bear in mind I'm teaching myself JavaScript as I do this) - I finished the tutorial and deployed my new blog. I was elated! Mission accomplished!

The Plot Thickens

The next day, after I'd triumphantly posted the URL for my shiny new redone blog to social media, the first fly appeared in the GatsbyJS soup manifested.

A friend rather understandably said "Hey, I use an RSS reader to keep up with blogs. You used to have an RSS feed. What's the URL for the new site?"

Uh. Woops. Truth be told I hadn't even thought about that, and the tutorial doesn't mention it. That's pretty understandable too, right? You don't want to overwhelm people who are trying to adopt your technology with a firehose of details to master when they're just starting out.

OK. Let's bring up GatsbyJS's (up to this point) excellent documentation and see how we add an RSS feed...

The Abstractions Begin To Crumble

Right then. I find a page describing how to add an RSS feed to your blog, Just add this handy dandy plugin to your site. This will let you do a similar GraphQL query to the one you used for your post list, only this time you'll be generating your RSS feed. Here we go!

Add the plugin and BOOM! My site build fails with an utterly inscrutable Webpack error. Why? Ah the plugin was for a prior version of GatsbyJS and isn't compatible with the one you used to build your site. OK, let's just use that older version of GatsbyJS instead!

Well crud. Turns out that the tutorial was written using MDX, a JSX flavored version of the popular Markdown mark-up language for React JSX pages, and the plugin used to provide MDX support both doesn't work with the RSS feed plugin and doesn't run in the prior version of GatsbyJS.

I said I wanted to learn. Let's roll up our sleeves and see about porting that plugin to the new GatsbyJS version.

Three agonizing weeks of pointless struggle later...

The Knee Bone DOESN'T Connect To The Shin Bone

One of the things that drew me to GatsbyJS to begin with was this beautiful abstraction: pages, plugins, and components all seamlessly interpolating with a GraphQL back end.

And it does work that way to an extent. All the plugins do indeed communicate with the rest of your system using GraphQL queries, but because the plugin architecture doesn't seem to enforce any interoperability rules, and also because successive GatsbyJS versions introduce radical breaking changes to the plugin API, you end up with a programmer who was sold a glorious GraphQL future but winds up holding a rather large bucket of bolts that refuse to work together to build a meaningful whole.

To Be Fair

I realize I'm painting a rather unappealing picture of GatsbyJS here, so I just want to say that it became a rather popular JavaScript framework a year or so back with good reason. There are many many competent programmers out there deploying beautiful sites that scale using these tools.

I also recognize that building abstractions that don't leak and that seamlessly work well together is incredibly challenging. I just feel that ultimately GatsbyJS isn't the tool for me and also that perhaps sme of the marketing should be adjusted to reflect reality on the ground.

It also needs to be said that GatsbyJS is 100% FLOSS, and as such is a gift freely given by its implementors which I very much appreciate. None of what I write here changes that in any way.

I should also say that my inability to build with it is as much due to my own lack of JavaScript acumen as it is with the tool itself.

These are just my opinions and personal experiences and thus should be taken with a grain (or perhaps a shaker) of salt, especially if you're a seasoned JavaScript developer.

The Happy Ending (But Not With GatsbyJS!)

Ultimately, I decided that the best thing I could possibly do for myself with this blog project was to go with what I know, so I chose Nikola - a superlative Python powered static site generator, and I'm just delighted with that choice.

I'm still planning on finding something to use JavaScript for, it's just not going to be this particular task :)

I'm switching the static site generator I ues for this blog from Pelican to Nikola.

There are a number of reasons for this - better/more theme options, a more active development community, and some really swank features I just love.

The up-shot however is that the RSS feeds will need to change. The new links are clearly signposted on the header at the top of the blog.

Also, I've switched the comment system from Disqus (Which I really Dis-like :) to Intense Debate which I've been using on my personal blog and love to bits.

You can find the old Disqus comments here

Thanks for understanding and updating your feed readers!

Hi all!

Just a brief house-keeping note.

This is going to hence-forth be strictly my professional blog.

I'll be cleaning out old posts that aren't tech related and maybe moving any I feel particularly inclined to keep over to my new personal blog, Playing With Quicksilver.

I'm not trying to hide anything, so feel free to browse my old posts via the Wayback Machine over here.

I look forward to blogging here more often! I'll likely also be switching to the Nikola static site generator because it offers a number of features and creature comforts that will make my life easier (Like having a command that will pre-seed a new post with all the right metadata fields. Yay :)

As always feel free to hit me up here or on social media if you have any thoughts.

Onward!

Don't Despair! This Is Gonna Be Easy! :)

Over the last few months I've seen a ton of confusion swirling around social media about how to help people who are new to Python development and software development in general get up and running on Windows quickly.

GOOD NEWS! You can ignore all that. Many of these suggestions are well meaning but come from people who don't actually do Python development on Windows regularly.

Note that I'm not going to tell you the 5 different ways to do a thing, I'm going to tell you the ONE easy way :)

So let's get started!

Side Note: Everyone Will Tell You To Use WSL - You Probably Don't Want That!

I hear a lot of incredibly talented and experienced well meaning UNIX/Mac enthusiasts tell newcomers looking to get going on Windows "Oh well you just use WSL! It's easy! Then you install python with apt get yada yada, and then you install pyenv, and then you configure your shell's environment to use the right shims... and..."

The truth is that if you want to learn Linux - you SHOULD! It's an incredible platform and a virtually bottomless font of power waiting for you to tap.

But if there's one thing I've learned in 30 years in the tech trenches, it's that trying to learn a million things all at once is the surest way to learn nothing. Why not focus on teaching yourself Python programming first and then exploring Linux as a separate effort?

Install Python

If you're on Windows 11 (And I recommend it if your hardware is new enough!) this is as simple as bringing up a Powershell in your swank Windows Terminal. If you want a look at my setup please go read my article on that.

Anyway, all you have to do is type:

winget install python

That's it! This will install the latest stable version of Python from The official Python website. If you're not lucky enough to be running Windows 11, just download the latest stable version from the website's Downloads page and double click on the executable file in your Downloads folder. If you're not sure just keep clicking Next on the dialogs and in a few minutes you'll have Python installed!

Embracing The Power of Py! (Psst. This Is The Easy Button!)

The 'py' Python launcher used to be a super power that only Windows versions of Python shipped with, but now thanks to Brett Cannon, an amazingly prolific Python core developer's hard work our UNIX friends can enjoy the easy button as well!

So let's get this party started and see how easy it is these days to get into the Python prompt:

Pretty easy, right?

Note that py has other superpowers as well. It can list all the python versions you have installed, and also let you choose which version to run with the arguments you choose. Run:

py --help

to see an exhaustive description of all the magic tricks py can do :)

Set The Stage

Again, from your Powershell, create a project directory for yourself. Programmers often create a 'src' directory wherever it's convenient (Maybe in your home directory?) and then put their projects underneath.

So let's go ahead and do that:

CD $HOME
mkdir src\starwars_characters

# Yes i'm a big nerd. Our project will pull down a list of characters in the movies from the Star Wars API :)

cd src\starwars_characters

Building A Sandbox

Now let's create a safe space for our project so we don't unintentionally install libraries or other experiments in public. This can lead to situations where you go to create another project but a library you installed previously either causes undesirable behavior or perhaps is even in a broken state, getting in the way of you getting your work done.

We're going to create a thing called 'virtual environment'. Don't worry too much about the details, just think about this as a nice tidy sandbox for us to play in where we don't have to worry about breaking anything outside.

Again from our Powershell prompt:

# Actually create our virtual environment. Put the bits in a folder called 'venv'.
# Most Pythonistas use this folder name, but you can use any name you want :)
py -m venv venv

# Actually activate our new virtual environment in our Powershell
.\venv\Scripts\Activate.ps1

Let's see what happens when we do that (Assuming you're running Oh My Posh, and you should be if you followed my setup article):

See what happens there? We create the virtual environment, and our shell notices which Python version it uses, and tells us that in our Powershell prompt!

Then, we activate our virtual environment, and now our prompt adds a (venv) at the beginning. With this visual indicator in plce, we know we're safely ensconced in our sandbox and can start playing with no worries!

Install Necessary Libraries

You'll need an editor or an IDE. I use Visual Studio Code. It's free, it runs on any modern computer, and it offers an incredibly rich set of tools to help you with your development. Bonus points? It's also free, and even largely open source.

Let's get started, and tell Python which libraries we'll need for our project:

code requirements.txt

In this file, add a single line:

requests

If you want to add a truly superlative interactive Python environment with which to prototype and explore, you could also add IPython:

IPython

(Note IPython here is case sensitive so be careful!)

Now, still in our Powershell prompt, type this command to actually install these required libraries into our virtual environment:

py -m pip install -r requirements.txt

You should see some messages about libraries being installed. Note that you invoke IPython this exact same way:

py -m IPython

Write The Code

Create a file in your project directory called starwars_characters.py with your editor that includes the following code:

import requests

STARWARS_API_URL_BASE = "https://swapi.dev/api"
GET_CHARACTERS_URL = f"{STARWARS_API_URL_BASE}/people"

response = requests.get(GET_CHARACTERS_URL)

response_json = response.json()

characters = response_json['results']

for character in characters:
    print(f"Character Name: {character['name']}")

Here's what my Visual Studio Code session looks like with this code loaded and the output shown in the run window:

Notice that bit at the bottom that says 3.10? That shows us that Visual Studio Code automatically detected the Python we have installed in our virtualenv. You can click on that to see a list of installed Python interpreters to choose from.

We can run our new project either by using the Run menu inside Visual Studio Code or on the command line with that very same py syntax we used before:

py starwars_characters.py

The End Is Just The Beginning

Well there we have it! You have Python installed and running, and you've built a small but interesting Python project right here on your Windows machine!

The truth is, a large number of people have put in innumerable hours making Python drive smooth like a Cadillac on Windows, so if you find yourself using and enjoying it, the next time someone tells you to abandon all hope and install Linux/WSL, be sure to point them at this article! Maybe if we change some minds we can help bring more Pythonistas into the incredibly awesome Python community!

Please feel free to leave comments with any questions, or hit me up on Mastodon or Twitter.

I'd love to hear from you!

[06/08/2021 Update: Added a bit about Windows Terminal. Dunno how I forgot that the first time around!]

As I've written about previously and elsewhere, I felt so badly burned by Apple's laptop hardware design decisions of a few years ago that I've rather fallen out of love with that platform for my personal work. The latest hardware is much better, but I feel like the message has been sent and received, so I'm not rushing back any time soon.

My first choice was the Linux desktop, and after months of struggling, instability and accessibility issues I'll admit I've been looking for stable, solid alternatives that are also powerful enough to get the job done and maybe even have something new to offer. As an old dog, sometimes it's really nice to be taught some new tricks!

If you've tangoed with Windows in the past, and found yourself struggling against its rather byzantine UI, I'd urge you to read on and see if maybe it's not time for another careful look.

Since choosing the right tools is all about your unique set of needs, I'll use those as categories to drive the discussion and showcase how Windows is doing a great job of satisfying them.

In The Beginning, There Was The Command Line!

One of the things that first drew me to UNIX decades ago was the shell. The ability to get pretty much ANYTHING I needed done by typing a few commands, and to create incredibly powerful mechanisms by stringing them together is downright intoxicating.

I still love me some Bourne shell, it's just as powerful today as it was decades ago. However Windows Powershell has some incredibly powerful paradigms of its own that are worth looking into.

In addition to things like object pipelines and desired state configuration that the UNIX shell simply can't do, modern Powershell has creature comforts like Oh My Posh that make for a very comfortable work environment. Here's a screenshot of mine:

You can see that I get all kinds of nice status like exit code of last command, current git branch, current Python version and virtual environment, etc.

While Powershell is indeed a totally different beast from the UNIX shell I'm used to, its built in help for every command and understandable syntax make the learning curve feel rather shallow indeed.

Another gripe most UNIX users have when coming to Windows is that everything requires mousing around and futzing with a GUI, and it can feel like you're trapped in an endless series of installer screens to get even your basic working tools installed. Modern Windows obviates all that with winget a full fat package manager with tons of common applications already in there. Just winget install git and you're off to the races :)

Before The Command Line, There Was The Terminal

As anyone who's worked with Windows for a long time could tell you, in the old days, the Windows Console was pretty primitive. You were basically dealing with MS-DOS CMD.EXE with the barest modicum of window dressing. Needless to say, this is 2021 and that's just not cutting it.

Thankfully, there is a superlative alternative that's really come into its own in 2021 - Windows Terminal. Finally all the features anyone who's been working in the UNIX world have come to expect over the last 30 years are available on Windows as well. Multiple tabs, excellent terminal emulation, great theming and color support, and the ability to seamlessly handle different shells per tab. I regularly keep a Powershell and a WSL tab open for my work.

If you've ever struggled along with the old CMD.EXE, fighting for every cut, paste or insert, this really is a game changer.

I Hate Meeces To Pieces! (Really, Don't Make Me Use the Mouse!)

As someone who's partially blind and has fine and gross motor impairment, using the standard Windows Icons Mice and Pointers user interface is downright agonizing. I need to be able to launch apps with just a few keystrokes. On the Mac, Alfred fulfills this role admirably and adds a whole host of next level productivity power ups in addition to keyboard app launching.

A very recent addition on the Windows side that seems equally powerful and easy is Microsoft Powertoys Powertoys Run feature. I can launch apps, bind keys to actions, and access system functions like sleep, restart and the like all 100% from the keyboard. Nice!

Another Powertoys addition that's frosting my Pop-tarts is Powertoys Keyboard Manager which allows me to remaps the dread CAPS LOCK key to Ctrl so it's right next to my pinky as God Intended :)

This simple enhanment has been impossible with Windows forever without resorting to more drastic measures, so having it work pretty much out of box now is super sweet.

Python - It Takes a Cast of Thousands to Raise a Platform

If you'd uttered the words "Python" and "Windows" in the same sentence a few years back, the average Pythonista would have rolled their eyes or groaned and then proceeded to tell you a war story about how they had to spend weeks trying to get some critical module or other working.

Those days are just straight up gone. Microsoft and the Python community at large have poured countless engineer hours into making the Python ecosystem a truly first class experience under Windows. Virtual environments, poetry, and even tools for installing and managing command line scripts that integrate with the shell like pipx work just great under Powershell on Windows.

And if by chance you're coding something up that's native UNIX to its very core, Windows *still has you covered. Cue the Windows Subsystem for Linux!

WSL - I See Penguins!

With the avent of WSL2 and most recently WSL-g you really do have everything you need to seamlessly build, debug and run Linux programs on Windows, including those that require X/Wayland GUI support or sound.

It's not just some kind of compatibility shim, it's an actual Linux kernel running in Windows. The integration is so complete these days that you can run things like Docker within WSL successfully. That's a really big deal!

Also, most of the popular mainstream development tools like Pycharm and Visual Studio Code have superlative support for deploying to and debugging in WSL built in.

The amount and quality of work Microsoft has put into this is truly impressive. And unlike the Mac, you're not actually using some oddball FreeBSD user space running atop the Mach microkernel, you're running 100% Linux. It's turtles all the way down.

I Go Where The Innovation Is & Use What Works

As a technologist, I'm proud of the success I've had in sniffing out pockets of high velocity, high value innovation in the technology landscape.

Years ago, I was running Linux when many people were either using DOS, pre OSX MacOS or super expensive prorietary vendor workstations like Sun's Sparc. Being able to have an honest to god UNIX environment on commodity hardware was like Prometheus's gift of fire to the mortals. It really was that big a deal.

I still think the Linux desktop has incredible potential, but for me the obstacles imposed by my disabilities and the lack of accessibility features can be a real buzz and productivity killer.

So here I am embracing a desktop that works, with an ecosystem that works and a rather large well heeled company pouring resources into bar raising and innovation.

I look forward to exploring and leveraging the incredible potential this platform represents. Stay tuned for more posts as I go!

I was recently in an online discussion about whether or not technologists should use "managed services" like the one I work for.

One gent essentially claimed that the move towards cloud and managed services represented an end game for our industry, and that we'd automate ourselves out of a job. This is what I said:

Will there be job reductions and resultant economic pain as a result of the move away from the data center and towards more cloud/managed services? Hell yes there will be!

Those who want to work in this business will find ways to adapt and continue to add value.

I’ve been working in technology for about 32 years. This doesn’t make me magical or omniscient or even smart but what it does make me is a witness to several waves of seismic change.

I came into the job market in the early 90s when DEC was imploding, PCs were ascendant, and the large expensive workstations vendors were dead companies walking but didn’t realize it yet.

Everyone thought the world would end and we’d all be out of work too.

Will it be hard? Yes. Will there be a lot of people who can’t or don’t want to adapt and will be left behind? Undoubtedly. Is that bad? Yes. However it’s the way our industry works and has since its inception.

We are all turning the crank that drives the flywheel of increased automation, whether we administer our own database clusters or not. The move to managed and cloud definitely represents a notable pinch point, and we’ll see how painful the transition will be, but it’s one paradigm shift in an industry that creates paradigm shifts for a living.

I’ve actually thought for a while that in the longer term, as compute at scale becomes quite a bit smaller and even cheaper, we could see a move back away from cloud because when you can host your company’s compute cluster in a cube the size of a printer and we have algorithms that can encode and enact services at scale in an operationally safe way, the value cloud adds will dwindle.

I love my job, and I love this industry, and plan to continue playing in this pool until I die, whether someone’s willing to pay me for it or not.

I assert that the constant negativity many of us exhibit is neither necessary nor desirable and represents a kind of immaturity that we will eventually grow out of as our industry continues to mainstream.

We’ll see.

[caption id="attachment_498" align="aligncenter" width="300"] A Musical Mind[/caption]

A coworker at my previous job once said in response to my whining a bit about having just lost 700 songs from my digital music library "Let go of your music collection!".  That got me thinking, long and hard.

Why?

Why am I lugging around this corpus of several thousand compact discs? Why do I have several shelves worth of DVDs? Why don't I simply let go of the whole idea of media ownership and just drink from the infinite fire-hose that is the internet?

Because, when push comes to shove, my media is part of me.  My CD collection represents innumerable fond memories of places, people and moments I've moved on from but that inhabit special places in my heart and mind.

You may have heard the theory that when one sense is diminished, the brain compensates by heightening another. I'm partially blind, so many if not most of my fond memories have a soundtrack. Being able to listen to that particular album and feel some semblance of that moment wash over me is like having a personal time machine.

Oh hey, there's that Autechre album, not available anymore anywhere as far as I can tell. That was one of the first times ever that I was really turned on by electronica. Music plays in my mind's eye - one of my best friends and I walking into a record shop on Newbury Street. And there's that Boiled In Lead CD I got the day after I was invited to an incredibly cool private concert at a farm in NH by a friend who's since passed away. Neither of these things is on Spotify near as I can tell. If I let go, I lose them, and an aspect of those fond memories along with them.

But I'm getting older - in fact I turn 50 in a few months. Having moved countless times I have begun to feel the weight of my possessions. I recognize that I am privileged in that I can make this choice. Not everyone is lucky enough to be able to buy a NAS (Network Attached Storage) and spend their leisure time converting their media collection into a giant bag of bits. I count myself lucky.

So, now that you've heard way more than you ever wanted to know about why I chose to embark upon this epic project, let's talk turkey in case you're thinking about doing this for yourself.

Preserving all your media in virtual form can create a veritable labyrinth of choices. As with all things, I suggest keeping it simple. I'll tell you what my choices were and maybe even offer another option or two along the way, but mostly I'll be guiding you through my process as it has evolved for my purposes.

First let's answer a basic question: What kind of media do you want to preserve? In my case the answer is simple. My collection consists entirely of CDs and DVDs, as well as a handful of Blu-ray discs.

The first big choice - to compress or not to compress? Compressing your media will save you disk space, but cost you in terms of fidelity. The movies you rip won't look exactly as they did on disc, and the music you rip won't be exactly as it was in its original form either.

I will not go down the rabbit hole of trying to convince you of any particular absolute truth, but my ultimate destination ended up being "I never want to have to do this again and I want no loss in quality".

So, on the video side, I don't use any compression at all. I rip my movies straight to Matroska video or ".mkv" files. This format is incredibly rich and versatile, and also has the advantage of being open source. Why do I care one might ask? See above - I NEVER want to do this again :) So I don't want to be beholden to some company or other deciding that the format I chose is changing, going away, or under license dispute. These are my bits embodying my legally purchased media. Full stop.

On the audio side, I was able to take advantage of an excellent format called FLAC. It provides some compression so as to save space, but most important for me is the fact that it's lossless, so I lose no fidelity at all in my ripped copy. This format is also open source, and the author is a fellow Somervillian :)

Having made the important decisions about format, let's choose appropriate tools.

For the video side, I chose MakeMKV. It's cross-platform (Mac, Linux, Windows), has a very straight forward user interface, and is dirt cheap. You can download it for free, but to rip Blu-ray discs, you have to pay them $5 or $10 to unlock the decoder. Trust me, for the power this tool offers, give them their money. They've earned it. This was one of the only tools I could find that made ripping both Blu-ray and DVD discs easy while also giving me the flexibility I needed. A number of lesser tools I tried had nice user interfaces, but were rife with bugs. making it nearly impossible to extract the raw bits with no compression.

Another critical tool if you want to rip TV series from DVD is a little thing called FileBot. It will rename the files to conform to what Plex requires.

For the rare case where multiple episodes in a TV show are glommed into one big file, I chose MKVToolNix to split .mkv files into usable chunks  I'm unsure whether it's worth the effort, especially for series with lots of episodes.

On the audio side, I used XLD for the Mac. It's on par with Xact Audio Copy for Windows, the most highly regarded audio ripper out there near as I can tell. It rips to FLAC effortlessly and the developer maintains a very high quality checksum database of every CD that's been ripped with the tool so you can be sure that your results are crystal clear and bit perfect. I've had Linux folks ask about something similar for them, and I haven't been able to find anything yet. Sorry about that. If someone finds something that does similar things and has a nice UI, do let me know.

You'll also need a place to put the resulting bits. A really safe place. This is a labor intensive effort, and you really don't want to lose all your countless hours of work due to a hard drive failure.

I chose a Synology DS216+ii for the task. It's a nice little box that holds two big hard drives that you can run as a RAID set. In addition to having hardware that meets my needs, I chose these folks because the software the NAS runs is easy to administer, incredibly flexible, and has a rich ecosystem. One great feature I've not seen elsewhere is with just a few clicks, you can back up your data to cloud services like Glacier. Synology has a proprietary suite of software you can use to manage your media. I chose Plex because it's feature rich and for a lifetime fee will automatically transcode movies and music to work with any viewing or listening device.

In the next installment we'll start getting into the nitty-gritty of how to get the job done. The good, the bad, and the ugly!

Working for Amazon has been a revelation to me in ways large and small.

One of the biggest by far has been coming to the understanding that career growth was not in fact laying in wait around the next corner if only I could master the latest technology trend. There are baseline skills that can either accelerate or retard your growth in profound ways, and many of them reduce down to changing the way you think in or react to various situations. Let's walk through a few of them.