Mis-Adventures In GatsbyJS

Key West Octopus by oe Parks is licensed under CC BY-NC 2.0

Over the last few years I've been feeling like living exclusively in my Python programming happy place is something I can't afford to do.

Setting The Scene

It's almost impossible to ignore the rise of JavaScript in our industry. What started out as an extension project for Mozilla has grown into an ecosystem with more tentacles than an H.P. Lovecraft novel.

So while there are aspects of the language that I find very uncomfortable to work with and that seem to be a bad fit for my brain, I feel compelled to give JavaScript a fair shake and make a point of using it "in anger" in an actual project that I can ultimately get something out of beyond just learning and keep using and maintaining over time.

For the last few years I've been running my blog on Pelican a beautiful Python based static site generator which has one of the most elegant and easy to understand and modify code bases of any open source project I've ever used.

However, Pelican's minimalist DIY spirit was forcing me to come to terms with an ugly reality I was loathe to admit: I know absolutely nothing about modern web development!

Sure, I've been able to cobble together simple HTML pages for years. Nothing difficult or involved there, but there is a very long way between being able to add some HTML tags to a text file and producing a website that people will actually want to visit in 2022.

O.K. This smells like an opportunity to lean into the discomfort and grow some new skills so I can hang out a shingle worthy of my brand in full tilt 2022 adaptive web style.

Enter The Great Gatsby(JS)

On the face of it, there's a lot to like about GatsbyJS. Here are a few things I found particularly attractive going in:

  • Layout will adaptively resize itself depending on the format and characteristics of the viewer's browser
  • Sites written with GatsbyJS degrade gracefully and work fine even when people have JS turned off
  • Site components communicate using an GraphQL back-end

So, I dove in head first and started working through the tutorial which, helpfully enough, shows the developer how to build a rather nice, feature-ful blog complete with rich tags support and an elegant navigation system.

And I've gotta say, the tutorial was a really good experience! Want to build an index page? Great! Write some JSX that queries the GraphQL back-end using a fairly elegant syntax that embeds the query right into the page. The query returns your post list which your page then renders. Hey this is pretty cool!

Here's what that first page looks like so you can get the flavor:

import * as React from 'react'
import { Link, useStaticQuery, graphql } from 'gatsby'
import {
  container,
  heading,
  navLinks,
  navLinkItem,
  navLinkText
} from './layout.module.css'

const Layout = ({ pageTitle, children }) => {
  const data = useStaticQuery(graphql`
    query {
      site {
        siteMetadata {
          title
        }
      }
    }
  `)

  return (
    <div className={container}>
      <title>{pageTitle} | {data.site.siteMetadata.title}</title>
      <header>{data.site.siteMetadata.title}</header>
      <nav>
        <ul className={navLinks}>
          <li className={navLinkItem}>
            <Link to="/" className={navLinkText}>
              Home
            </Link>
          </li>
          <li className={navLinkItem}>
            <Link to="/about" className={navLinkText}>
              About
            </Link>
          </li>
        </ul>
      </nav>
      <main>
        <h1 className={heading}>{pageTitle}</h1>
        {children}
      </main>
    </div>
  )
}

export default Layout

A couple of weeks of evening spare time (probably a couple of days of wall-clock time - bear in mind I'm teaching myself JavaScript as I do this) - I finished the tutorial and deployed my new blog. I was elated! Mission accomplished!

The Plot Thickens

The next day, after I'd triumphantly posted the URL for my shiny new redone blog to social media, the first fly appeared in the GatsbyJS soup manifested.

A friend rather understandably said "Hey, I use an RSS reader to keep up with blogs. You used to have an RSS feed. What's the URL for the new site?"

Uh. Woops. Truth be told I hadn't even thought about that, and the tutorial doesn't mention it. That's pretty understandable too, right? You don't want to overwhelm people who are trying to adopt your technology with a firehose of details to master when they're just starting out.

OK. Let's bring up GatsbyJS's (up to this point) excellent documentation and see how we add an RSS feed...

The Abstractions Begin To Crumble

Right then. I find a page describing how to add an RSS feed to your blog, Just add this handy dandy plugin to your site. This will let you do a similar GraphQL query to the one you used for your post list, only this time you'll be generating your RSS feed. Here we go!

Add the plugin and BOOM! My site build fails with an utterly inscrutable Webpack error. Why? Ah the plugin was for a prior version of GatsbyJS and isn't compatible with the one you used to build your site. OK, let's just use that older version of GatsbyJS instead!

Well crud. Turns out that the tutorial was written using MDX, a JSX flavored version of the popular Markdown mark-up language for React JSX pages, and the plugin used to provide MDX support both doesn't work with the RSS feed plugin and doesn't run in the prior version of GatsbyJS.

I said I wanted to learn. Let's roll up our sleeves and see about porting that plugin to the new GatsbyJS version.

Three agonizing weeks of pointless struggle later...

The Knee Bone DOESN'T Connect To The Shin Bone

One of the things that drew me to GatsbyJS to begin with was this beautiful abstraction: pages, plugins, and components all seamlessly interpolating with a GraphQL back end.

And it does work that way to an extent. All the plugins do indeed communicate with the rest of your system using GraphQL queries, but because the plugin architecture doesn't seem to enforce any interoperability rules, and also because successive GatsbyJS versions introduce radical breaking changes to the plugin API, you end up with a programmer who was sold a glorious GraphQL future but winds up holding a rather large bucket of bolts that refuse to work together to build a meaningful whole.

To Be Fair

I realize I'm painting a rather unappealing picture of GatsbyJS here, so I just want to say that it became a rather popular JavaScript framework a year or so back with good reason. There are many many competent programmers out there deploying beautiful sites that scale using these tools.

I also recognize that building abstractions that don't leak and that seamlessly work well together is incredibly challenging. I just feel that ultimately GatsbyJS isn't the tool for me and also that perhaps sme of the marketing should be adjusted to reflect reality on the ground.

It also needs to be said that GatsbyJS is 100% FLOSS, and as such is a gift freely given by its implementors which I very much appreciate. None of what I write here changes that in any way.

I should also say that my inability to build with it is as much due to my own lack of JavaScript acumen as it is with the tool itself.

These are just my opinions and personal experiences and thus should be taken with a grain (or perhaps a shaker) of salt, especially if you're a seasoned JavaScript developer.

The Happy Ending (But Not With GatsbyJS!)

Ultimately, I decided that the best thing I could possibly do for myself with this blog project was to go with what I know, so I chose Nikola - a superlative Python powered static site generator, and I'm just delighted with that choice.

I'm still planning on finding something to use JavaScript for, it's just not going to be this particular task :)

A Teency Bit More Housekeeping

I'm switching the static site generator I ues for this blog from Pelican to Nikola.

There are a number of reasons for this - better/more theme options, a more active development community, and some really swank features I just love.

The up-shot however is that the RSS feeds will need to change. The new links are clearly signposted on the header at the top of the blog.

Also, I've switched the comment system from Disqus (Which I really Dis-like :) to Intense Debate which I've been using on my personal blog and love to bits.

You can find the old Disqus comments here

Thanks for understanding and updating your feed readers!

Some Brief But Important House Keeping

Hi all!

Just a brief house-keeping note.

This is going to hence-forth be strictly my profeesional blog.

I'll be cleaning out old posts that aren't tech related and maybe moving any I feel particularly inclined to keep over to my new personal blog, Playing With Quicksilver.

I'm not trying to hide anything, so feel free to browse my old posts via the Wayback Machine over here.

I look forward to blogging here more often! I'll likely also be switching to the Nikola static site generator because it offers a number of features and creature comforts that will make my life easier (Like having a command that will pre-seed a new post with all the right metadata fields. Yay :)

As always feel free to hit me up here or on social media if you have any thoughts.

Onward!

Getting Started With Python On Windows 2021 Edition: Push The Easy Button!

Don't Despair! This Is Gonna Be Easy! :)

Over the last few months I've seen a ton of confusion swirling around social media about how to help people who are new to Python development and software development in general get up and running on Windows quickly.

GOOD NEWS! You can ignore all that. Many of these suggestions are well meaning but come from people who don't actually do Python development on Windows regularly.

Note that I'm not going to tell you the 5 different ways to do a thing, I'm going to tell you the ONE easy way :)

So let's get started!

Side Note: Everyone Will Tell You To Use WSL - You Probably Don't Want That!

I hear a lot of incredibly talented and experienced well meaning UNIX/Mac enthusiasts tell newcomers looking to get going on Windows "Oh well you just use WSL! It's easy! Then you install python with apt get yada yada, and then you install pyenv, and then you configure your shell's environment to use the right shims... and..."

The truth is that if you want to learn Linux - you SHOULD! It's an incredible platform and a virtually bottomless font of power waiting for you to tap.

But if there's one thing I've learned in 30 years in the tech trenches, it's that trying to learn a million things all at once is the surest way to learn nothing. Why not focus on teaching yourself Python programming first and then exploring Linux as a separate effort?

Install Python

If you're on Windows 11 (And I recommend it if your hardware is new enough!) this is as simple as bringing up a Powershell in your swank Windows Terminal. If you want a look at my setup please go read my article on that.

Anyway, all you have to do is type:

winget install python

That's it! This will install the latest stable version of Python from The official Python website. If you're not lucky enough to be running Windows 11, just download the latest stable version from the website's Downloads page and double click on the executable file in your Downloads folder. If you're not sure just keep clicking Next on the dialogs and in a few minutes you'll have Python installed!

Embracing The Power of Py! (Psst. This Is The Easy Button!)

The 'py' Python launcher used to be a super power that only Windows versions of Python shipped with, but now thanks to Brett Cannon, an amazingly prolific Python core developer's hard work our UNIX friends can enjoy the easy button as well!

So let's get this party started and see how easy it is these days to get into the Python prompt:

Windows Python Prompt With Py

Pretty easy, right?

Note that py has other superpowers as well. It can list all the python versions you have installed, and also let you choose which version to run with the arguments you choose. Run:

py --help

to see an exhaustive description of all the magic tricks py can do :)

Set The Stage

Again, from your Powershell, create a project directory for yourself. Programmers often create a 'src' directory wherever it's convenient (Maybe in your home directory?) and then put their projects underneath.

So let's go ahead and do that:

CD $HOME
mkdir src\starwars_characters

# Yes i'm a big nerd. Our project will pull down a list of characters in the movies from the Star Wars API :)

cd src\starwars_characters

Building A Sandbox

Now let's create a safe space for our project so we don't unintentionally install libraries or other experiments in public. This can lead to situations where you go to create another project but a library you installed previously either causes undesirable behavior or perhaps is even in a broken state, getting in the way of you getting your work done.

We're going to create a thing called 'virtual environment'. Don't worry too much about the details, just think about this as a nice tidy sandbox for us to play in where we don't have to worry about breaking anything outside.

Again from our Powershell prompt:

# Actually create our virtual environment. Put the bits in a folder called 'venv'.
# Most Pythonistas use this folder name, but you can use any name you want :)
py -m venv venv

# Actually activate our new virtual environment in our Powershell
.\venv\Scripts\Activate.ps1

Let's see what happens when we do that (Assuming you're running Oh My Posh, and you should be if you followed my setup article):

Create/Activate Virtual Environment

See what happens there? We create the virtual environment, and our shell notices which Python version it uses, and tells us that in our Powershell prompt!

Then, we activate our virtual environment, and now our prompt adds a (venv) at the beginning. With this visual indicator in plce, we know we're safely ensconced in our sandbox and can start playing with no worries!

Install Necessary Libraries

You'll need an editor or an IDE. I use Visual Studio Code. It's free, it runs on any modern computer, and it offers an incredibly rich set of tools to help you with your development. Bonus points? It's also free, and even largely open source.

Let's get started, and tell Python which libraries we'll need for our project:

code requirements.txt

In this file, add a single line:

requests

If you want to add a truly superlative interactive Python environment with which to prototype and explore, you could also add IPython:

IPython

(Note IPython here is case sensitive so be careful!)

Now, still in our Powershell prompt, type this command to actually install these required libraries into our virtual environment:

py -m pip install -r requirements.txt

You should see some messages about libraries being installed. Note that you invoke IPython this exact same way:

py -m IPython

Write The Code

Create a file in your project directory called starwars_characters.py with your editor that includes the following code:

import requests

STARWARS_API_URL_BASE = "https://swapi.dev/api"
GET_CHARACTERS_URL = f"{STARWARS_API_URL_BASE}/people"

response = requests.get(GET_CHARACTERS_URL)

response_json = response.json()

characters = response_json['results']

for character in characters:
    print(f"Character Name: {character['name']}")

Here's what my Visual Studio Code session looks like with this code loaded and the output shown in the run window:

Starwars Project VSCode Screenshot

Notice that bit at the bottom that says 3.10? That shows us that Visual Studio Code automatically detected the Python we have installed in our virtualenv. You can click on that to see a list of installed Python interpreters to choose from.

We can run our new project either by using the Run menu inside Visual Studio Code or on the command line with that very same py syntax we used before:

py starwars_characters.py

The End Is Just The Beginning

Well there we have it! You have Python installed and running, and you've built a small but interesting Python project right here on your Windows machine!

The truth is, a large number of people have put in innumerable hours making Python drive smooth like a Cadillac on Windows, so if you find yourself using and enjoying it, the next time someone tells you to abandon all hope and install Linux/WSL, be sure to point them at this article! Maybe if we change some minds we can help bring more Pythonistas into the incredibly awesome Python community!

Please feel free to leave comments with any questions, or hit me up on Mastodon or Twitter.

I'd love to hear from you!

2021 - The Year Windows Became a First Class Python Development Environment

[06/08/2021 Update: Added a bit about Windows Terminal. Dunno how I forgot that the first time around!]

As I've written about previously and elsewhere, I felt so badly burned by Apple's laptop hardware design decisions of a few years ago that I've rather fallen out of love with that platform for my personal work. The latest hardware is much better, but I feel like the message has been sent and received, so I'm not rushing back any time soon.

My first choice was the Linux desktop, and after months of struggling, instability and accessibility issues I'll admit I've been looking for stable, solid alternatives that are also powerful enough to get the job done and maybe even have something new to offer. As an old dog, sometimes it's really nice to be taught some new tricks!

If you've tangoed with Windows in the past, and found yourself struggling against its rather byzantine UI, I'd urge you to read on and see if maybe it's not time for another careful look.

Since choosing the right tools is all about your unique set of needs, I'll use those as categories to drive the discussion and showcase how Windows is doing a great job of satisfying them.

In The Beginning, There Was The Command Line!

One of the things that first drew me to UNIX decades ago was the shell. The ability to get pretty much ANYTHING I needed done by typing a few commands, and to create incredibly powerful mechanisms by stringing them together is downright intoxicating.

I still love me some Bourne shell, it's just as powerful today as it was decades ago. However Windows Powershell has some incredibly powerful paradigms of its own that are worth looking into.

In addition to things like object pipelines and desired state configuration that the UNIX shell simply can't do, modern Powershell has creature comforts like Oh My Posh that make for a very comfortable work environment. Here's a screenshot of mine:

My Oh My Posh Prompt

You can see that I get all kinds of nice status like exit code of last command, current git branch, current Python version and virtual environment, etc.

While Powershell is indeed a totally different beast from the UNIX shell I'm used to, its built in help for every command and understandable syntax make the learning curve feel rather shallow indeed.

Another gripe most UNIX users have when coming to Windows is that everything requires mousing around and futzing with a GUI, and it can feel like you're trapped in an endless series of installer screens to get even your basic working tools installed. Modern Windows obviates all that with winget a full fat package manager with tons of common applications already in there. Just winget install git and you're off to the races :)

Before The Command Line, There Was The Terminal

As anyone who's worked with Windows for a long time could tell you, in the old days, the Windows Console was pretty primitive. You were basically dealing with MS-DOS CMD.EXE with the barest modicum of window dressing. Needless to say, this is 2021 and that's just not cutting it.

Thankfully, there is a superlative alternative that's really come into its own in 2021 - Windows Terminal. Finally all the features anyone who's been working in the UNIX world have come to expect over the last 30 years are available on Windows as well. Multiple tabs, excellent terminal emulation, great theming and color support, and the ability to seamlessly handle different shells per tab. I regularly keep a Powershell and a WSL tab open for my work.

If you've ever struggled along with the old CMD.EXE, fighting for every cut, paste or insert, this really is a game changer.

I Hate Meeces To Pieces! (Really, Don't Make Me Use the Mouse!)

As someone who's partially blind and has fine and gross motor impairment, using the standard Windows Icons Mice and Pointers user interface is downright agonizing. I need to be able to launch apps with just a few keystrokes. On the Mac, Alfred fulfills this role admirably and adds a whole host of next level productivity power ups in addition to keyboard app launching.

A very recent addition on the Windows side that seems equally powerful and easy is Microsoft Powertoys Powertoys Run feature. I can launch apps, bind keys to actions, and access system functions like sleep, restart and the like all 100% from the keyboard. Nice!

Another Powertoys addition that's frosting my Pop-tarts is Powertoys Keyboard Manager which allows me to remaps the dread CAPS LOCK key to Ctrl so it's right next to my pinky as God Intended :)

This simple enhanment has been impossible with Windows forever without resorting to more drastic measures, so having it work pretty much out of box now is super sweet.

Python - It Takes a Cast of Thousands to Raise a Platform

If you'd uttered the words "Python" and "Windows" in the same sentence a few years back, the average Pythonista would have rolled their eyes or groaned and then proceeded to tell you a war story about how they had to spend weeks trying to get some critical module or other working.

Those days are just straight up gone. Microsoft and the Python community at large have poured countless engineer hours into making the Python ecosystem a truly first class experience under Windows. Virtual environments, poetry, and even tools for installing and managing command line scripts that integrate with the shell like pipx work just great under Powershell on Windows.

And if by chance you're coding something up that's native UNIX to its very core, Windows *still has you covered. Cue the Windows Subsystem for Linux!

WSL - I See Penguins!

With the avent of WSL2 and most recently WSL-g you really do have everything you need to seamlessly build, debug and run Linux programs on Windows, including those that require X/Wayland GUI support or sound.

It's not just some kind of compatibility shim, it's an actual Linux kernel running in Windows. The integration is so complete these days that you can run things like Docker within WSL successfully. That's a really big deal!

Also, most of the popular mainstream development tools like Pycharm and Visual Studio Code have superlative support for deploying to and debugging in WSL built in.

The amount and quality of work Microsoft has put into this is truly impressive. And unlike the Mac, you're not actually using some oddball FreeBSD user space running atop the Mach microkernel, you're running 100% Linux. It's turtles all the way down.

I Go Where The Innovation Is & Use What Works

As a technologist, I'm proud of the success I've had in sniffing out pockets of high velocity, high value innovation in the technology landscape.

Years ago, I was running Linux when many people were either using DOS, pre OSX MacOS or super expensive prorietary vendor workstations like Sun's Sparc. Being able to have an honest to god UNIX environment on commodity hardware was like Prometheus's gift of fire to the mortals. It really was that big a deal.

I still think the Linux desktop has incredible potential, but for me the obstacles imposed by my disabilities and the lack of accessibility features can be a real buzz and productivity killer.

So here I am embracing a desktop that works, with an ecosystem that works and a rather large well heeled company pouring resources into bar raising and innovation.

I look forward to exploring and leveraging the incredible potential this platform represents. Stay tuned for more posts as I go!

Plus ça change, plus c'est la même chose

I was recently in an online discussion about whether or not technologists should use "managed services" like the one I work for.




One gent essentially claimed that the move towards cloud and managed services represented an end game for our industry, and that we'd automate ourselves out of a job. This is what I said:

Will there be job reductions and resultant economic pain as a result of the move away from the data center and towards more cloud/managed services? Hell yes there will be!

Those who want to work in this business will find ways to adapt and continue to add value.

I’ve been working in technology for about 32 years. This doesn’t make me magical or omniscient or even smart but what it does make me is a witness to several waves of seismic change.

I came into the job market in the early 90s when DEC was imploding, PCs were ascendant, and the large expensive workstations vendors were dead companies walking but didn’t realize it yet.

Everyone thought the world would end and we’d all be out of work too.

Will it be hard? Yes. Will there be a lot of people who can’t or don’t want to adapt and will be left behind? Undoubtedly. Is that bad? Yes. However it’s the way our industry works and has since its inception.

We are all turning the crank that drives the flywheel of increased automation, whether we administer our own database clusters or not. The move to managed and cloud definitely represents a notable pinch point, and we’ll see how painful the transition will be, but it’s one paradigm shift in an industry that creates paradigm shifts for a living.

I’ve actually thought for a while that in the longer term, as compute at scale becomes quite a bit smaller and even cheaper, we could see a move back away from cloud because when you can host your company’s compute cluster in a cube the size of a printer and we have algorithms that can encode and enact services at scale in an operationally safe way, the value cloud adds will dwindle.

I love my job, and I love this industry, and plan to continue playing in this pool until I die, whether someone’s willing to pay me for it or not.

I assert that the constant negativity many of us exhibit is neither necessary nor desirable and represents a kind of immaturity that we will eventually grow out of as our industry continues to mainstream.

We’ll see.

The Eternal Novice Trap

Introduction

It was the dawn of a new century in the tech world. We'd just survived the great Y2K scare which, for many of us, turned out to be nothing. I spent a memorably pleasant Y2K eve playing board games and sipping champagne at my company's East Cambridge office, enjoying the company of my co-workers. I felt like I was on top of the world.

Right around that time I encountered a book that would change the way I thought about computing. It contained a huge amount of overall incredibly positive and helpful advice. That book was The Pragmatic Programmer.

The Trap is Set

This book contains innumerable bits of incredibly useful advice. The pearls contained within its pages will help you build a successful career as a technology professional.

But I write this in the hope that you won't misinterpret one particular bit of advice like I unintentionally did. In some ways it hurt my career. Thankfully I became aware of it a few years back and am now headed in the right direction.

Here's the advice I used to hamstring myself:

Learn at least one new language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut.

The Pragmatic Programmer, 2nd edition

This is superb advice for its time, when the majority of programmers spent their days coding away in a single language or two.

It's superb advice to this day for that kind of person. If you've spent years coding in Python, why not try your hand at something with a radically different paradigm? By doing so you'll learn new ways of solving problems.

But zooming back to post Y2K me, that picture didn't fit me at all. I'd started out as a sysadmin, learning a huge amount about the care and feeding of UNIX based systems. I'd done some simple scripting to automate administrative tasks, but nothing at all complex or strenuous. A little bit of shell, some Perl, and I had things humming along just great.

Only I took the advice literally, despite my total lack of mastery of any programming language. With wild abandon, I threw myself into every new programming language I encountered..

The Bright, Shiny, Infinite Rabbit Hole

I played with countless languages, FORTH, a myriad of LISPs. You name it, I played with it. If there was a new hot language on the scene you can bet I'd be downloading it and figuring out how to write Hello, World and a few other toy examples.

But that's as far as it went, because, if I'm honest with myself, playing with new toys doesn't feel like work. It's a shot of dopamine. It feels really good in the moment. You feel accomplished because now you can talk about $LANGUAGE along with the rest of the cool kids who are exploring it.

There are also external reinforcements for this behavior. By being able to talk the talk, you can appear smarter than you are. You can probably even answer surface level questions in job interviews which can, in the short term, help your career.

But not unlike living on sugar and caffeine, that short term buzz will wear off, leaving you feeling empty and, ultimately, like the emperor walking around with no clothes. You can talk the talk like a champ, and be up with the latest buzz, but in some corner of your mind you may recognize that your basic skills are fundamentally lacking.

Brush Your Teeth, Eat Your Veggies, Get Sleep & Exercise

Thankfully, the fix is at least conceptually easy. Just stop doing that :) Pick a language that will become your daily driver, and focus on building your core skills using that language.

Build deep, featureful applications. Learn and exercise your skills using the full breadth of your language's ecosystem, and solve real problems using your language of choice.

For me, that language is Python. That's an easy choice because my day job is building infrastructure for a major cloud vendor and 90%+ of our tooling is written in Python. For the last few years I've promised myself to not learn any new programming languages until I become at least a Journeyman with Python.

You may find, as I did, that this is easier in concept than in practice. You may need some structure to help you achieve this goal. Everyone is different, so this isn't a one size fits all suggestion. I'll share some resources that may help.

Resources to Help Sharpen Your Tools

These days there is an embarrassment of riches out there, free for the taking to anyone with a computer and an internet connection to leverage. I'm just going to cover a few that I've found particularly helpful.

100 Days of Code with Python

If you're like me, having a free form set of exercises can be good, but adding some structure with definite time constraints and deadlines, even if you set them yourself, can be immensely helpful. This course was pretty much ideal for my needs. The idea is that you commit code every day for 100 days. This course uses that framework to take you on a guided smorgasbord tour of the incredibly rich Python ecosystem. The course introduces a particular topic or module and then has you build an interesting near real world application with it. I learned so much from this course that it alone has been a real boost to my career.

Completing a course like this can be a daunting task, especially while working full time. However I found the results are well worth the effort. Hopefully you can find something similar for your language of choice.

Exercism

Exercism is an incredible resource regardless of the programming language you work in. It's not just a set of practice exercises, but a community of people who'll help review your code and provide mentorship.

Project Euler

These are language agnostic problem sets that will stretch your skills and force you to really exercise your problem solving skills. I myself have only scratched the surface with these, and plan to delve deeper in the future.

Parting Thoughts

I hope you'll find value in what I wrote here. I wish I'd had this article earlier on in my career. Maybe I can help someone avoid falling into the trap I did. To sum up:

  • Do learn new programming languages and paradigms, but learn them from a place of confidence and mastery with your primary tool of choice.

  • Don't fall for the trap of perpetually chasing after the bright shiny thing that's hot right now. Recognize that what's new isn't necessarily better. Take what will meaningfully help you advance your career and let the rest flow by. There'll always be more tomorrow.

  • Do keep having fun! You'll learn more quickly and retain more if you're finding enjoyment in what you do. Sometimes it means looking at things a little differently, but often that open mindedness can pay off.

I don't WANT to let go! The Media Rip Saga - Part 1: Formats and Tools.

[caption id="attachment_498" align="aligncenter" width="300"]image1 A Musical Mind[/caption]

A coworker at my previous job once said in response to my whining a bit about having just lost 700 songs from my digital music library "Let go of your music collection!".  That got me thinking, long and hard.

Why?

Why am I lugging around this corpus of several thousand compact discs? Why do I have several shelves worth of DVDs? Why don't I simply let go of the whole idea of media ownership and just drink from the infinite fire-hose that is the internet?

Because, when push comes to shove, my media is part of me.  My CD collection represents innumerable fond memories of places, people and moments I've moved on from but that inhabit special places in my heart and mind.

You may have heard the theory that when one sense is diminished, the brain compensates by heightening another. I'm partially blind, so many if not most of my fond memories have a soundtrack. Being able to listen to that particular album and feel some semblance of that moment wash over me is like having a personal time machine.

Oh hey, there's that Autechre album, not available anymore anywhere as far as I can tell. That was one of the first times ever that I was really turned on by electronica. Music plays in my mind's eye - one of my best friends and I walking into a record shop on Newbury Street. And there's that Boiled In Lead CD I got the day after I was invited to an incredibly cool private concert at a farm in NH by a friend who's since passed away. Neither of these things is on Spotify near as I can tell. If I let go, I lose them, and an aspect of those fond memories along with them.

But I'm getting older - in fact I turn 50 in a few months. Having moved countless times I have begun to feel the weight of my possessions. I recognize that I am privileged in that I can make this choice. Not everyone is lucky enough to be able to buy a NAS (Network Attached Storage) and spend their leisure time converting their media collection into a giant bag of bits. I count myself lucky.

So, now that you've heard way more than you ever wanted to know about why I chose to embark upon this epic project, let's talk turkey in case you're thinking about doing this for yourself.

Preserving all your media in virtual form can create a veritable labyrinth of choices. As with all things, I suggest keeping it simple. I'll tell you what my choices were and maybe even offer another option or two along the way, but mostly I'll be guiding you through my process as it has evolved for my purposes.

First let's answer a basic question: What kind of media do you want to preserve? In my case the answer is simple. My collection consists entirely of CDs and DVDs, as well as a handful of Blu-ray discs.

The first big choice - to compress or not to compress? Compressing your media will save you disk space, but cost you in terms of fidelity. The movies you rip won't look exactly as they did on disc, and the music you rip won't be exactly as it was in its original form either.

I will not go down the rabbit hole of trying to convince you of any particular absolute truth, but my ultimate destination ended up being "I never want to have to do this again and I want no loss in quality".

So, on the video side, I don't use any compression at all. I rip my movies straight to Matroska video or ".mkv" files. This format is incredibly rich and versatile, and also has the advantage of being open source. Why do I care one might ask? See above - I NEVER want to do this again :) So I don't want to be beholden to some company or other deciding that the format I chose is changing, going away, or under license dispute. These are my bits embodying my legally purchased media. Full stop.

On the audio side, I was able to take advantage of an excellent format called FLAC. It provides some compression so as to save space, but most important for me is the fact that it's lossless, so I lose no fidelity at all in my ripped copy. This format is also open source, and the author is a fellow Somervillian :)

Having made the important decisions about format, let's choose appropriate tools.

For the video side, I chose MakeMKV. It's cross-platform (Mac, Linux, Windows), has a very straight forward user interface, and is dirt cheap. You can download it for free, but to rip Blu-ray discs, you have to pay them $5 or $10 to unlock the decoder. Trust me, for the power this tool offers, give them their money. They've earned it. This was one of the only tools I could find that made ripping both Blu-ray and DVD discs easy while also giving me the flexibility I needed. A number of lesser tools I tried had nice user interfaces, but were rife with bugs. making it nearly impossible to extract the raw bits with no compression.

Another critical tool if you want to rip TV series from DVD is a little thing called FileBot. It will rename the files to conform to what Plex requires.

For the rare case where multiple episodes in a TV show are glommed into one big file, I chose MKVToolNix to split .mkv files into usable chunks  I'm unsure whether it's worth the effort, especially for series with lots of episodes.

On the audio side, I used XLD for the Mac. It's on par with Xact Audio Copy for Windows, the most highly regarded audio ripper out there near as I can tell. It rips to FLAC effortlessly and the developer maintains a very high quality checksum database of every CD that's been ripped with the tool so you can be sure that your results are crystal clear and bit perfect. I've had Linux folks ask about something similar for them, and I haven't been able to find anything yet. Sorry about that. If someone finds something that does similar things and has a nice UI, do let me know.

You'll also need a place to put the resulting bits. A really safe place. This is a labor intensive effort, and you really don't want to lose all your countless hours of work due to a hard drive failure.

I chose a Synology DS216+ii for the task. It's a nice little box that holds two big hard drives that you can run as a RAID set. In addition to having hardware that meets my needs, I chose these folks because the software the NAS runs is easy to administer, incredibly flexible, and has a rich ecosystem. One great feature I've not seen elsewhere is with just a few clicks, you can back up your data to cloud services like Glacier. Synology has a proprietary suite of software you can use to manage your media. I chose Plex because it's feature rich and for a lifetime fee will automatically transcode movies and music to work with any viewing or listening device.

In the next installment we'll start getting into the nitty-gritty of how to get the job done. The good, the bad, and the ugly!

Hacking the Wetware

Working for Amazon has been a revelation to me in ways large and small.

One of the biggest by far has been coming to the understanding that career growth was not in fact laying in wait around the next corner if only I could master the latest technology trend. There are baseline skills that can either accelerate or retard your growth in profound ways, and many of them reduce down to changing the way you think in or react to various situations. Let's walk through a few of them.

Taking Ownership

You are well and truly the captain of your own ship. Your performance is your responsibility. While these statements may seem blindingly self evident, in fact there are some subtle traps for the unwary who may not even realize they are rationalizing about failure by abdicating responsibility for their own success.

This can take the form of seemingly innocuous behaviors like not wanting to burn bridges or be the bad guy, but when push comes to shove everybody loves a winner and nobody respects a human door mat. Find ways to constructively deal with conflict head on and ultimately attain your goals. The inherent tension of conflict pales in comparison to the confidence you will gain by standing your ground.

Learning How to Learn Effectively

What do you do when presented with a new problem that you find daunting or confusing? Most of the time this happens to me it's because there's something I'm unfamiliar with, that lies outside my current wheelhouse.

It's very easy to feel utterly overwhelmed by the sheer scope of a problem or by the fact that the problem seems to be so foreign as to not offer any foot holds or surfaces within reach.

I have found that keeping a very simple check list in mind can keep me from feeling overwhelmed and help avoid "analysis paralysis".

  • Just try something. Doing literally anything can help un-stick your brain and get past a roadblock.

  • Write an outline, create a very rough draft.

    • Write "throw away" code / prose fully intending to not save any of it. This can be very liberating when you're stuck!

    • Turn to novel sources of action oriented inspiration, I have found Brian Eno's Oblique Strategies deck helpful at times like this.

  • Narrow or broaden your focus. Are you actually concentrating on the key thing you need to accomplish? Does what you're working on create value for your stake holders?

    • Is your problem well defined? Flesh out the problem statement. Even if you're working on something creative, limitations can be incredibly inspirational! (There's a pithy quote about this I can't bring to mind right now :)

    • Break off the tiniest piece of the problem. Whatever you're comfortable with. Write the introductory paragraph or a few lines of code, even if it's just the header! Sometimes the simple act of getting started can create a momentum of its own.

    • Use tools like outliners or mind maps to work out the general shape of a problem. I've often written an outline and literally cut & pasted that outline into the document or code I need to write as a starting point.

  • Practice! The process of building a skill is universal across virtually every discipline. In Peak: Secrete from the New Science of Expertise, Robert Pool introduces us to the idea of "deliberate practice". More than just doing a thing over and over, deliberate practice involves being taught by an expert and applying those lessons to the process of practice. While Pool specifically dresses fields like athletics or piano playing, we can apply this idea to all kinds of disciplines. One great example of applying deliberate practice in my field is Katrina Owen's superb exercism.io which poses carefully crafted problems to students and then lets experts critique them and suggest better, more elegant or concise ways of solving them.

Emotional Maturity

I used to pride myself on being a passionate practitioner of my craft. Lately I've been feeling like that's less of an advantage and more of a liability. Avid Grtimm's excellent blog post on this topic had a great deal to do with changing my mind, and I suggest you run out and read it and other posts he's made along those lines post haste. They're worth it.

In point of fact I feel as though I should strive for dispassionate responses to problems and situations, especially those that are emotionally charged.

Give yourself that fraction of a second to take a breath, look at the response that's on the tip of your tongue or keyboard. Ask yourself "Is this really what I want to say? Am I being fair to the person I'm interacting with? Am I really acting in the best interests of both myself and my team?"

This can be incredibly difficult in the heat of the moment. It's a skill I am working very hard on cultivating, and I have achieved some modicum of success here, but my personal evolution in this area is far from complete.

Healthy in Body and Mind

There was a time when the stereotypes about people working in technology were more true than not. Many of us didn't take care of ourselves all that well. Obesity and even poor sanitary habits were not at all uncommon.

Thankfully, I've been a member of the once a day shower club my entire adult life, but my diet and exercise regimen were pretty atrocious for quite a long time.

I'm still not quite as disciplined as I'd like to be where diet is concerned, but I work out every day and have been losing some weight recently. I just can't say enough about how much this one simple change has benefited my mental well being both in terms of mental acuity but also in terms of being able to handle stressful situations in a more productive and professional way.

I've found my Apple Watch to be incredibly helpful in this regard. It tracks many different kinds of work-outs, in addition to tracking my heart rate and calories burned through the day. Its "Activity Rings" display tracks the amount of time I stand as well as how much exercise I get, and I'm a total sucker for gamification, so its achievements system keeps me motivated, pathetic though that may be :)

Exercising regularly is the only way to go. Full stop.

I still have a long way to go before I'm at my ideal weight, but I feel like I'm well and truly on the path to getting there. I start every day mentally refreshed and ready to conquer whatever challenges come my way. A big improvement over falling asleep at my desk :)

Another often overlooked aspect of living a healthy life is getting enough sleep. A lot has been written about how getting enough sleep can be critical in maintaining willpower, self discipline, and self control. I definitely struggle with this, as after getting up early to work out and working a long day it can be tempting to goof off for a while rather than getting to bed.

Something I've found rather helpful in this regard is an app called Autosleep. It uses the data on pulse rate and movement collected from my Apple Watch to give me a really rich report on not just how many hours I slept, but also the quality of sleep and periods during the night when I was awake, whether or not I was aware of it. Those old cheeky after school special slogans had some truth to them - "Because knowing is half the battle!".

Discipline

Personal discipline is at the heart of good decision making and the bedrock of any successful career.

The most effective and productive members of my field utterly regiment their days. What goes where is of less importance than the fact that they do the same thing pretty much every day. Code reviews in the morning, followed by whatever meetings are required for the day, and then another burst of focused productivity before heading home.

To be honest, I have yet to attain this kind of self regimentation, but it's good to have goals to work towards :)

Closing

These are just a few of the things that have been on my mind of late. I wrote them down in the hope that someone else may find them useful. If I have learned anything over the last few years it is that "common sense" is anything but common.

"disce semper quasi victurus vive quasi cras moriturus" - Learn as if you're always going to live; Live as if tomorrow you will die.

IOS 11 Makes the iPad Pro Sing!

image1

A while ago I wrote a post about the then new Apple iPad Pro with its awesome keyboard.

I gushed about it already in that post, and that aspect hasn’t changed, but I’ve gotta say, Apple’s new version of IOS really catapults this device into new planes of usefulness.

An issue I’ve had with IOS for as long as it’s been around is Apple’s insistence that the user forget about things like files and just let the software keep track of everything.

While I applaud the idea behind this, the actual implementation meant that using your iPad for actual work in day to day life became needlessly complex. Where did my file go? I have no idea! How do I get the data out of here and copy it to that app over there? Not a clue! It’s all magic and jazz hands!

With IOS 11, Apple has finally reconciled itself to the fact that people tend to think of their data in terms of files. It may not be the world’s most perfect abstraction for everything, but given that people have been getting on with their lives just fine with it ever since computers were a thing, it’s about time that they caved and made it actually easy for thoser of us trying to do actual work.

To wit - The new “Files” application. To their credit, this isn’t your ordinary file manager - it presents a unified view of your files whether they’re on your iPad or a cloud service like Dropbox, Amazon Drive, etc.

They’ve also added the ability to drag and drop files between applications! Finally I can write a document using my favorite editor - Editorial, and check it into source control using Working Copy.

The implications of this are huge. IOS has always had an incredibly rich ecosystem of very rich, high powered applications, but until now wrangling data between them was an exercise in gymnastics I could do without.

There are plenty of places a Google search away that will detail all the awesome features in IOS 11, but to me this is far and away the most important. Overall, the changes in 11 makes it feel like I’m working on a regular laptop when I’m working on my iPad Pro. Given how the whole thing is no bigger than any regular iPad case you might have, it’s really liberating! I could go for a walk or a bike ride, throw the iPad in a backpack, and think nothing of it.