This week we welcome Robert Smallshire (@robsmallshire) as our PyDev of the Week! Robert is the founder of the Sixty-North consulting company. Robert also produces amazing Python videos for Pluralsight. Robert also has several Python books that he has authored.
Let’s take a few minutes to get to know Robert better!
Can you tell us a little about yourself (hobbies, education, etc):
I got into computing at around the age of nine in the early 1980s home computing revolution and started programming around that time. I’m 48 now and it’s peculiar to step back for a moment and reflect that I’ve been tinkering for almost 40 years. Growing up, and throughout my formal education, I never felt that programming would be turn out to be my career. I have only one small qualification in computing (Computer Studies GCSE – a British qualification taken at age 16), and after school my studies took me into the natural sciences, culminating with a Ph.D. in geology. I’d been doing hobby programming on and off throughout my school and college years, including writing bits of software to help my Dad’s engineering surveying business, but more novel software was needed during my Ph.D., so I had to create it. I divided my time between spending long summers camping and doing fieldwork in the French and Swiss Alps, and spending wet British winters holed up in the lab or at home writing analytical and modelling software in C++ to deal with the results.
While still writing-up my Ph.D. I started work for a university spin-off company doing commercial R&D in energy and mining. This was also a fairly equal split between fieldwork and desk-work, with multi-month trips out in the desert, near Moab, in Utah. During the day, we’d work in the field or from a light-aircraft from which we had persuaded the pilot to remove the doors to facilitate better low-level aerial photography. Back at our base in the evenings we’d hack on Perl scripts to process our data. On occasional days off we would hike in Arches and Canyonlands national parks. It was intense mental and physical work, and I was hugely fortunate at the beginning of my working life to work alongside folks so immensely smart and motivated; people from whom I learned an enormous amount.
Notwithstanding the privilege of being paid to do fieldwork, I quickly realised that being able to program is a sort of super-power, and doubly-so when you’re naturally curious about, and motivated to solve, the problem at hand. People with a good understanding of geology, who also have an aptitude for programming seem to be relatively rare, and realising this, I decided to apply for a job with a company which supplied some of the analytical software I had encountered at the R&D company.
This took me to Glasgow, Scotland for my first real employment as a “developer”. We worked on a large and complex, graphically intensive C++ system, which at the time I joined ran on expensive Silicon Graphics and Sun workstations. The software allow us to build 3D models of the inside of the Earth and run them backwards and forwards through the fourth dimension of geological time, algorithmically stripping off layers of sediment, folding and unfolding bent rocks, and breaking and unbreaking faulted rocks. It seemed almost magical! Again, I had the good fortune to work with and learn from some very intelligent, motivated, and energetic folks, including my future wife. Within a few years I was leading the development team at this company and I’m glad to report that a new system for which I initiated the design around this time (some time in 2002) is still being developed and sold today some twenty years on. After five years, and some difficult times caused by ups and downs in the energy market, my partner and I decided to look around at other options. She was anyway commuting regularly from the UK to Norway, and after a false start with ideas of moving to Canada, I ended up taking a job in Oslo, Norway and relocating within a matter of weeks. My partner followed a few months later.
In Norway, I was again working on simulation software in the oil and gas sector, but everything at my new employer was an order of magnitude larger than I had hitherto experienced, including a monolithic code base of 2.5 million lines of C++, which took overnight to compile from scratch. After only a few weeks I felt I had made a colossal career blunder, and had I not just migrated across the North Sea, I would probably have left as quickly as I had arrived. There was another option though. As Martin Fowler said, “You can either change your company or you can change your company.”
I decided to stick around long enough to see if I could make a difference, and within a few months found that things were going my way. Some other new blood was brought into what had been a stultifying software development culture, and together we began to turn things around. One such character was a fellow called Austin Bingham, with whom I had much in common, both being immigrants to Norway, and both having had positive prior experience with Python, and both taking software engineering and design, rather than just ‘coding’, seriously. Over seven years I rose to the heady heights of Chief Software Architect, which I assure you sounds rather more grand that it actually was. But still, I was the most senior person working on the internal design and programming of our products, in a business with a turnover of hundreds of millions of dollars. One of my key decisions was to introduce a scripting API in terms of Python by embedding the CPython interpreter into our core product.
Towards the end of my seven-year tenure, the business was sold and passed around a series of private-equity concerns, it becoming clear that financial engineering was more valued – and quite possibly more valuable – than software engineering. Shortly afterwards the company was acquired again in a complex transaction by a large U.S.-based conglomerate who seemed somewhat surprised
to discover that they had bought a software company along with the division of the business they actually thought they were buying. Now, as one of 135 000 employees – a small cog in a very large machine – I decided it was time to move on again.
Another factor in my desire to move on was a desire to get out of the oil and gas sector. The reasons were two-fold: First, as both my wife and I worked in the energy sector, our by then growing family was particularly exposed to the notorious boom-and-bust cyclicity of that industry. Second of all, I was concerned about the negative impact of my work on the climate. Having some training in Earth systems science, and recognising that software can be a huge force-multiplier of human capability, it became clear that somebody in my position could have a disproportionally large and negative impact on the climate. A significant fraction of the world’s oil fields were modelled and simulated in software for which I was at least nominally technically responsible, and in which my designs and code were part.
After a few sessions in the pubs of Stavanger and Oslo with my colleague Austin Bingham, we decided to set up on our own in 2013, to offer software training and consulting services.
Our new company, Sixty North, would naturally be focussed on the software technologies we knew well – particularly Python – and on serving problem domains with a significant science or engineering content, but also those where we could bring our software architecture skills to bear on managing the complexity of large systems. Our experience was that many scientists and engineers are competent at coding in the small, but lack the knowledge, skills and experience to design systems so they can grow relatively gracefully – something probably more true today than ever.
We’ve been running Sixty North for a decade now, largely as a lifestyle company rather than chasing perpetual growth. It turns out we’re pretty good at what we do, and more than able to keep a roof over our heads, and sustain a business with a handful of folks.
I’ve talked a lot about my professional career, so what do I do outside? I’ve tried to cultivate hobbies that get me away from screens and which keep me active. Much of my twenties was spent climbing mountains, descending caves and cycling, but my levels of physical activity declined significantly during my thirties when my wife and I were busy – not to say overwhelmed – by juggling busy careers, business travel and child-rearing. In my forties, I’ve got back into cycling, and now try to cycle most days in the summer, and ski as often as I can in the winter. I’m fortunate to live in a very beautiful part of the world.
Why did you start using Python?
I first used Python seriously in 2001 when I encountered it in the SCons build tool. At the time I was working on graphically intensive commercial earth science simulation software written in C++ for a company in Scotland. The code had a horrible build-system implemented recursively with make (see Recursive Make Considered Harmful) which was difficult to understand, and unreliable. In those days, our large C++ code base had to be built overnight, on Irix, Solaris and Windows, so errors and blunders in the build system were costly. After evaluating some alternatives to make, we stumbled upon SCons and fell into the world of Python. We were using Python 1.5.2 as that was the latest version we could build on all the systems we needed to support, with the compilers we had. At the time Perl was my go-to scripting language, but over the next year or two my use of Perl for ad hoc utilities and programs was almost entirely usurped by Python, although all of my “serious” programming was still done in C++.
I felt that higher-level languages would allow our team to be much more productive than we were in C++, to the extent that I had put a lot of effort into embedding a Perl interpreter in our large C++ application. In retrospect, we had chosen a reasonable software architecture – C++ an with embedded script interpreter, similar to a modern web-browser – but in Perl had erred by plumping for one of the few languages that was even less readable and maintainable than C++!
Around this time I was experimenting with Java and encountered Jython – Python for the JVM. I was very excited about this combination as it promised to marry a fast-ish compiled language (Java) with a high-level language (Python) both of which would avoid the many notorious pitfalls in C++ related to memory management. In particular, Java provided the Swing GUI toolkit and the Java 2D and Java 3D graphics APIs, which could be exercised beautifully from Python code executing on the Jython interpreter. I recall enthusing to a colleague about Jython Essentials (2002) by Samuele Pedroni and Noel Rappin – a better Python introduction than most other straight Python books available at the time – and building interesting prototype applications in Jython running on the JVM, which were portable across all the operating systems we used, and which avoided tedious compile-link-run cycles.
Sadly, Jython never really achieved escape-velocity, though having both the Pythonand Java standard libraries it provided a lot of what regular CPython still lacks out of the box today, particularly in terms of GUI and graphics toolkits. Since then, I’ve introduced Python into other C++-based companies, also via the vector of SCons, and latterly, with the help of Austin Bingham, by embedding the Python interpreter into C++ applications.
What other programming languages do you know and which is your favorite?
I’ve mentioned Perl, C++ and Java already, but I learned to program in the mid-1980s in BBC BASIC, subsequently taking a long journey through COMAL (obscure!), 6502 and ARM assembly language, Pascal, C, C++, Delphi (Object Pascal). I’ve also developed fairly significant code bases in C# and F#, and even done bits of Haskell in a professional context. Much – or perhaps most – of this is now forgotten, but the languages I use regularly today are Python (every day), JavaScript (many days), and Java (occasional substantial stints), a combination which reflects the languages I use at work. I still enjoy exploring languages new and old (but new to me). Recently I’ve dabbled with the Julia programming language, and I’m writing an assembler (in Python) for the vintage 6809 microprocessor for a home-brew 8-bit computer I’m designing and building. If I needed to work on greenfield projects with the performance profile of C++ again, I would put a serious effort into learning Rust. If I need to do much more JavaScript (likely), I can see myself wanting to get into TypeScript.
I see many programmers go through their careers waiting for the next wonder programming language to solve all their problems. I’ve experienced emotions like that too – notably when experiencing Lisp for the first time, or the excitement of interop on the .NET Common Language Runtime – but I feel like I’m at least a decade past that phase now, and look back on myself as being quite naïve. We
have some excellent programming languages and ecosystems currently, and rather than shiny new languages, there are easy gains to be had by using the languages we already have. The key is to use them smartly and diligently, taking system and software architecture seriously. If you know one language for high-performance/low-level such as C++, know JavaScript for the web, and know a general-purpose low-friction language like Python, you can achieve almost anything.
Of all these, Python is the language that keeps drawing me back and is the language I first reach for unless a design constraint forces me in another direction. Python facilitates a particularly short time from initial idea to useful solution. That said, it’s really important to understand when Python is not appropriate, and I’ve been responsible for some costly mistakes in that regard.
What projects are you working on now?
About a decade ago, when Austin Bingham and I founded our consulting and training business Sixty North, we – through a chance encounter at a software conference in Gothenburg, Sweden – fell into making online training course material for Pluralsight. As anybody who has made prerecorded training knows, it’s an enormous effort to design training material, craft good examples, manually
capture high-quality demo videos, record crisp audio, and edit it all together to produce a high-quality product. For the first iteration of our courses we did everything much the same way as most folks still do, with video captures of us developing code “live”, with countless retakes and much editing, pasting snippets of code into Keynote slides and manually annotating it, and so on.
When the time came to update our courses to keep up with the evolution of Python, the latest versions of tools such as PyCharm, higher resolution output, and more stringent and stylish graphic design requirements, I think it’s fair to say that the
prospect of hundreds of hours of manual rework didn’t immediately fill us with joy.
Instead, we figured we could, in principle at least, produce all of our material (demo videos, slides, diagrams) from a machine-readable description of the course. We could automatically synchronise the visuals with the audio voiceover, and then, when a new version of Python or PyCharm was released, or when the need arose to deliver courses at a different video resolution, we could literally make a few updates to configuration files or demo example code and re-render the whole course.
Naturally, the difference between an ‘in principle’ and ‘in practice’ solution to this need is a huge amount of work on building the tools to do this, and describing all of our video courses to the system. Needless to say, we have around 25 hours of Python video training material published on Pluralsight which is renderable and – crucially – cheaply modifiable, completely automatically.
At the time of writing, we’re exploring our options for bringing this technology to a wider audience, and removing some of the rough edges from the now substantial video course production system my colleagues and I at Sixty North have built.
Which Python libraries are your favorite (core or 3rd party)?
Many of the Python packages we make are designed to have a clear Python API and on top of that a CLI. I’ve found click to be an excellent library for specifying command-line interfaces. For testing, I regularly turn to David MacIver’s excellent property-based testing library, Hypothesis.
What are the top 3 things you learned while writing a book or video course?
1. Having to teach a topic in an excellent way to learn it well.
2. Finding good teaching examples which exhibit Goldilocks “just right” complexity requires long walks or bike rides, but also a grounding in experience to understand and demonstrate their relevance.
3. Most books have a hopeless financial return compared to invested effort, but are good for gaining credibility. I wouldn’t advise anybody to write a technical book for the money from sales, but to instead write one to support other aspects of a broader consulting or training business. For example. our Python Craftsman series, The Python Apprentice, The Python Journeyman, and The Python Master, are derived directly from our work on our classroom and Pluralsight training materials and they mutually support each other.
Is there anything else you’d like to say?
This open ended question made me contemplate the things which have transformed my ability to build quality software. Along with using a language you love – or at least learning to love the language you use – I would add the following advice:
First of all, taking testing seriously has had a big impact on my work. I sometimes, though by no means always, practice Test-Driven Development (TDD). Even when I’m not using TDD the tests are rarely far behind and are usually written contemporaneously with the production code. The effort in arranging for code to be testable will be repaid many times over, not just in terms of correctness, but for other desirable qualities of the system.
Secondly, taking architecture and design seriously has been transformational for me. Largely this is about following a handful of maxims: “Do one thing and do it well”, “Separate concerns”, “An architecturally significant decision is one which is costly to change later”, “Instead of asking objects for their state, tell objects what to do, and give them what they need to do it”, “Prefer pure functions”, and so on.
Many of these boil down to keeping control of coupling and cohesion, and it’s hard to overstate the importance of these for sustained success.
The third, least technical, and most enjoyable practice that has made a huge impression on me in recent years is daily pair or ensemble programming. This really took off for us during the Covid-19 pandemic to the extent that the majority of code we write at Sixty North has at least two people involved at the point of creation, and I feel our code, happiness and team spirit is much better for it. I wouldn’t like to go back to long stretches of working alone, to asynchronous code review, or to modes of working based around pull-requests.
Finally, I’d like to thank you for giving me opportunity to tell a bit of my story.
Thanks for doing the interview, Robert!