Stop Programming and Start Integrating

There are two trends that have been accelerating over the last two decades that I believe are changing the landscape of software development. The first is the rise of scripting languages.

My favorite programming language is Python. I started using it in 1993 while at Texas A&M to develop a command line tool to manage computer networking equipment using a combination of databases and SNMP. (You can read about it in this USENIX paper published in 2000.) Since that time, it has gone from being my favorite but occasional tool to being integral to almost any project I'm involved with.

You can see this same trend in the TIOBE Index. Over the last 10 years, interest in scripting languages has generally trended upward while interest in compiled languages has generally trended downward.

Scripting Vs. Compilation

People will get into long arguments about why this trend exists - usually focusing on the technical merits of compilation versus scripting. However, I believe there's another, more interesting trend at work.

The concept of open source software has been around for almost as long as computing has. But in the last, 10 years, there's been almost an explosion of open source. The result has been a commoditization of source code - meaning, you can find an open source solution for almost any common computing task you might be working on. It's pointless to go build another GUI when GNOME and Qt are out there. And it's not likely anybody is going to spend the time to build a new operating system kernel with Linux available. The amount and quality of open source is devaluing work related to common computing tasks.

Productivity and Technology

There's plenty of precedent for this in history. The introduction and continuous development of technology has always commoditized human output - forcing society to adapt to the changes. The steam engine may have displaced many laborers but it was the catalyst for entire new industries that employed a tremendous number of people in higher level jobs. And these were better paying jobs which ultimately raised the standard of living for society at large.

I believe this same force in software development is one of the primary reasons scripting languages are gaining popularity. With so much commodity software out there, software development is moving from programming solutions to common problems to integrating common open source solutions into new higher level solutions. 

Integration

The concept of wide-spread integration gained a lot of popularity in the early 2000s with the introduction of SOAP and the more general concept of Web Services. While integration at the software component level will never be that standardized (at least one drawback of the open source ecosystem), the "messiness" of it can be mitigated at the programming language level.

I'm not aware of any metric that measures the amount of open source solutions that a programming language is integrated with, but if it did exist, I'm pretty confident that Python would be ranked #1. There is a tremendous ecosystem of Python modules that integrate with many open source packages - to the point where many projects now strive to provide Python integration by default.

Ultimately, integration is about moving up the food chain. Instead of solving common and mundane programming problems, we are shifting to integrating these existing solutions into new and more interesting solutions. Python (and scripting languages in general) excel at quickly and efficiently assembling high-level integration solutions.

So, next time you need to choose between a compiled language (and its ecosystem of integrated solutions) and a scripting language like Python, just be careful that you aren't choosing a plow over a steam engine....