Enterprises increasingly shifted to open source software solutions in 2020 to meet their remote organizational needs and address new market demands for quality and speed. COVID-19 challenged not only the economy, but also enterprises’ existing frameworks for how, when, and at what volume people use information technology.
Last year, major players like LinkedIn and Spotify open-sourced tools they developed — from Java machine learning libraries to audio file processing ecosystems — for non-proprietary IT team members like data scientists and software engineers to use. Seed-level startups like Eradani and RudderStack that built their products on open source thrived despite launching only a few months before the pandemic began to escalate.
Most companies don’t publicize their internal tooling and infrastructure strategy transformations, but GitHub’s data suggests these have much to do with open source solutions. GitHub says 72% of Fortune 50 companies used GitHub Enterprise, which runs GitHub services on their local networks, between Q4 2019 and Q3 2020. GitHub also found over 40% YOY growth in open source project creation per any active user between late April 2019 and late April 2020. This first spiked with the pandemic in early March, when countries began to close schools, ban visitors, and initiate lockdowns en masse.
In an interview with VentureBeat, GitHub VP Mario Rodriguez said, “Open source project creation just kind of shoots up” after March. He added, “2020 is interesting because everything from a technology perspective got accelerated, and you were trying to do more and more.”
According to Rodriguez, “2020 also opened up a new set of talent from a software development perspective.”
Companies were previously limited by their geography and could only hire within certain cities, like San Francisco or New York. But the transition to remote work has begun to change that.
“Now, you do not have those restrictions … and the majority of software developers out there use open source today. And so you bring it into your enterprise now at an accelerated rate, which allows you to learn and continue to evolve your practices on how you develop software and how you use open source,” he said.
The pandemic disrupted existing tech trends and likely helped amplify the growing movement toward open source software solutions as enterprises’ distributed, remote workforces needed to use more custom applications internally, to innovate their IT quickly with the help of reusable code, and to retain developers by using the tools they prefer.
Speed: Accelerating technology with a remote, distributed workforce
Internet traffic skyrocketed by over 30% in March, especially to platforms for online learning and telecommuting. Microsoft Teams users set a new record for 2.7 billion meeting minutes in one day, and Microsoft, along with Netflix and YouTube, temporarily reduced video streaming quality and download speeds to cut back on bandwidth consumption.
These changes highlighted consumer demands for new digital communication tools and challenged enterprise IT teams to create and manage them — quickly. And enterprise IT teams, now distributed and in some cases fully remote, had to organize their own work around new kinds of applications and develop them in days or weeks, as opposed to months or years.
“From an IT perspective, you have to accelerate the [number] of apps that you are creating for your internal use as an enterprise,” Rodriguez said. “So that has actually allowed the enterprises to say, ‘You know what, if we are going to have these constraints, maybe we should start to research and figure out a way to empower more use of open source internally, as well.’ … You’re trying to go faster.”
While some enterprise frameworks and custom logic for business applications remain fully proprietary, integrating open source code can be a faster way to develop most software. Developers can import the existing work of thousands of people with a few inputs, making it easier to pull new applications together. For example, companies expedite their training and development of machine learning models with Google’s TensorFlow. Now that information is more democratized with open source software, some enterprise leaders suggest it’s hard to compete without it.
“I’m a member of the CTO forum, which is a group of 150 CTOs around the globe that talks a lot about tech,” Pinterest head of engineering Jeremy King said in a recent interview with VentureBeat. “And for sure, lifecycles have sped up.”
King described how companies used to try out maybe three or four vendors with an open source technology stack at a time for six months. Afterward, the company might figure out which vendor performed best and negotiate a rate.
“All that has gone away,” King said. He said if a company knows that a given open technology works, they’ll make a prototype and they’ll have changed the cycle by the next week.
“People are more rapidly adopting [open source], and I think it’s just because the tolerance for failure and making mistakes and moving quick has gone up,” he said. He explained that this is also true when it comes to “dealing with the consequences of moving fast in production, which, a year ago was unheard of.”
Elephant Ventures CEO Arthur Shectman also commented on the pandemic’s disruption of enterprise IT in an interview with VentureBeat. “The market kind of collapsed around people. In that moment of high volatility or prices and market stress, you reach for data,” Shectman said. “You’re like, ‘These decisions are going to have profound impacts … the net impact a week from now is huge.’”
Elephant Ventures is a digital transformation consultancy that helps corporations like Pfizer and American Express build their engineering capabilities. Shectman said he applies a technology readiness framework to his approach, finding ways for clients to create increments of business value with ETL technologies, API tooling patterns, and other strategies for deploying applications and workflows.
According to Shectman, “People were clamoring for data, and they were clamoring to transform their kind of data ecosystems, very rapidly.” He said that in the past few months, the conversation went from planning out three years of digital transformation retool to looking for immediate answers with a return on investment in 90 days. Shectman noted that waiting to purchase proprietary software could cost his enterprise clients $3 million in some instances.
Shectman said proprietary software’s cost and speed of deployment became greater roadblocks for his clients during 2020. “I felt like there was a lot more willingness to instantly adopt open source technologies and start applying them without any additional kind of software purchase cost to get a rapid ROI cycle project,” Shectman added. “Over the last year, if you could demonstrate that you knew how to implement it, you had a pattern that generated success in critical dimensions.”
Volume: Innovating with code reuse and existing tools
Developers can build applications at greater volumes by reusing open source code instead of starting from scratch. Enterprise IT teams can also integrate open source tools into their existing workflows to manage their data with improved precision. This control is increasingly important, given the pandemic’s overall shift toward digitalization, which increases the amount of data itself.
“The number of apps that get created right now are at an all-time high. And then the number of those apps that [use] open source are also at an all-time high. It’s probably because of code reuse and ability to just go from zero to 60% or 70% of what you need to create very, very quickly,” Rodriguez said.
These current trends in open source software are a continuation of many enterprises’ strategies. “I think we use a lot of open source technologies, largely due to the fact that our scale often prevents us from using a commercial product,” King said.
“Pinterest is a 10-year-old company, and we have billions of pins … and that’s not always something you can just buy off the shelf. And so whether it’s logging data, understanding images, searching — all these technologies didn’t exist when we started; they’re getting better and better over time. Even the off-the-shelf cloud providers are getting better. But we’ve made Pinterest very unique to what we’re looking for as a result of us using a lot of open source technology,” he added.
Other enterprises have tailored their tech stacks to maximize productivity and output with open source solutions more recently. Even maintaining the same products requires additional speed of deployment when competing in a volatile COVID-19 era market.
McKinsey reports that North American businesses accelerated their share of digitized product offerings by 20% between March and October 2020. Open-source software might play a role in expediting this process if the bulk of digitized products’ code can be built from an existing repository’s code.
“For a long time, the digital economy and the computer revolution were just driven by Moore’s law, with your chips getting better and faster at this crazy growth rate,” Shectman said as he traced his client’s technology solutions from the early 2000s to late 2020. “Now, after you go through the early iterations of your product, you are able to do a certain amount of things with commercially available software, [but] you’re gonna have typically an easier time unwrapping and fixing or customizing the open source stuff.”
According to Shectman, understanding and acting upon data has become key to enterprises. “I think it’s the scale of data production. I don’t think anybody really had a great sense of how rapidly data would proliferate in the world.”
The market’s space for tools, particularly open source technologies that sieve and curate data, has widened as a result. In the past 30 years — and mostly in the last 10 — over 200 companies with open source technology at their core have raised over $10 billion in capital.
Startups such as data warehousing platform RudderStack, which provides an open source-based alternative to Segment, have recently capitalized on this change. RudderStack was launched in late 2019, and its 2020 growth mirrors the growth patterns of other early-stage startups, like Prisma and Streamlit, which contribute to GitHub community code and base their business strategies in open source software.
RudderStack marketing lead Gavin Johnson said in an interview with VentureBeat, “Being open source makes it easy to deploy RudderStack in your own environment. … Anyone can look through the code and figure out what is being done with their customer data in-product, something you can’t do with closed-source products.” According to Johnson, RudderStack’s open source software reduces the number of tools enterprise engineering teams would have to build for sending and collecting data and allows the teams to modify the platform with custom integrations. He also suggested end-user enterprise IT teams tend to save more money with these alternatives to proprietary platforms.
Novelty: Energizing IT teams and long-term growth
The Linux Foundation reports that in 2020, 93% of hiring managers found it difficult to recruit employees with open source software skills and 37% of them wanted to hire more skilled IT professionals, despite the pandemic’s economic impact.
Rodriguez told VentureBeat that in 2020, “We didn’t see companies shrink in developers. We actually saw them expand in developers.” He believes this growth has to do with enterprises’ goals that now, more than ever, revolve around IT capabilities.
“You’re trying to create all this software, which means that you need more developers in order to do it right. And if you’re trying to actually have more, [you need] to not only have the best tools for your developers but have the best practices as well.” He said that is another reason he thinks open source accelerated significantly in 2020.
The open source community organizes conferences, groups, and companies around making code more accessible. Developers contribute to repositories like Open Source for Good and develop personal projects with open source code.
Johnson said that data engineers and data scientists work a lot in open source. He said, “RudderStack’s mission is to help make engineering and data teams be heroes of their organizations. So we needed to build our product in a way they like, which means open source.”
Some enterprises are turning to open source tools to bring in skilled technical employees. King said open source attracts and retains engineers. “And the people who are the best in the world want to publish, and they want to work on things that are well known, and they want to contribute back. And so it is definitely a good return on investment as well,” he said.
In addition to using open source for internal transformations, enterprises have been contributing open source tools to the community. Open source contributions can improve enterprises’ visibility and relevance, and this business strategy is important now that most, if not all, of their public operations are digital. RedHat brings open source technologies to enterprises that need them and has since it was founded in 1993. But in 2020, end-user companies like Facebook, LinkedIn, Spotify, and Uber have also begun to open-source their own tools for the public.
“The open source movement is pretty well established and reasonably mature at this point, so it’s not a surprise for any large corporation,” Shectman said. “If there’s a bit of awareness, people can afford to shy away from some of the convenience of a particular vendor-specific thing in order to allow themselves fluidity.” Shectman suggested that businesses face similar constraints with their staffing, technology, and ability to create effective products. He added that he’s seen open source software help businesses remove constraints, including proprietary vendor lock-in.
Rodriguez suggested that the advancements open source can provide in regard to speed, volume, and overall business strategy aren’t easily matched. “What everyone now is realizing is you cannot out-innovate, or you cannot out-execute open source,” he said.
Will open source continue to grow in 2021 like it did in 2020? Rodriguez thinks it could. “For any company out there, from the most powerful company in the world to a startup, you’re not going to be able to hire [enough] people to build the software of that quality — open source gives you that immediately.”
