Thanks to the Solarwinds security breach, software supply chain attacks have become an important issue. Naturally enough, there’s a lot of research being done into these attacks. Two graduate students at the University of Minnesota working on a paper entitled, “On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits” tried to put the Use-After-Free (UAF) vulnerability into the Linux kernel. This kind of Red Team security testing is commonplace… when the project includes people who know what’s going on beforehand. That wasn’t the case here. When they tried it again, Greg Kroah-Hartman, the Linux kernel maintainer for the stable branch, had had enough.
Kroah-Hartman, one of the most respected of all the Linux kernel developers, tweeted, “Linux kernel developers do not like being experimented on, we have enough real work to do.”
In the Linux Kernel Mailing List (LKML), Kroah-Hartman made this even clearer when they tried again to introduce a bogus patch. “If you look at the code, this is impossible to have happen[ed]. Please stop submitting known-invalid patches. Your professor is playing around with the review process in order to achieve a paper in some strange and bizarre way. This is not ok, it is wasting our time, and we will have to report this, AGAIN, to your university…”
Leon Romanovsky, a senior Linux kernel developer explained to those who came in late that, “They introduce kernel bugs on purpose.” That’s a huge no-no in any open-source community, but especially in the Linux kernel community where trust between programmers is a vital part of the development process. As Kroah-Hartman continued, “All contributions by this group of people need to be reverted, if they have not been done so already, as what they are doing is intentional malicious behavior and is not acceptable and totally unethical.”
You might think that these graduate students might get the hint. They didn’t. One of the researchers, Aditya Pakki, doubled down. Pakki sent Kroah-Hartman a message stating, “I respectfully ask you to cease and desist from making wild accusations that are bordering on slander.” He also claimed these patches were the result of a new static analyzer he’d written. Pakki closed, “I will not be sending any more patches due to the attitude that is not only unwelcome but also intimidating to newbies and non-experts.”
Kroah-Hartman had had enough. He replied:
You, and your group, have publicly admitted to sending known-buggy patches to see how the kernel community would react to them and published a paper based on that work.
Now you submit a new series of obviously incorrect patches again, so what am I supposed to think of such a thing?
They obviously were _NOT_ created by a static analysis tool that is of any intelligence, as they all are the result of totally different patterns and all of which are obviously not even fixing anything at all. So what am I supposed to think here, other than that you and your group are continuing to experiment on the kernel community developers by sending such nonsense patches?
When submitting patches created by a tool, everyone who does so submits them with wording like “found by tool XXX, we are not sure if this is correct or not, please advise.” which is NOT what you did here at all. You were not asking for help, you were claiming that these were legitimate fixes, which you KNEW to be incorrect.
A few minutes with anyone with the semblance of knowledge of C can see that your submissions do NOT do anything at all, so to think that a tool created them, and then that you thought they were a valid “fix” is totally negligent on your part, not ours. You are the one at fault, it is not our job to be the test subjects of a tool you create.
Our community welcomes developers who wish to help and enhance Linux. That is NOT what you are attempting to do here, so please do not try to frame it that way.
Our community does not appreciate being experimented on, and being “tested” by submitting known patches that are either do nothing on purpose or introduce bugs on purpose. If you wish to do work like this, I suggest you find a different community to run your experiments on, you are not welcome here.
These developers won’t be coming back again. And, because the University of Minnesota didn’t stop them after being warned, Kroah-Hartman hit the entire school with the biggest club in the Linux kernel arsenal: “I will now have to ban all future contributions from your University and rip out your previous contributions, as they were obviously submitted in bad-faith with the intent to cause problems.”
Most Linux kernel developers and other programmers agree with Kroah-Hartman. Ted T’so, a senior Linux kernel developer and Google engineer, notes that while the professor in charge of this project, Kangjie Lu, has done useful security work in the past:
The problem is that Prof. Lu and his team seem to be unrepentant, and has some very… skewed… ideas over what is considered ethical, and acceptable behavior vis-a-vis the Kernel development community. The fact that the UMN IRB [Institutional Review Board] team believes that what Prof. Lu is doing isn’t considered in scope for human experimentation means that there isn’t any kind of institutional controls at UMN for this sort of behavior — which is why a University-wide Ban may be the only right answer, unfortunately.
Red Hat Technology Strategist, Jered Floyd, went farther in his tweet, “This is worse than just being experimented upon; this is like saying you’re a ‘safety researcher’ by going to a grocery store and cutting the brake lines on all the cars to see how many people crash when they leave. Enormously unethical.”
The researchers claim in their paper that none of their patches actually ever made it into any Linux code repositories, that they only appeared in an e-mail rather than becoming a Git commit to any Linux kernel branch. That is not the case.
Romanovsky reported that he had looked at four accepted patches from Pakki “and 3 of them added various severity security ‘holes.'” Sudip Mukherjee, Linux kernel driver and Debian developer, followed up and said “a lot of these have already reached the stable trees.” These patches are now being removed.
So, not only did these “researchers” waste the time of Linux committers but they actually introduced bad code into the Linux kernel. After this, I think I can safely say none of them will ever be welcome in the Linux kernel or any other open-source project in the future.
Related Stories:
