{"id":1674,"date":"2019-04-11T13:18:05","date_gmt":"2019-04-11T17:18:05","guid":{"rendered":"https:\/\/www.bu.edu\/rhcollab\/?p=1674"},"modified":"2019-04-26T08:59:35","modified_gmt":"2019-04-26T12:59:35","slug":"a-fork-in-the-road","status":"publish","type":"post","link":"https:\/\/www.bu.edu\/rhcollab\/2019\/04\/11\/a-fork-in-the-road\/","title":{"rendered":"A fork() in the road – conversation between Uli Drepper & Orran Krieger"},"content":{"rendered":"
\n
\n
\"\"<\/figure>\n

Uli Drepper<\/strong>, Red Hat Distinguished Engineer<\/p>\n<\/div>\n

\n
\"\"<\/figure>\n

Orran Krieger<\/a><\/strong>, Lead\/PI Mass Open Cloud (MOC)\u00a0 and Red Hat Collaboratory<\/p>\n<\/div>\n<\/div>\n

Next month at the 17th Workshop on Hot Topics in Operating Systems<\/a> in Bertinoro, Italy, there will be a session on a paper entitled “A fork() in the road<\/a>” by Andrew Baumann<\/strong> (Microsoft Research), Jonathan Appavoo<\/strong> (Boston University), Orran Krieger<\/strong> (Boston University), and Timothy Roscoe<\/strong> (ETH Zurich.) This paper discusses the usage of fork() as a fundamental abstraction for operating systems, detailing issues with fork() in the modern era as well as suggesting alternative solutions.<\/p>\n

We have made the paper available here:
\nA fork() in the road<\/a><\/strong> (PDF, 528K)<\/em><\/p>\n

This paper has quickly generated a lot of discussion: Hacker News<\/span><\/a>, Twitter<\/span><\/a>, LWN<\/span><\/a>, Reddit<\/span><\/a>, and Lobsters<\/span><\/a> all have lengthy ongoing discussion threads on the points raised by the paper.<\/p>\n

We would like to share another discussion of the paper, this one between Red Hat DE Uli Drepper<\/strong> and Orran Krieger<\/strong>. Their discussion follows a particular format as agreed ahead of time: Uli wrote an initial response to the paper, Orran provided a rebuttal, and finally Uli was given the opportunity for a final rebuttal. <\/span><\/p>\n

Follow the conversation below!<\/span><\/p>\n

Uli\u2019s Initial Response<\/span><\/h2>\n

First and foremost, this response reflects exclusively my opinion and must not be understood as Red Hat\u2019s position and especially not that of the many software developers at Red Hat who probably have a better understanding of the implementation details. I dealt with the subject matter for many years now through the work on the C library, the participation in the POSIX standard committee, and, together with Ingo Molnar, working on the extension to Linux\u2019s <\/span>clone()<\/span> interface to enable the implementation of Linux\u2019s POSIX thread library (NPTL).<\/span><\/p>\n

The paper actually deals with two different aspects of <\/span>fork()<\/span> which are conflated and used in support of each other. This was pointed out during the review but the limiting format of a conference paper made it difficult to address this point. The main points that are made are:<\/span><\/p>\n

    \n
  1. fork()<\/span> was not really designed, it is more an accidental implementation detail<\/span><\/li>\n
  2. The implementation as required by historic practice and then later standardization require many parts of the OS to have ties to the <\/span>fork()<\/span> implementation<\/span><\/li>\n
  3. These dependencies and ties make it hard to implement an OS differently from a traditional Unix system while maintaining application compatibility at the source code level<\/span><\/li>\n
  4. Empirical results suggest that <\/span>fork()<\/span> is the predominant way to start a new program<\/span><\/li>\n
  5. Starting a new program through fork() can have a high latency, depending on the parent process\u2019 resource use (mostly memory)<\/span><\/li>\n
  6. Despite the drawbacks, <\/span>fork()<\/span> is taught as good design<\/span><\/li>\n<\/ol>\n

    I do not disagree with all points:<\/span><\/p>\n

      \n
    • Yes, <\/span>fork()<\/span> is not the most well-thought-out interface<\/span><\/li>\n
    • Yes, there are major problems in some situations when a new program has to be started<\/span><\/li>\n
    • Whether or not <\/span>fork()<\/span> is taught is a good design I cannot say, for the longest time I thought of it lacking<\/span><\/li>\n<\/ul>\n

      My main grievances are:<\/span><\/p>\n

        \n
      • The mixing of arguments about <\/span>fork()<\/span> and <\/span>fork()<\/span>+<\/span>exec<\/span>. These are separate problems but\u2026<\/span><\/li>\n
      • \u2026 the different situations when <\/span>fork()<\/span> is useful without <\/span>exec<\/span> are, at best, underrepresented<\/span><\/li>\n
      • No realistic general replacement is \u00a0provided<\/span><\/li>\n
      • All arguments about <\/span>fork()<\/span> also apply to Linux\u2019s <\/span>clone()<\/span> interface (and whatever other OSes use) but that is not spelled out but has to be considered in the arguments (as I will do below)<\/span><\/li>\n
      • Much of the argumentation is about the standardized semantic of the <\/span>fork()<\/span> interface. I would argue that almost no program depends deeply on most of the requirements. Why should a new OS development or research OS therefore be limited? Alternative interfaces with just the right semantic can be used.<\/span><\/li>\n<\/ul>\n

        The facts include:<\/span><\/p>\n

          \n
        • Starting a new program through <\/span>fork()<\/span>+<\/span>exec<\/span> is horrible because all\/most the requirements of the <\/span>fork()<\/span> semantic are wasted<\/span><\/li>\n
        • The work needed for <\/span>fork()<\/span> scales with the number of pages in the parent process\u2019 address space, the number of file descriptors, and some more entities which can lead to high latencies (e.g., gigantic Java processes starting a program)<\/span><\/li>\n
        • There are many reasons to use <\/span>fork()<\/span> (or <\/span>clone()<\/span> etc) without <\/span>exec<\/span>. Modern Linux has namespaces which can be used this way, traditionally the <\/span>daemon()<\/span> interface etc are implemented with <\/span>fork()<\/span>. \u00a0And\u2026<\/span><\/li>\n
        • \u2026 in general multi-process programs can and should be implemented with fork(). Sharing the address space layout leads to major design advantages (and, no, it is not an additional security problem because of the lack of ASLR because the alternative, threads, <\/span>all<\/b> have the same address space layout as well.<\/span><\/li>\n
        • While on the topic, threads are just as well or even more a design flaw because of implementation history and cannot possibly be thought as a valid alternative to multi-process applications.<\/span><\/li>\n<\/ul>\n

          In general, the <\/span>vfork()<\/span> interface shows that a useful interface does not have to come with a lot of semantic attached. In fact, we removed it from the POSIX standard in the 2008 revision because standardizing it did not serve much of a purpose and nevertheless it is extremely useful today. Because a runtime for a system like Linux can rely on more than just the least common denominator of the semantic of all systems <\/span>vfork()<\/span> can be used outside the scopes of the specification and it is used to implement <\/span>posix_spawn()<\/span> on Linux in many situations. As the paper shows, this eliminates the performance problems but it does it without removing <\/span>fork()<\/span>-like semantic entirely.<\/span><\/p>\n

          I would like to see a native implementation of <\/span>posix_spawn()<\/span> in the kernel. In fact, many moons ago I started the discussion but I never took it to the end since the interface was deemed ill-specified unless it can fulfill the needs of the most prominent users of <\/span>fork()<\/span>+<\/span>exec<\/span> (including shells). I did and do agree with that and perhaps this effort should be picked up again. <\/span><\/p>\n

          My most fundamental disagreement is about the future of <\/span>fork()<\/span>-like interfaces. While the paper argues that they must replaced I think that <\/span>fork()<\/span> is too high-level and, just like in the origins of <\/span>fork()<\/span> itself, we need to aggregate around some lower-level interfaces, like <\/span>clone()<\/span>. I do not suggest that <\/span>clone()<\/span> is ideal but it served us well in implementation a whole bunch of new technologies, from a real thread implementation to containers. This is a far cry from the claim that <\/span>fork()<\/span> prevents<\/span><\/i> innovation is OS research.<\/span><\/p>\n

          Also, threads are a problem and not a solution. It was ill-advised not just from the implementation\u2019s perspective to provide programmers direct access to thread interfaces. Those who participated in the relevant standard committees and who implemented thread libraries will testify to the pain. Threads should be regarded as an implementation detail, perhaps with system-specific semantic reflecting the underlying OSes capabilities and design decisions. Fortunately we are on the way to provide better alternatives to using concurrency with extensions to programming languages and extensions like OpenMP. If one really wants to control concurrency directly then implementing multi-process applications is much better since the multiple processes prevent a whole bunch of classes of problems, they are more resilient, and easier to debug since there are fewer points of contact. All that is needed is to share exactly the resources that need to be shared and that is possible with today\u2019s interfaces (especially when including non-standard ones included in Linux and perhaps other OSes).<\/span><\/p>\n

          If it could be shown that we could have a (set of) interfaces to construct a new process from the ground up under control of the parent code this would be indeed a nice development. We all hopefully agree that such a method of process construction can only be judged a success if it can handle all situations. This means such a process composition must be able to<\/span><\/p>\n

            \n
          • Share part of the address space with the child<\/span><\/li>\n
          • Provide a subset of file descriptors and similar resources from the parent to the child<\/span><\/li>\n
          • Do this without high latencies<\/span><\/li>\n<\/ul>\n

            To be general, this has to be possible for processes with different privileges, be it SUID\/SGID binaries or binaries with different execution contexts \u00e0 la SELinux. How can I trust a program that has been constructed by the parent from pieces? At least so far the security models depend on the semantic of <\/span>exec<\/span> as a system call and if <\/span>exec<\/span> is needed, so is <\/span>fork()<\/span>.<\/span><\/p>\n

            If the purpose of the paper is to point to the inadequacies of <\/span>fork()<\/span> and the ill-advised teaching of <\/span>fork()<\/span> as a jewel of software engineering I can concur. But I cannot agree with most of the conclusions and implied solutions which I found not suitable.<\/span><\/p>\n

             <\/p>\n

            Orran’s Rebuttal<\/span><\/h2>\n

            Thanks Uli! \u00a0You raise a good point, that the majority of the paper is fundamentally about the combination of fork+exec as a mechanism for creating new programs. \u00a0It sounds like we are in strong agreement on the problems with that combination. I think we are also in strong agreement that posix_spawn should be implemented in the kernel and enhanced. <\/span><\/p>\n

            You raise a number of other points that we\/I are not fully in agreement with. \u00a0First, you say “in general multi-process programs can and should be implemented with fork()”. \u00a0\u00a0This is likely something we won’t agree on. If the need for isolation is important, then it makes sense to use multiple processes rather than threads\/events, however, I would argue that it would be better to create multiple processes using spawn, and then establish any sharing required (e.g., passing fds and establishing common mappings). \u00a0\u00a0The memory can be shared copy-on-write using de-duplication between the processes – ai-prori sharing the memory is problematic for all the reasons discussed. Forcing the programmer to make explicit where they want to share (rather than having it the default) seems much less error prone. Yes, there is an argument that the re-initialization can be substantial, but, forcing applications to have a somewhat larger pool of long-lived processes seems a modest cost to not polluting the interface of our OSes with fork(), with all the attendant problems we describe. \u00a0As Jonathan just reminded me, there are other alternatives that have been developed in the research community to accelerate process startup without fork(), but … thats probably outside the scope of this discussion. <\/span><\/p>\n

            You state “I do not suggest that <\/span>clone()<\/span> is ideal but it served us well in implementation a whole bunch of new technologies, from a real thread implementation to containers. \u00a0This is a far cry from the claim that <\/span>fork()<\/span> prevents<\/span><\/i> innovation is OS research.” Agreed, it doesn’t prevent innovation, it just makes it more difficult (we say “limits innovation”, which I agree is an overstatement). \u00a0Obviously with the huge open source community of Linux, innovation is going to happen; but the more we remove impediments the better. Every time we add an abstraction to our operating systems, we need to think about what it means if an application forks\/clones, or decide that the abstraction will be broken in the child. \u00a0I won’t repeat the arguments of the paper about multi-threading, DPDK, application level buffering, accelerators; all not working. Having the fundamental abstraction of the OS for creating new processes having undefined semantics for a whole series of abstractions is just wrong. <\/span><\/p>\n

            You say “If it could be shown that we could have a (set of) interfaces to construct a new process from the ground up under control of the parent code this would be indeed a nice development.” It has been shown, multiple times by multiple research operating systems. \u00a0It has been demonstrated that mechanisms that manipulate child processes from parent enabling sharing and can have high performance,… Your main concern appears to be “How can I trust a program [with elevated privileges] that has been constructed by the parent from pieces?” \u00a0I would agree with that concern. We would argue that this concern applies to fork\/exec, where the child is starting with an inherited environment created by the parent (and the chain of parents that created it). My\/our view (and memory of what we did in past research OSes I was involved in) suggests that the way to create a process with elevated\/different privileges is to make the request to create that process via another server\/process (my preference) or, if spawn is built into the kernel, to use that; ensuring that all the state of the new process is carefully validated. \u00a0\u00a0<\/span><\/p>\n

            Overall, however, I would agree, while we tried to suggest a path to alternatives, we certainly (while barely fitting into the space) didn’t treat the topic in any depth. \u00a0Our fundamental goal is to raise the problem and point to potential paths. While you may already agree that the combination of fork()\/exec() is bad, that conclusion is certainly not shared by the majority of textbooks, and that was the main point of the paper. \u00a0<\/span><\/p>\n

             <\/p>\n

            Uli\u2019s Rebuttal<\/span><\/h2>\n

            Starting multi-process programs with <\/span>posix_spawn()<\/span> is makes it pretty much impossible to have identical address space layouts because of ASLR. Aside from having possible address range conflicts even for the explicitly shared regions it is useful to have fixed positions for the rest (including text segments) since it allows to use pointers much more freely. Aside, de-duplication will less likely work when the address space layout differs because usually pointers are stored throughout the data segment.<\/span><\/p>\n

            Furthermore, <\/span>\u201c(e)very time we add an abstraction to our operating systems, we need to think about what it means if an application forks\/clones\u201d<\/span><\/i> is not really necessary. It is perfectly fine to leave (part of) the state undefined. It might be that after a <\/span>fork()<\/span> certain functionality is not available anymore or that programs using certain functionality cannot <\/span>fork()<\/span> in the first place. That is perfectly fine, programmers can react to that. This applies to uses like DPDK as well: I doubt a user of DPDK will argue much if after <\/span>fork()<\/span> the stack is not available anymore.<\/span><\/p>\n

            Suggesting to implement program startup using micro-kernel-like services is not really something that I can take serious. The empty list of widely used, general purpose micro-kernel OSes speaks for itself. It is not as if you could not have developed something already to sit on top of an existing OS kernel to move some of the functionality to userlevel. It simply never worked up, the implementation complexity (which you try to reduce!) and security model to the best of my knowledge stopped all such attempts in their tracks.<\/span><\/p>\n

            Overall it is the goal of the OS to enable more productive and efficient development and deployment of applications. It is <\/span>not<\/b> the goal of the OS to make the life of OS developers and researchers more pleasant. Unless there is actually functionality which cannot be implemented while preserving the ability to use the <\/span>fork()<\/span> functionality there is no justification to abandon the interface.<\/span><\/p>\n

            Continuing the Conversation
            \n<\/span><\/h1>\n

            Uli and Orran agreed to end things there, but what do you think? We welcome your feedback on the paper and this discussion in the comments below!
            \n<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

            Uli Drepper, Red Hat Distinguished Engineer Orran Krieger, Lead\/PI Mass Open Cloud (MOC)\u00a0 and Red Hat Collaboratory Next month at the 17th Workshop on Hot Topics in Operating Systems in Bertinoro, Italy, there will be a session on a paper entitled “A fork() in the road” by Andrew Baumann (Microsoft Research), Jonathan Appavoo (Boston University), […]<\/p>\n","protected":false},"author":14350,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6,9],"tags":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/posts\/1674"}],"collection":[{"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/users\/14350"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/comments?post=1674"}],"version-history":[{"count":12,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/posts\/1674\/revisions"}],"predecessor-version":[{"id":1698,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/posts\/1674\/revisions\/1698"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/media?parent=1674"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/categories?post=1674"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bu.edu\/rhcollab\/wp-json\/wp\/v2\/tags?post=1674"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}