View source for End-to-end principle

{{short description| design principle for computer networking}}

{{Net neutrality}}

The '''end-to-end principle''' is a design framework in computer networking. In networks designed according to this principle, application-specific features reside in the communicating end nodes of the network, rather than in intermediary nodes, such as gateways and routers, that exist to establish the network.

The essence of what would later be called the end-to-end principle was contained in the work of Paul Baran and Donald Davies on packet-switched networks in the 1960s. Louis Pouzin pioneered the use of the end-to-end strategy in the CYCLADES network in the 1970s.<ref name="Bennett2009">{{cite web|last1=Bennett|first1=Richard|date=September 2009|title=Designed for Change: End-to-End Arguments, Internet Innovation, and the Net Neutrality Debate|url=https://www.itif.org/files/2009-designed-for-change.pdf|accessdate=11 September 2017|publisher=Information Technology and Innovation Foundation|pages=7, 11}}</ref> The principle was first articulated explicitly in 1981 by Saltzer, Reed, and Clark.<ref name="SRC1981" />{{refn|The 1981 paper<ref name="SRC1981" /> was published in ACM's TOCS in an updated version in 1984.<ref name="SRC1984" /><ref name="SAL1980" />|group=nb}} The meaning of the end-to-end principle has been continuously reinterpreted ever since its initial articulation. Also, noteworthy formulations of the end-to-end principle can be found before the seminal 1981 Saltzer, Reed, and Clark paper.<ref name="SRC1984" />

A basic premise of the principle is that the payoffs from adding features to a simple network quickly diminish, especially in cases in which the end hosts have to implement those functions only for reasons of conformance, i.e. completeness and correctness based on a specification.{{refn|The full quote from the Saltzer, Reed, Clark paper states:<ref name="SRC1984" /> "In a system that includes communications, one usually draws a modular boundary around the communication subsystem and defines a firm interface between it and the rest of the system. When doing so, it becomes apparent that there is a list of functions each of which might be implemented in any of several ways: by the communication subsystem, by its client, as a joint venture, or perhaps redundantly, each doing its own version. In reasoning about this choice, the requirements of the application provide the basis for the following class of arguments: The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible, and moreover, produces a performance penalty for all clients of the communication system. (Sometimes an incomplete version of the function provided by the communication system may be useful as a performance enhancement.) We call this line of reasoning against low-level function implementation the end-to-end argument." (p. 278).|group=nb}} Implementing a specific function incurs some resource penalties regardless of whether the function is used or not, and implementing a specific function ''in the network'' distributes these penalties among all clients.

==Concept==
[[File:End-to-end Concept-en.svg|thumb|According to the end-to-end principle, the network is only responsible for providing the terminals with connections, Any kind of intelligence should be located at the terminals.]]
The fundamental notion behind the end-to-end principle is that for two processes communicating with each other via some communication means, the ''reliability'' obtained from that means cannot be expected to be perfectly aligned with the reliability requirements of the processes. In particular, meeting or exceeding very high-reliability requirements of communicating processes separated by networks of nontrivial size is more costly than obtaining the required degree of reliability by positive end-to-end acknowledgments and retransmissions (referred to as PAR or ARQ).{{refn|In fact, even in local area networks there is a non-zero probability of communication failure{{snd}} "attention to reliability at higher levels is required regardless of the control strategy of the network".<ref name="Clark1978" />|group=nb}} Put differently, it is far easier to obtain reliability beyond a certain margin by mechanisms in the ''end hosts'' of a network rather than in the ''intermediary nodes'',{{refn|Put in economics terms, the marginal cost of additional reliability in the network exceeds the marginal cost of obtaining the same additional reliability by measures in the end hosts. The economically efficient level of reliability improvement inside the network depends on the specific circumstances; however, it is certainly nowhere near zero:<ref name="SRC1984" /> "Clearly, some effort at the lower levels to improve network reliability can have a significant effect on application performance. (p. 281)."|group=nb}} especially when the latter are beyond the control of, and not accountable to, the former.<ref group="nb">The possibility of enforceable contractual remedies notwithstanding, it is impossible for any network in which intermediary resources are shared in a non-deterministic fashion to guarantee perfect reliability. At most, it may quote statistical performance averages.</ref> Positive end-to-end acknowledgments with infinite retries can obtain arbitrarily high reliability from any network with a higher than zero probability of successfully transmitting data from one end to another.{{refn|More precisely:<ref name="Sunshine1975" /> "THM 1: A correctly functioning PAR protocol with infinite retry count never fails to deliver, loses, or duplicates messages. COR 1A: A correctly functioning PAR protocol with finite retry count never loses or duplicates messages, and the probability of failing to deliver a message can be made arbitrarily small by the sender." (p. 3).|group=nb}}

The end-to-end principle does not trivially extend to functions beyond end-to-end error control and correction. E.g., no straightforward end-to-end arguments can be made for communication parameters such as latency and throughput. In a 2001 paper, Blumenthal and Clark note: "[F]rom the beginning, the end-to-end arguments revolved around requirements that could be implemented correctly at the endpoints; if implementation inside the network is the only way to accomplish the requirement, then an end-to-end argument isn't appropriate in the first place."<ref name="BC2001" />{{rp|80}}

The end-to-end principle is closely related, and sometimes seen as a direct precursor, to the principle of net neutrality.<ref>{{cite web|url=https://www.theatlantic.com/technology/archive/2014/04/the-best-writing-on-net-neutrality/361237/|title=Net Neutrality: A Guide to (and History of) a Contested Idea|publisher=The Atlantic|author1=Alexis C. Madrigal |author2=Adrienne LaFrance |name-list-style=amp|date=25 Apr 2014|accessdate=5 Jun 2014|quote=This idea of net neutrality...[Lawrence Lessig] used to call the principle e2e, for end to end}}</ref>

==History==
In the 1960s, Paul Baran and Donald Davies, in their pre-ARPANET elaborations of networking, made brief comments about reliability that capture the essence of the later end-to-end principle. To quote from a 1964 Baran paper, "Reliability and raw error rates are secondary. The network must be built with the expectation of heavy damage anyway. Powerful error removal methods exist."<ref name="Bar1964" />{{rp|5}} Similarly, Davies notes on end-to-end error control, "It is thought that all users of the network will provide themselves with some kind of error control and that without difficulty this could be made to show up a missing packet. Because of this, loss of packets, if it is sufficiently rare, can be tolerated."<ref name="Dav1967" />{{rp|2.3}}

The ARPANET was the first large-scale general-purpose packet switching network{{snd}} implementing several of the basic notions previously touched on by Baran and Davies.

Davies had worked on simulation of datagram networks.<ref name="Hempstead20052">{{cite book|author1=C. Hempstead|url=https://books.google.com/books?id=2ZCNAgAAQBAJ&pg=PA574|title=Encyclopedia of 20th-Century Technology|author2=W. Worthington|date=2005|publisher=Routledge|isbn=9781135455514|location=|pages=|quote=Simulation work on packet networks was also undertaken by the NPL group.}}</ref><ref name="Pelkey">{{cite book|last=Pelkey|first=James|url=http://www.historyofcomputercommunications.info/Book/6/6.3-CYCLADESNetworkLouisPouzin1-72.html|title=Entrepreneurial Capitalism and Innovation: A History of Computer Communications 1968-1988|publisher=|year=|isbn=|location=|pages=|chapter=6.3 CYCLADES Network and Louis Pouzin 1971-1972|quote=He had done some simulation of datagram networks, although he had not built any, and it looked technically viable.}}</ref> Building on this idea, Louis Pouzin's CYCLADES network was the first to make the hosts responsible for the reliable delivery of data, rather than this being a centralized service of the network itself.<ref name="Bennett2009" /> Concepts of this network influenced later ARPANET architecture.

==Applications==
===ARPANET===
The ARPANET demonstrated several important aspects of the end-to-end principle.
;Packet switching pushes some logical functions toward the communication endpoints
:If the basic premise of a distributed network is packet switching, then functions such as reordering and duplicate detection inevitably have to be implemented at the logical endpoints of such a network. Consequently, the ARPANET featured two distinct levels of functionality:
:# a lower level concerned with transporting data packets between neighboring network nodes (called Interface Message Processors or IMPs), and
:# a higher level concerned with various end-to-end aspects of the data transmission.{{refn|In accordance with the ARPANET RFQ<ref name="RFQ1968" /> (pp. 47 f.) the ARPANET conceptually separated certain functions. As BBN points out in a 1977 paper:<ref name="BBN1977" /> "[T]he ARPA Network implementation uses the technique of breaking messages into packets to minimize the delay seen for long transmissions over many hops. The ARPA Network implementation also allows several messages to be in transit simultaneously between a given pair of Hosts. However, the several messages and the packets within the messages may arrive at the destination IMP out of order, and in the event of a broken IMP or line, there may be duplicates. The task of the ARPA Network source-to-destination transmission procedure is to reorder packets and messages at their destination, to cull duplicates, and after all the packets of a message have arrived, pass the message on to the destination Host and return an end-to-end acknowledgment. (p. 284)."|group=nb}}
:Dave Clark, one of the authors of the end-to-end principle paper, concludes: "The discovery of packets is not a consequence of the end-to-end argument. It is the success of packets that make the end-to-end argument relevant." <ref name="Clark2007" />{{rp|slide 31}}

;No arbitrarily reliable data transfer without end-to-end acknowledgment and retransmission mechanisms
: The ARPANET was designed to provide reliable data transport between any two endpoints of the network{{snd}} much like a simple I/O channel between a computer and a nearby peripheral device.{{refn|This requirement was spelled out in the ARPANET RFQ, "From the point of view of the ARPA contractors as users of the network, the communication subnet is a self-contained facility whose software and hardware is maintained by the network contractor. In designing Interconnection Software we should only need to use the I/0 conventions for moving data into and out of the subnet and not otherwise be involved in the details of subnet operation. Specifically, error checking, fault detection, message switching, fault recovery, line switching, carrier failures and carrier quality assessment, as required to guarantee reliable network performance, are the sole responsibility of the network contractor."<ref name="RFQ1968" />{{rp|25}}|group=nb}} In order to remedy any potential failures of packet transmission normal ARPANET messages were handed from one node to the next node with a positive acknowledgment and retransmission scheme; after a successful handover they were then discarded,{{refn|Notes Walden in a 1972 paper, "Each IMP holds on to a packet until it gets a positive acknowledgment from the next IMP down the line that the packet has been properly received. If it gets the acknowledgment, all is well; the IMP knows that the next IMP now has responsibility for the packet and the transmitting IMP can discard its copy of the packet."<ref name="Walden1972" />{{rp|11}}|group=nb}} no source-to-destination re-transmission in case of packet loss was catered for. However, in spite of significant efforts, perfect reliability as envisaged in the initial ARPANET specification turned out to be impossible to provide{{snd}}a reality that became increasingly obvious once the ARPANET grew well beyond its initial four-node topology.{{refn|By 1973, BBN acknowledged that the initial aim of perfect reliability inside the ARPANET was not achievable, "Initially, it was thought that the only components in the network design that were prone to errors were the communications circuits, and the modem interfaces in the IMPs are equipped with a CRC checksum to detect 'almost all' such errors. The rest of the system, including Host interfaces, IMP processors, memories, and interfaces, were all considered to be error-free. We have had to re-evaluate this position in the light of our experience.<ref name="BBN1973" />{{rp|1}} In fact, as Metcalfe summarizes by 1973, "there have been enough bits in error in the ARPANET to fill this quota [one undetected transmission bit error per year] for centuries."<ref name="Met1973" />{{rp|7–28}} See also BBN Report 2816<ref name="BBN1974" />{{rp|10 ff}} for additional elaboration about the experiences gained in the first years of operating the ARPANET.|group=nb}} The ARPANET thus provided a strong case for the inherent limits of network-based hop-by-hop reliability mechanisms in pursuit of true end-to-end reliability.{{refn|Incidentally, the ARPANET also provides a good case for the trade-offs between the cost of end-to-end reliability mechanisms versus the benefits to be obtained thus. Note that true end-to-end reliability mechanisms would have been prohibitively costly at the time, given that the specification held that there could be up to 8 host-level messages in flight at the same time between two endpoints, each having a maximum of more than 8000 bits. The amount of memory that would have been required to keep copies of all those data for possible retransmission in case no acknowledgment came from the destination IMP was too expensive to be worthwhile. As for host-based end-to-end reliability mechanisms{{snd}}those would have added considerable complexity to the common host level protocol (Host-Host Protocol). While the desirability of host-host reliability mechanisms was articulated in {{IETF RFC|1}}, after some discussion they were dispensed with (although higher-level protocols or applications were, of course, free to implement such mechanisms themselves). For a recount of the debate at the time see Bärwolff 2010,<ref name="Bae2010" /> pp. 56-58 and the notes therein, especially notes 151 and 163.|group=nb}}

;Trade-off between reliability, latency, and throughput
: The pursuit of perfect reliability may hurt other relevant parameters of a data transmission{{snd}}most importantly latency and throughput. This is particularly important for applications that value predictable throughput and low latency over reliability{{snd}}the classic example being interactive real-time voice applications. This use case was catered for in the ARPANET by providing a raw message service that dispensed with various reliability measures so as to provide faster and lower latency data transmission service to the end hosts.{{refn|Early experiments with packet voice date back to 1971, and by 1972 more formal ARPA research on the subject commenced. As documented in {{IETF RFC|660}} (p. 2),<ref name="Walden1974" /> in 1974 BBN introduced the raw message service (Raw Message Interface, RMI) to the ARPANET, primarily in order to allow hosts to experiment with packet voice applications, but also acknowledging the use of such facility in view of possibly internetwork communication (cf. a BBN Report 2913<ref name="BBN1974b" /> at pp. 55 f.). See also Bärwolff 2010,<ref name="Bae2010" /> pp. 80-84 and the copious notes therein.|group=nb}}

===TCP/IP===
Internet Protocol (IP) is a connectionless datagram service with no delivery guarantees. On the internet, IP is used for nearly all communications. End-to-end acknowledgment and retransmission is the responsibility of the connection-oriented Transmission Control Protocol (TCP) which sits on top of IP.  The functional split between IP and TCP exemplifies the proper application of the end-to-end principle to transport protocol design.

===File transfer===
An example of the end-to-end principle is that of an arbitrarily reliable file transfer between two endpoints in a distributed network of a varying, nontrivial size:<ref name="SRC1984" />  The only way two endpoints can obtain a completely reliable transfer is by transmitting and acknowledging a checksum for the entire data stream; in such a setting, lesser checksum and acknowledgment (ACK/NACK) protocols are justified only for the purpose of optimizing performance{{snd}}they are useful to the vast majority of clients, but are not enough to fulfill the reliability requirement of this particular application. A thorough checksum is hence best done at the endpoints, and the network maintains a relatively low level of complexity and reasonable performance for all clients.<ref name="SRC1984" />

==Limitations==
The most important limitation of the end-to-end principle is that its basic premise, placing functions in the application endpoints rather than in the intermediary nodes, is not trivial to implement.

An example of the limitations of the end-to-end principle exists in mobile devices, for instance with mobile IPv6.<ref>{{Cite IETF|rfc=3724|author1=J. Kempf|author2=R. Austein|date=March 2004|title=The Rise of the Middle and the Future of End-to-End: Reflections on the Evolution of the Internet Archichecture|publisher=Network Working Group, IETF}}</ref> Pushing service-specific complexity to the endpoints can cause issues with mobile devices if the device has unreliable access to network channels.<ref>{{Cite web|url=http://www.winlab.rutgers.edu/docs/focus/CNF/protocol%20architecture.html|title=CNF Protocol Architecture|last=|first=|date=|website=Focus Projects|publisher=Winlab, Rutgers University|access-date=May 23, 2016}}</ref>

Further problems can be seen with a decrease in network transparency from the addition of network address translation (NAT), which IPv4 relies on to combat address exhaustion.<ref>{{Cite news|url=https://www.bbc.com/news/technology-19600718|title=Europe hits old internet address limits|last=Ward|first=Mark|date=2012-09-14|work=BBC News|access-date=2017-02-28|language=en-GB}}</ref> With the introduction of IPv6, users once again have unique identifiers, allowing for true end-to-end connectivity. Unique identifiers may be based on a physical address, or can be generated randomly by the host.<ref>{{Cite web|url=ftp://ftp.cuhk.edu.hk/pub/doc/ipng/html/ipv6-address-privacy.html|title=Statement on IPv6 Address Privacy|last=Steve Deering & Bob Hinden|first=Co-Chairs of the IETF's IP Next Generation Working Group|date=November 6, 1999|website=|access-date=2017-02-28}}</ref>

==See also==
*Peer-to-peer

==Notes==
{{Reflist|group=nb}}

==References==
{{Reflist|30em|refs=
<ref name="Bar1964">Baran, P. (1964). "On Distributed Communications Networks". In: IEEE Transactions on Communications 12.1, pp. 1–9.</ref>
<ref name="Bae2010">Bärwolff, M. (2010). "End-to-End Arguments in the Internet: Principles, Practices, and Theory". Self-published online and via Createspace/Amazon ([http://xn--brwolff-5wa.de/publications/2010-10-PhD-thesis.html PDF, errata, etc.])</ref>
<ref name="BBN1974">Bolt, Beranek and Newman Inc. (1974). Interface Message Processors for the Arpa Computer Network. BBN Report 2816. Quarterly Technical Report No.5, 1 January 1974 to 31 March 1974. Bolt, Beranek and Newman Inc. (BBN). ([http://xn--brwolff-5wa.de/public/BBN-1974--Interface-Message-Processors-for-the-ARPA-Computer-Network--Report-2816--Quarterly-Technical-Report-5.pdf Private copy, courtesy of BBN]).</ref>
<ref name="BBN1977">McQuillan, J. M. and D. C. Walden (1977). "The ARPA Network Design Decisions". In: Computer Networks 1.5, pp. 243–289. ([http://www.walden-family.com/public/whole-paper.pdf Online copy]). Based on a Crowther et al. (1975) paper, which is based on BBN Report 2918, which in turn is an extract from BBN Report 2913, both from 1974.</ref>
<ref name="BC2001">Blumenthal, M. S. and D. D. Clark (2001). "Rethinking the Design of the Internet: The End-to-End Arguments vs. the Brave World". In: ACM Transactions on Internet Technology 1.1, pp. 70–109. ([https://dspace.mit.edu/bitstream/handle/1721.1/1519/TPRC_Clark_Blumenthal.pdf?sequence=1&origin=publication_detail Online pre-publication version]).</ref>
<ref name="Clark1978">Clark, D. D., K. T. Pogran, and D. P. Reed (1978). “An Introduction to Local Area Networks”. In: Proceedings of the IEEE 66.11, pp. 1497–1517.</ref>
<ref name="Clark2007">Clark, D. D. (2007). Application Design and the End-to-End Arguments. MIT Communications Futures Program Bi-Annual Meeting. Philadelphia, PA. May 30–31, 2007. Presentation slides. ([http://cfp.mit.edu/events/may07/presentations/CLARK%20Application%20Design.ppt Online copy]).</ref>
<ref name="Dav1967">Davies, D. W., K. A. Bartlett, R. A. Scantlebury, and P. T. Wilkinson (1967). "A Digital Communication Network for Computers Giving Rapid Response at Remote Terminals". In: SOSP '67: Proceedings of the First ACM Symposium on Operating System Principles. Gatlinburg, TN. October 1–4, 1967. New York, NY: ACM, pp. 2.1–2.17.</ref>
<ref name="BBN1973">McQuillan, J. M. (1973). Software Checksumming in the IMP and Network Reliability. {{IETF RFC|528}}. Historic. NWG.</ref>
<ref name="RFQ1968">Scheblik, T. J., D. B. Dawkins, and Advanced Research Projects Agency (1968). RFQ for ARPA Computer Network. Request for Quotations. Advanced Research Projects Agency (ARPA), Department of Defense (DoD). ([http://www.cs.utexas.edu/users/chris/DIGITAL_ARCHIVE/ARPANET/RFQ-ARPA-IMP.pdf Online copy] {{webarchive|url=https://web.archive.org/web/20110815020022/http://www.cs.utexas.edu/users/chris/DIGITAL_ARCHIVE/ARPANET/RFQ-ARPA-IMP.pdf |date=2011-08-15 }}).</ref>
<ref name="SRC1981">Saltzer, J. H., D. P. Reed, and D. D. Clark (1981) "End-to-End Arguments in System Design". In: Proceedings of the Second International Conference on Distributed Computing Systems. Paris, France. April 8–10, 1981. IEEE Computer Society, pp. 509-512.</ref>
<ref name="SRC1984">Saltzer, J. H., D. P. Reed, and D. D. Clark (1984) "End-to-End Arguments in System Design". In: ACM Transactions on Computer Systems 2.4, pp. 277-288. (See also [http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf here] for a version from Saltzer's MIT homepage.)</ref>
<ref name="Sunshine1975">Sunshine, C. A. (1975). Issues in Communication Protocol Design &ndash; Formal Correctness. Draft. INWG Protocol Note 5. IFIP WG 6.1 (INWG). ([http://xn--brwolff-5wa.de/public/Sunshine-1975-Issues-in-Communication-Protocol-Design--Formal-Correctness--INWG-Note-5.pdf Copy from CBI]).</ref>
<ref name="Walden1972">Walden, D. C. (1972). "The Interface Message Processor, Its Algorithms, and Their Implementation". In: AFCET Journées d’Études: Réseaux de Calculateurs (AFCET Workshop on Computer Networks). Paris, France. May 25–26, 1972. Association Française pour la Cybernétique Économique et Technique (AFCET). ([http://www.walden-family.com/public/1972-afcet-paris.pdf Online copy]).</ref>
<ref name="Met1973">Metcalfe, R. M. (1973). "Packet Communication". PhD thesis. Cambridge, MA: Harvard University. [http://publications.csail.mit.edu/lcs/pubs/pdf/MIT-LCS-TR-114.pdf Online copy] (revised edition, published as MIT Laboratory for Computer Science Technical Report 114). Mostly written at MIT Project MAC and Xerox PARC.</ref>
<ref name="BBN1974b">BBN (1974). Interface Message Processors for the Arpa Computer Network. BBN Report 2913. Quarterly Technical Report No. 7, 1 July 1974 to 30 September 1974. Bolt, Beranek and Newman Inc. (BBN).</ref>
<ref name="Walden1974">Walden, D. C. (1974) Some Changes to the IMP and the IMP/Host Interface. {{IETF RFC|660}}. Historic. NWG.</ref>
<ref name="SAL1980">Saltzer, J. H. (1980). End-to-End Arguments in System Design. Request for Comments No. 185, MIT Laboratory for Computer Science, Computer Systems Research Division. ([http://web.mit.edu/Saltzer/www/publications/rfc/csr-rfc-185.pdf Online copy]).</ref>
}}

[[Category:Internet architecture]]
[[Category:Net neutrality]]
[[Category:Network architecture]]
[[Category:Programming paradigms]]