Saturday, August 30, 2008

END-TO-END ARGUMENTS IN SYSTEM DESIGN

The end-to-end principle is suggested as a design principle for defining boundaries between system functions. It hasn't received any formal addressing up til this paper. The author opened with the file transfer example that argues redundancy at the communication layer reduce the number of retries at the application level but does little in terms of ensuring correctness. Since faults or errors can occur anywhere along the chain (disk, filesystem, file transfer app., communication layer), detecting or correcting errors must be implement at the end-points.

The justification for implementing the checking at the low-level is bettering performance due to finer granularity of information blocks (e.g. per-packet or -file block instead of per-file) and access to the low-level machinery. The low-level does not have to require strict correctness, since the app layer requires that already. Other examples discussed in the paper includes: delivery guarantees, secure transmission of data, duplicate message suppression, guaranteeing FIFO message delivery, and transaction management (e.g. whether a network is acted upon in the app layer). In short, better performance may be gained from implementing functions at low-level but forces all apps above to bear this functionality, while pushing functionality onto higher-level reduces design redundancy. This principle, though illustrated mostly in networking applications in the paper, also applies throughout different areas in system design.

This paper is a classic in networking. It was written back in 1984. The first authors, Jerome H. Saltzer, was heavily involved with the (MIT) development of CTSS, Multics and Project Athena, where X and Kerberos came from. I think this paper definitely has its place on the class reading list, just because it shows the early design principle for today networking protocols like TCP and UDP. Btw, David P. Reeds, the another author of the paper, designed UDP in 1980 prior to publishing this paper. So, this paper definitely exposed some concrete meta wisdom in terms networking design principles.

I can also think of microkernel as another example that takes advantage of the end-to-end principle, though its focus is more on security rather than functional divisions.

End-to-end is not an absolute rule. I personally don't think we can get to the bottom of the end-to-end arguments. It can only be at best a suggested perspective, rather than hard truth, when designing a system because the arguments can't be taken to the extreme, for both sides. The higher abstractions should provide customizations for plugging in or swapping out unnecessary functions. A shoe size won't fit everyone's need! Having our networking infrastructure as a layered stack gives user the flexibility in swifting the functional boundaries.