http://shlang.com/ http://shlang.com/startups-viral-marketing.php <blockquote><strong>Ads are best. Free ads are cheapest, but unpredictable. Viral is often misunderstood.</strong></blockquote> <p>Any startup with an inexpensive or free product needs to sell many units to make it. For the sake of making the back of this envelope tidy, let's say the conversion rate from trying it out or looking at it to buying (or using enough to have time to see ads) is an optimistic 5%. Let's say the value of a user/customer is $10. And let's consider a modest target of $10M. This means that 1M people need to become paying customers (or regular users of an ad-supported service) and, thus, 20M need to try it out. <p>How does one get from 0 to 20M? Ignoring partnerships and early acquisitions, there are three main ways: <ol> <li>Ads <li>Free ads <li>Viral </ol> <h2>Ads</h2> <p>Ads are the easiest to understand: the startup pays someone money, less than user value multiplied by conversion rate, to send prospective customers to try out the product or service. With our example, the startup needs to pay less than $0.50/click. Depending on the kind of product or service, this may or may not be realistic while preserving good targeting. Best way to target ads is currently by keywords, and keywords can be worth anywhere between $0.05 and $10 per click. (Anecdotal spikes above $10 CPC usually average down, and nothing for less than $0.05 CPC is credible.) <p><strong>Ads are the best way</strong> for a startup to attract prospective users, because it allows the startup to concentrate on its core competence, presumbly building its product or service, and essentially outsource user acquisition. Unless the startup's product or service is about marketing, purchasing prospective customers on an efficient market is probably the right thing: the startup creates its value with its product instead. <p>With $0.20 CPC, our startup would need to shell out $4M for the ads. It's worth it if it has the funding. If it doesn't, it starts looking down the list at free ads and viral. <p>Naturally, no sane startup would spend all of its ad budget in one swoop. Instead, they will spend in stages, looking at conversion, secondary word-of-mouth growth, etc., etc. This allows to have more than a single shot and is obviously prudent. It doesn't change the total amount by much: it still remains a formidable pile of money, usually requiring VC funding. <h2>Free ads</h2> <p>Free ads work essentially like ads, except the startup doesn't pay for them: someone with a lot of eyeballs directs them towards the startup, often by placing links to the startup or articles about it, or perhaps by giving it other forms of publicity. <p>The provider of free ads might be Google, Digg, or Yahoo. They all have in common one thing: the value they create for the user comes from directing the user to places interesting for him. Thus, while these companies may also run advertizing, their core business from users' point of view is providing them with free ads. <p>Free ads sound great until you realize that they are unpredictable, uncontrollable, and usually aren't quite free: SEO and social media marketing consultants don't work for free, and you've little chance without them. <p>Still, free ads are nearly free compared to ads. And users you get this way may well be more passionate. Nearly every startup that runs ads also tries to get the free ads. With what they spend on ads, trying PR and SEO and various forms of guerilla marketing is surely worth it. <p>Free ads are often confused with viral marketing. The difference is that viral spread involves person-to-person propagation, often through word of mouth, with one person telling a small number. Free ads are directed at huge numbers of people simultaneously and are very similar to actual ads. If your site ends up the top hit for "paris hilton cellphone hack" on Google, that's not viral spread. That's lucking out on free ads. (In this famous case that made Digg, Google was also wrong: instead of sending people directly to information about Paris Hilton and the contents of her cellphone, it sent its users to Digg, which in turn sent them to their destination, adding little value from Google's point of view and users' point of view.) <h2>Viral</h2> <p>Tim tries ACME. Tim likes ACME. Tim tells Tom and Tony, who in turn like the product and each tell two more people. <p>Right? Of course not. The numbers are completely wrong as telling two friends assumes a completely unrealistic conversion rate greater than 50% from hearing to spreading. <p>If the conversion rate is more realistic, we see that a user converted to spreading needs to tell about ACME essentially <strong>everyone he knows well</strong>: some tens of people, and usually not much less than 20. The number 20 should be familiar from chain letters and the current cap on Facebook app invitations. <p>Startups that expect to spread virally with no provisions for letting users conveniently invite 20 friends suffer the same fate. Don't let it happen to you. <p>The mere technical convenience of inviting 20 naturally doesn't mean people will actually do it. It is a necessary condition, but not anywhere near sufficient. I don't know what triggers viral spread. Do you? Stanislav Shalunov http://shlang.com/php-perl-comparison.php <blockquote><strong>Likes: arrays, their nesting, and web integration. Dislikes: no lexical scope, short cicruits, or seatbelts.</strong></blockquote> <p>I've been actively using Perl since 1997. For background, before Perl, the only high-level language I used was Lisp. I found Perl's text-processing capacity impressive and stayed. Recently (a year ago), I was exposed to using Python for webdev. <p>Last Sunday, I decided to hold my nose and give PHP a try in a toy Sunday project. I liked it a lot more than I expected I would. For a taste, this implements a poor man's proxy/mirror: <pre> $fp = fopen($_REQUEST['url'], 'r', false); fpassthru($fp); </pre> (Before jumping in with LWP::Simple, note that it starts sending before the entire file is downloaded.) <p>I have only looked at PHP 5. But then again, why would you look at the old version? <p>These are my very first impressions of the language. They may yet change. <p>PHP provokes strong reactions, much like alleged English food substance marmite. I'll start with the good. <h2>Like: PHP arrays are universal composite data structures</h2> <p>Perl merges lists and arrays. PHP goes a final step further. <p>PHP does not separate hashes/dictionaries/maps, lists, and arrays. Instead, it uses a single composite data structure. That hash table with next links you implemented in C? They have that, plus optimization for random access by index, built right into the language. Powerful and frees you to think of what you actually want to do. <p>Make no mistake: this is a data structure from a real first-class programming language. To the point that its power feels out of place in a little webdev language. <h2>Like: web integration</h2> <p>It just works with little more effort than HTML. (I use Apache anyway.) No deciding between mod_perl, CGI, FastCGI, SCGI, internal server, and whatever else you were considering. Things like persistent database connections are trivial and taken for granted. No wonder 40 out of 100 top web sites run on PHP. <p>And it's fast. On a server where locally getting static HTML file takes 13ms, getting a lightweight PHP file takes 15ms with the default configuration and no tuning. <h2>Like: No auto-flattening</h2> <p>Perl's auto-flattening of lists is ugly and one of few Perl features that I intensely dislike. In PHP, you don't need to use references to use arrays to represent a tree. <p><code>array(1, array(2, 3), 4)</code> <h2>Like: Functions do what you want</h2> <p>There are a lot of standard functions and they are generally well thought out and do what you want to do during webdev. Good standard library is no small feat. <h2>Like: It's obvious how to do things</h2> <p>Well, it you know Perl and HTML, that is. I hardly had to use the docs, and then mostly to look up standard functions. Easy entry. <h2>Lukewarm like: Class syntax</h2> <p>I don't expect anyone likes Perl's objects. I don't, anyway. I'm not much into OOP in any language, but in Perl I avoid it completely. Pretty much anything is better than Perl's OOP system, and PHP's classes are just that: anything. Bland and Java-like. I haven't had a chance to use them, so I don't know if PHP supports generic functions. <h2>Dislike: No lexical scope</h2> <p>PHP has no lexical scope. Worse, the local scope that they have in functions does not even shadow variables from above. Instead, you can use things like <code>$GLOBALS['x']</code> or <code>global $x; $y = $x ...;</code>. Both variations are ugly. What is this? Some sort of misguided attempt to encourage functional programming? <p>Once the lack of lexical scope sinks in, you stop looking for anonymous functions. You couldn't form closures even if they had lambdas. <h2>Dislike: No short-circuits</h2> <p>Did you like Perl's <code>$x ||= $y</code>? That's too bad, because it doesn't work in PHP. It's syntactycally kosher, but logical operations evaluate to 0 and 1. While my other dislikes are to some extent a matter of taste and design, this is inexcusable. PHP gains nothing by destroying information in logical operations. <h2>Dislike: no strict or warnings</h2> <p>Remember the bad old days of Perl 4, when you could spend minutes debugging a problem that turned out to be a simple misspelled variable name? No? That's the state of the art in PHP in 2007. I hope that I am missing something and the equivalent of <code>use warnings; use strict;</code> is possible, but I haven't found anything of the sort. <h2>Conclusion: Give it a whirl for your next webdev project</h2> <p>Warts and all, it deserves a chance. It's easy to whip something small out, and learning pays for itself within first hour or two. Don't go for tutorials. Just jump in. <p>Note that most of the warts were also present in Perl when it was PHP's age. And everything I list is easy to fix, except for lack of lexical scope. Let's hope they fix it soon. There's no shortage of uses. Stanislav Shalunov http://shlang.com/rss-feed-comments.php <blockquote><strong>Script to process HTML into an RSS feed and a comments widget choice -- Disqus.</strong></blockquote> <p>The advantages of having a home page since 1995 are search engine rankings and established links and visitors. The disadvantage is that I am stuck using old technology. Some of the new web tech is just fluff, but RSS, comments, and trackbacks/pingbacks are useful. After procrastinating too long, I set up RSS and comments today. <h2>Comments system for plain-HTML site</h2> <p>Disqus is a no-brainer. There are several competing comment widgets, and I have not evaluated them, because Disqus is used by Dave Winer, Fred Wilson, Russell Beattie, and Steven Hodson. <p>I decided to give it a try. Give it a whirl below, will ya? <h2>RSS feed for HTML site</h2> Feedity did not look right: <ol> <li>If they fold up shop, losing RSS readers will hurt. <li>Not full-content RSS. Not acceptable. </ol> <p>I simply made my own <a href="create-rss.pl">RSS feed script</a> to run from Cron. <h2>PHP templates</h2> <p>While I was mucking with it, I switched to hand-made PHP templates. I could have used server-side includes, but PHP is more flexible and more widely supported. <h2>Feedburner?</h2> <p>I made a Feedburner feed. I must be missing something. What exactly do I get from them I don't already have in my access logs? Any Feedburner power user care to enlighten me if I should use them and why? Stanislav Shalunov http://shlang.com/writing/mean-delay-considered-harmful.html <blockquote> <strong>Summary:</strong> When <a href="../linkknot/delay.html">network delay</a> is reported as a single number, a common approach is to report the arithmetic average (mean) of observed delay singleton measurements. This practice yields a poor metric. Median is superior to mean in every way save simplicity of computation. </blockquote> <p>The delay (one-way or round-trip) of a single packet is a singleton measurement of delay of a network. When network delay is reported, singleton is usually not the value to report; instead, some number of packets is sent, and some function of their delays is reported. Such a set of delays is called a sample. The function yields some empirical characteristic of the sample. If the sample is thought to come from an underlying random distribution, the function can be a parameter estimator for the distribution. <p>A common function used in this context is arithmetic average. It has several problems: <ol> <li>Treatment of lost packets. Some packets that were sent might not be received within the period of time the receiver is willing to wait for them. These packets are called lost. It is important to realize that the only thing observed about such lost packets is that they didn't make it within the timeout period. For all we know, they might arrive later. What is the delay of the lost packet? We can only say that it is greater than the timeout. (It might be infinite or finite.) The treatment the mean estimator must give to these packets is to simply omit (!) them from the sample. Why does it omit the packets? Because it's the right thing to do? Of course not---it biases the result in a timeout-dependent way, which can't be desirable. It omits the packets because, within the arithmetic average framework, there's no better answer. <li>Wrong answer. Reporting ``mean delay'' implies an assumption of a distribution from which delay values are drawn and reporting the mean of such distribution. Needless to say, arithmetic average of values in the sample below a certain value is not a consistent, or even asymptotically consistent, estimator of the mean. This can be insignificant for certain classes of distributions if the cutoff value is chosen so that it's large enough. However, for other classes of distributions, e.g., heavy-tailed distributions, any cutoff value is bad and extremely distorting; with a heavy-tailed distribution, for any given cutoff, the bulk of the contribution to the estimate of the mean will come from values above the cutoff. It is unknown whether network delay has a heavy tail, but using an estimator that is known to give the wrong answer for some distribution, when a much better choice is available is folly. Note that this problem stems from the first one. <li>Lack of robustness. An even more general property is robustness of the estimator. Robustness, in statistics, is a general property of analysis: robust analysis does not change significantly when the data does not change significantly. One property of robust estimators is that when only a small fraction of values in a sample is changed, the estimator does not change significantly. Arithmetic average is not a robust estimator. For any given sample and any given value of the estimate, changing just a single given value in the sample is enough to make the arithmetic average equal to the desired estimate. The problem becomes especially acute when the tail of the distribution is relatively heavy (as it appears to often be with network delay). </ol> <p>A better statistic than mean is the median. It lacks any of the problems above, without contributing any new problems, other than relative difficulty of computation (arithmetic average computation is trivial). It must be noted here that it is possible to <a rel="nofollow" href="http://www-db.stanford.edu/~manku/papers/98sigmod-quantiles.ps">compute the median approximately in O(n) time (with a single pass over the data) and O(1) space</a>. The amount of space is traded against precision of the computation and is not large at all in practical terms. The code in <a href="../thrulay/">thrulay</a> (currently in src/thrulayd.c, search for ``Quantiles'') provides a free implementation of this algorithm with a BSD-like license (so it can be used in closed-source commercial products, too). Stanislav Shalunov http://shlang.com/writing/net-neutrality.html <p>Let us differentiate between three questions: (i) what should happen now when the telcos want to exploit their monopoly position to extract extra value out of information services, (ii) how should this develop in the future, and (iii) whether to support the net neutrality bill as it appears to be shaping up. The third (and the first) questions are more urgent, but the second one is more important. <h2>Net neutrality in the near term</h2> <p>The phone and cable TV companies do not enjoy their monopoly position because they smartly invested in the right market. The monopoly was created by the government and to a large extent continues due to regulation. It is disingenuous at best for them to claim the right to do what they want with their pipes when their pipes are the only pipes into consumers' residences because of government-granted monopoly. The government, in the form of the Congress, the FCC, the courts, and the municipalities, significantly affects the sector and, to a large extent, prevents new entries. <p>The phone and cable companies now have the ability to blackmail the consumer and the content provider. The only reason this can happen is because of lack of competition at the last mile. In the long term, the best solution is to have competition there and let the consumers choose whatever service they want. In the short term, when there's no competition (a duopoly does not create conditions for competition because it is a repeated prisoner's dilemma), the blackmail must be stopped. <p>The main danger of the blackmail isn't even in the extra costs of information services phone and cable companies seek to impose. These extra costs would act as privately imposed taxes (with all the negative effects of taxes and none of the positive ones), benefiting shareholders of the phone and cable companies at the expense of everyone else, but also distorting the allocation of resources so that less information services are produced. The resulting reduced number of high-tech startups in this country would be bad enough, but there's even a greater danger. The particular way in which the phone and cable companies seek to impose extra costs can destroy the Internet as we know it and replace it with a pay-to-play complex-billing system. This would greatly multiply the tax-like effects of extra charges. <p>It is desirable to stop the blackmail in a way that does not itself significantly distort the future evolution of the Internet. The disease of last-mile monopoly is a dangerous one, but the cure must not be worse than it. <p>What is needed is a small and a measured change in the regulatory regime that allows the Internet to be free of the blackmail until a better solution (competition at the last mile) arrives. <p>The phone and cable companies, today, are allowed to have the cake and eat it with respect to the common carrier status. They enjoy all the traditional benefits of the status (not being liable for what passes through their networks) with none of the obligations. If this loophole is removed, their entire scheme crumbles. Of course they would not accept the liability for viruses and copyright infringement that might occur on their networks; the cost of such liability exclusion for them must be network neutrality. Networks that wish to retain liability exclusion must accept obligations of the common carrier status. <h2>Internet access in the long term</h2> <p>Once the immediate threat of the Internet being replaced with a system that requires a quarter for each connection is stopped, we need to think about the future and fostering competitive environment in the area of last-mile access. <p>A desirable situation there would be consumers' ability to switch ISPs, choosing from at least half-dozen (and preferably more), with low transaction costs. A switch from one ISP to another should cost at most tens of dollars. It is realistic to expect that in such an environment in a few years from now, 1-Gb/s home connectivity would cost small number of tens of dollars per month---what the consumer now pays for measly connectivity roughly 1000 times more narrow. <p>First, let us examine two non-solutions: (i) wireless and (ii) municipally operated broadband. It is tempting to consider these as at least partial solutions to the problem. However, they provide no more than nice extras; talking about them, thus, obscures the real problem. <p>Wireless (in the form of wifi) is an important instrument to provide connectivity, especially to mobile devices. However, it does not, per se, solve the problem of Internet access for residences. The access points need to be connected somewhere. Visions of self-organizing hive-like ad hoc networks haven't yet realized. Research in this area might yield more results, but so far, the only practical way to provide wifi coverage is to connect the wireless access points to a fixed network. The density of access points would not be significantly lower than the density of residences, especially in suburban (and certainly in exurban) areas. More importantly, there is not enough spectrum where propagation has desirable properties to give every household a 1-Gb/s link. Every street would still need to have fiber or copper, to which access points on poles would need to connect. Having fiber or copper on each street isn't much easier than having it in each residence. <p>Municipal broadband is another appealing possibility. The roads, after all, are maintained by the municipality; why not the information superhighway? However, two counter-points are strong: (i) municipal broadband would freeze the pace of innovation (the roads are built using the same technology as decades ago) and (ii) municipal services are not delivered as efficiently as private ones. <p>There are two ways in which the goal of being able to switch from one ISP to another could be accomplished: the fiber that goes to the home might not be the property of the ISP, and there might be many strands of fiber that go in. Dark fiber is quite inexpensive, and having more strands is by far the easiest solution. It will also allow marketplace experimentation with multiple ISPs. <p>The crucial resource that needs to be managed by the municipalities and that the municipalities are not managing very well is digging up the road to lay fiber. The municipalities, rightly, refuse to let the road to be dug up too frequently (usually, no more often than once in five years). The municipalities, however, wrongly allow the road to be dug up to lay just half-dozen strands of fiber. This is a waste; the reason the phone and cable companies do it is to create artificial scarcity. The municipalities need to auction off permits to dig up the street with the purpose of laying down hundreds and thousands of strands of fiber in trunk fiber optic cables (a 1-inch duct can contain up to 96 strands of fiber). Consumers, then, will have multiple strands of dark fiber coming to their houses. A consumer who switches from one ISP to another will simply have the other ISP buy several strands for his house from the municipality's agent that put the fiber in (or perhaps from the first ISP if it is cheaper) and light them. Inexpensive, off-the-shelf components (e.g., Gigabit Ethernet) would be used to light the fiber. <h2>The Congress bill</h2> <p>I have not seen the bill that is reportedly shaping up. However, what I know of it indicates that it is likely to do much more harm than good. A good bill needs to have two things: (i) sunset provision and (ii) possibility to price the network in an economically efficient way. It appears, from what we know so far, that the bill will permanently prohibit economically efficient pricing and mandate a flat pricing structure, where users pay a monthly fee only. This would be extremely harmful, as it would instill the incentive to regulate the amount of traffic a user sends by giving him a tiny straw instead of a fat pipe. It would be tragic if we started with the belief in fat cheap network pipes and ended up outlawing them. <p>[<b>Update, May 2, 2006:</b> It appears that the draft Steven's bill satisfies both conditions: it <i>does</i> have a five-year sunset provision and does not prohibit economically efficient pricing. It appears to be very good, at least for now.] <h2>QoS shenanigans</h2> <p>The phone and cable companies claim that they need QoS to deliver services with guaranteed quality. QoS does nothing but discard or delay some traffic so that some other traffic can pass through. QoS is rationing of a scarce resource. But network capacity isn't a scarce resource. Tens of dollars a month can buy a 1-Gb/s connection; at such speed, no congestion will happen. As applications get faster in the future and consume more network capacity, the network can simply be made faster. With fiber, the sky is the limit. It is perhaps instructive to realize that <i>all</i> the traffic on the Internet is close in volume to what could be pushed through a single strand of fiber. When 1-Gb/s connection becomes too tight (with current trends, it might become too tight in perhaps a decade), it can simply be replaced with a 10-Gb/s connection, good for a few more years. Once the fiber is in the ground (and multiple strands go everywhere), this is easy to do. <h2>Two visions for the future</h2> <p>The phone and cable companies' vision of the future is that the American consumer, for years to come, will be tied to narrow-band access measured in tens of megabits per second (narrower than the default 100-Mb/s $10/month service in Seoul today). The content providers will need complex arrangements and negotiation with the phone company before even starting to offer their services to the consumer. Artificial scarcity will be created and profited from by the phone and cable companies. The phone and cable companies will delight in complex QoS schemes that discard or degrade non-affiliated traffic and impose difficult-to-understand billing structure. <p>Our vision of the future is wide-band Internet access (1Gb/s in both directions in a few years and more in the future) for the same amounts consumer pays today. With such speeds, no QoS or complex billing is necessary. Abundant network capacity encourages creation and delivery of new content and various information services. Simple and robust network architecture provides a solid foundation for the future, where even faster data transfers will be possible over the same fiber. <hr> <p>Additionally, see my presentation <a href="../talks/20060205-Tempe-Net-Neutrality.pdf">QoS Debate --- Net Neutrality, given at a panel at Net@EDU</a> meeting (<a href="../talks/20060205-Tempe-Net-Neutrality.mp3">Podcast of the talk in MP3 format</a>). <hr> <p>I do a bit of work related to net neutrality on behalf of Internet2; however, this essay is not part of it and represents my personal opinion only. Stanislav Shalunov http://shlang.com/writing/threads-tasks.html <p><blockquote> <strong>Summary:</strong> The common fallacy that concurrent tasks require threads is debunked. I am not an expert on threads and feel awkward writing this essay, but I have not found any concise formulation of this argument anywhere and had to write it down. Feel free to correct me. </blockquote> <p align="center"><i>An event queue is even cute.</i></p> <h2>Definitions</h2> The following are some of the terms used below. I don't insist that these are the correct definitions; I simply need them to explain the differences. <dl> <dt>Kernel Scheduling Entity (KSE) <dd>The unit of kernel scheduling. Something that can receive timeslices from the kernel, be preempted, and block on kernel events. <dt>Task <dd>A sequence of actions to be performed by a program while keeping the same context. <dt>Coroutines <dd>Concurrently executing pieces of code within the same program that use cooperative multitasking to ensure each makes progress. <dt>Thread <dd>The term is very much overloaded. I use it here to refer to POSIX threads, Microsoft Windows threads, Solaris threads, etc. Details of implementations can differ (e.g., FreeBSD has several different POSIX thread implementations that work differently). <dt>Process <dd>I'm afraid to try to give a definition of a process after all the giants who failed. You know what a process is. An operating system process, a KSE with its own address space. <dt>Event loop <dd>A common technique of organizing a program that obtains external events of some kind and processes them as they come. </dl> <h2>An example</h2> Suppose we need to write a program that: <ul> <li>Binds to a TCP socket and listens for connections. <li>Within each TCP connection, implements a calculator that reads lines of arithmetic expressions and outputs their results. The client can either enter things like ``<code>(3+4)*6</code>'' or can enter assignments like ``<code>size=6+15/3</code>''; variables to which something was assigned before can be used in subsequent expressions. Different requests are separated by newlines. The requests can be pipelined; i.e., the client does not need to wait for a response to a previous request before sending the next one. </ul> How would you solve this toy problem? Stop reading until you formulate the answer. <p>Programmers that come from different traditions can suggest different ways of implementing it: <ol> <li>With processes: a process enters a blocking <code>accept()</code> loop; when a new connection is established, a child process is spawned with a <code>fork()</code> call; any given child process handles its own session: it reads requests, keeps a data structure for the context (perhaps a hash with variable names as keys), and writes responses. There's no communication between the children, and very little communication between a child and the parent (the parent needs to spawn the child, then read and discard or record its exit status). <li>With threads: a similar architecture with threads replacing processes. <li>With an event loop: only a single routine executes; a data structure records the state of the system; the data structure could be, e.g., hash of hashes (client identifiers such as sockets on first level of hashing and variable names on the second level of hashing); another data structure is used to record partial inputs (e.g., a hash with client identifiers as keys and strings as values); an analogous data structure holds outputs to be sent to different clients; all sockets are put into non-blocking mode; a <code>select()</code> loop governs program execution. </ol> Do you fit into any of these categories? If you've selected the solution with processes, you're probably a UNIX programmer who hasn't been exposed to Solaris for too long; you might enjoy reading this. If you've selected an event loop, you already know what I'm going to say; don't waste your time---go program something. If you've selected the threads solution, I wrote this for you. If you didn't get the event loop business or think that parallelism (in the sense of multiple tasks) necessarily requires threads, I would like to point out that it does not. <p>(If your solution doesn't fit into this trichotomy, you can go back to writing your program that constructively extends Rice's theorem by taking two programs, performer and detector, on input, and always, as long as the detector can print ``no'', always stops and prints ``yes'' or ``no'', and prints ``yes'' with performer on input, outputs a program that represents the same function as the performer and that elicits ``no'' from the detector. I know, junior-level nonsense. Well then, try it in Lisp. Sorry about the digression; I do need to deal with hecklers.) <h2>Other solutions</h2> Many other solutions for the problem exist [C10K]. They are, generally, either refinements of one of the approaches (e.g., a particular mechanism, such as <code>select()</code>, <code>poll()</code>, <code>/dev/poll</code>, or <code>kqueue()</code> to obtain notification of events), or combinations of the approaches (e.g., an event loop running within multiple threads). These details can be extremely important and difficult to get right [DARC04], but here only the basic approach is discussed. <h2>The Polemics</h2> The widely cited argument against threads [OUST95] actually advises to use threads for high-end applications (e.g., databases) and applications that require true CPU concurrency on multi-processor systems. It advises to mostly keep even multi-threaded code single-threaded, with just event-specific threads. Interestingly, the author both claims that threads can have (usually have? always have?) poor performance and advises to use them only in performance-critical contexts, reserving event programming for distributed systems, GUIs, and the like. <p>What is widely seen as an attempt to refute [BEHR03] the anti-thread thesis uses, as its core argument, the fact that threads and events are dual (this was known for a long time [LAUE78]). Thus, you can do the same things with the two mechanisms, with essentially the same performance. Therefore, one should use the mechanism that's more natural for the task at hand. The authors proceed to claim that for most tasks, threads are the more natural approach; two very valid points are (i) the difficulty of error handling with events and (ii) the greater probability of memory leaks with events. The authors admit that for fan-in and fan-out operations, such as multicasting, the event model is more natural. <p>But (yes, there is a ``but'' the size of a blue whale), the paper states that there is no well-performing thread implementation (other than what the authors rigged up for their paper). The authors propose a number of improvements to address performance and other limitations. Until these improvements are understood, implemented, deployed, and tested at least by a few of popular operating systems, the authors' argument that threads <i>can</i> work well is also an argument that they currently <i>do not</i>. Incidentally, the authors argue for a user-level thread implementation with cooperative multitasking, with the kernel knowing little about threads. Most of the recent movement in this area is towards more kernel involvement in threads, not less (e.g., Solaris 10 replaces an N:M implementation, where multiple threads, implementing multiple tasks, can exist within a KSE, with a 1:1 implementation, where one thread is exactly one KSE). <h2>So, should I use threads or events?</h2> In theory, a proper implementation of threads is as good as a proper implementation of events, so one can choose freely the mechanism that is more convenient. If it's convenient to use a thread for each task, do so; if fan-out makes it more convenient to write the application as an event loop, do so. <p>However, in practice, currently, since all threading libraries are so bad, you should seriously consider using events. Here are the possible reasons to use threads: <ol> <li>You need concurrency for a performance-critical application, and a hardware source of concurrency exists; <li>You don't need portability and are using a good threads library not available to the rest of the world (and threads are the more natural approach to your problem); <li>You were taught thread-based programming, and cannot think off-hand of another way to implement tasks. </ol> The first reason is a solid one. The second is dubious: you trade off portability for being able to write more natural code; if one always did that, almost all programs would have been written in domain-specific languages running on exotic platforms. The third, needless to say, is not a reason at all. <h2>OK, I'm using threads. How many should I have?</h2> What is your reason for using threads? If it's 3 (that you want to avoid using events), I can't really help you, as I don't consider this a valid reason. If it's 2 (that it's more natural and hurts nothing), then use as many threads as you have tasks or whatever feels natural. If it's 1 (that you need to exploit potential parallelism in hardware), then the number should be related to the number of things that can happen in parallel. <p>The number of parallel processes that can happen in a computer is related to the number of CPUs, disks (disk I/O events, unfortunately, cannot be efficiently used in event-based programs on UNIX: even if you set the file descriptor to nonblocking, you will still wait, as <code>select()</code> will always report that file descriptors for regular files are ready, even if it'll take 10ms for a disk seek before data comes), and various other I/O channels that behave like disks. The number of threads should be related to this degree of potential parallelism; not necessarily equal (performance might be better if the number of threads is the degree of parallelism plus or times some small constant), but related to it, rather than the number of tasks. Note that as the number of threads gets larger than the number of CPUs, the potential for unnecessary context switches increases, so the small constant mentioned above should actually be small. <h2>References</h2> <dl> <dt>[LAUE78] <dd><a href="http://www.sics.se/~adam/pt/duality78.pdf" rel="nofollow">Hugh C. Lauer, et al. On the duality of operating system structures. IR1A. October 1978.</a> <dt>[OUST95] <dd><a href="http://home.pacbell.net/ouster/threads.pdf" rel="nofollow"> John Ousterhout. Why threads are a bad idea (for most purposes). September 1995.</a> <dt>[BEHR03] <dd><a href="http://www.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren.pdf" rel="nofollow">Rob von Behren, et al. Why events are a bad idea (for high-concurrency servers). HotOS IX. May 2003.</a> <dt>[C10K] <dd><a href="http://www.kegel.com/c10k.html" rel="nofollow">Dan Kegel. The C10K problem. November 2003.</a> <dt>[DARC04] <dd><a href="http://pl.atyp.us/content/tech/servers.html" rel="nofollow">Jeff Darcy. High-performance server architecture. March 2004.</a> </dl> Stanislav Shalunov http://shlang.com/writing/soc2005.html <blockquote> <strong>Summary:</strong> I record my experience so far as an Internet2 mentor for the Summer of Code program. The primary audience are students, especially those who plan to apply for similar grants in the future. Secondarily, I address grant-giving organizations and individuals and fellow mentors. I give my best advice on writing a good application to the students, so that the best of them can make it through the sieve. </blockquote> <p>Google's Summer of Code program in 2005 not only allowed Internet2 to fund students to work on our open-source projects, but, by virtue of attracting the best students, allowed us to find contributors of quality we would have difficulty finding otherwise---money or no money. We are immensely grateful to Google for this. <h2>Time line</h2> <p>I wrote the original version of the <a rel="nofollow" href="http://transport.internet2.edu/student-projects.html">Summer of Code ideas page</a> on May 31. June 14 was the deadline for applications. June 24 was the deadline for notifications. Internally, Google set June 23 as the deadline for mentoring organizations to finish ranking their applicants. The ideas page contains our suggestions on proposal content, etc. This is much more informal advice to future applicants. <h2>Value of transparency</h2> <p>I believe it benefits everyone to make the process as transparent as possible. Having a transparent process will help the best students win the grants, thus giving Google the most bang for their buck and the mentoring organizations the most amount of output. Having an opaque process, on the other hand, will make students guess at the best strategy, and, thus, introduce some randomization into the process, which will help only one party: the group of students who would not have received the grants meritoriously. This year, the process was being developed as we went. I will attempt to record here what happened inside the selection process and give my best advice on how to prepare the application the next time, or generally for a similar grant. I am only giving my best advice; it would have worked on me; I don't know whether I am a typical reviewer of these applications, but I have no reason to believe I am not. I would encourage other mentors to record their thoughts: the increased amount of information will help the best students win. As mentors, we're interested in getting the best students, so self-interest dictates the need for making one's decision process transparent. <h2>Where to aim</h2> <P>If you want to win in a competitive stipend program like the Summer of Code, you'll be competing against an unknown number of people of unknown quality from all over the world. I know you're good, but they're good, too. You need to try to write the best application that the mentor will see. This is an easy guide to follow; it depends on your imagination, so let the imagination run free. Are you an undergraduate? You'll be competing against graduate students. Are you a grad? There will be ABDs in the pile. ABD from a top school with a list of relevant peer-reviewed publications and many years of experience? A few people of this level of qualification were rejected by us this time around due to lack of commitment to writing a good proposal. There will be smart and eager undergraduates who will put days or weeks of full-time effort into the proposal. You get the drift. You need to aim to write the best application. You might miss; your application might turn out to be second-best or even fifth. The point is, if you aim this high, and are serious about it, and are any good, you'll get the grant. <h2>How to write the best application in the pile</h2> <p>As I read the applications, I am concerned most about one thing: can the applicant really do what he promises to do? There are many ways to convince me, of course, but here are some typical ones: resume, related prior work, started work on the project towards which the applicant is submitting his proposal. Of these, resume is by far the least efficient. Don't try to emulate cookie-cutter (i.e., ``professional-looking'') resumes. The value of such resumes in job hunting is dubious, but for grants, they simply don't help. Usually, resumes are read by human resources specialists, i.e., people whose job is to read resumes. HR specialists will typically not understand what you do (unless you're applying for an HR position, that is), and will not judge the merit. They judge ``fit.'' To fit the position well, you need to submit a resume that has a set of keywords as closely resembling the job ad as possible. I am not an HR specialist, so I don't even read your list of skills. Keyword comparison is a poor predictor of performance, or Google would have automated the whole selection process. Prior related work (and any prior work that I can see) is much better than resume. Having performed similar work in the past is a good predictor of ability to perform such work in the future. Even better is work on the suggested project: if you can show that you're already doing the work that needs to be done, and doing it successfully, you have solved the problem of convincing me you can do it. You might still burn out, have a personal emergency, or get hit by a bus, but I have high confidence that you're capable of doing the work. Another excellent way to convince me in your ability to do the work is to present a detailed plan for doing it, with a good schedule. Such a plan is a great complement to any other ways. All applications we accepted had good plans, and not having a good plan was grounds for rejection even for people with very impressive resumes. <p>As you read this advice, I'm sure you're thinking ``That's a lot of work for an application.'' Exactly. That's the point. You prove your ability to do work with work. The more work you put into an application, the better are your chances of success. As the number of submitted applications increases, your chances of being accepted for at least one project actually decrease. Put all the work into one (or, at most, two) applications. Be prepared to spend at least two full days preparing your application. If you want to make sure you succeed, spend a week. Study the literature. Read the code. Prepare a plan. Ask questions. (No, I won't be annoyed. Not even a little bit. OK, I might be annoyed if you're too lazy to look something up, but never if you want to ask about the best approach or the meaning of something.) Write code. Can you write a prototype that implements some part of the system that I am asking for? If not, you probably won't be able to do the project. If you can, and do so, you stand out from the crowd immediately. If you can, and don't, you've just harmed your chances. The more work, the better. This isn't wasted work, either. If you put a week into preparing your application (and if you're any good), it won't be rejected. So, you simply did the work ahead of time. <h2>The bad applications</h2> <p>We received 126 applications. We selected 12 of these for approval. Google gave us 10 slots, so the top 10 choices were funded. It was very easy to reject perhaps a third of applications that were very poor, indeed. Where do people get the notion that cutting and pasting my own text from the project description so that I can review it is worth $4500? Who are the people who think that one-line or one-paragraph applications will get them the stipend? Some of these very brief applications were written by people with impressive resumes. It was obvious they could do better. Yet they doomed themselves to failure. I'm guessing that many of them didn't want to do work before being paid for it. I'm guessing some of them applied for too many projects and thus could not write good applications. I don't know what exactly were the reasons, but it isn't possible, for the vast majority of people (OK, I'll make an exception for celebrity hackers), to convince a mentor you'll be able to do the job with an application that was written in five minutes. These applications consume a bit of mentors' time to reject them, but they aren't a threat to serious applications. There is no way a slot would be taken by a one-liner. <h2>The best applications</h2> <p>It was almost as easy as the first round of rejection to select the few top choices. These were the people who did what I advise above to do; they submitted outstanding applications. You want to be in this part of the pile, where we're not talking about whether to take you, but about the exact deliverables or other minor tweaks to your project. <h2>The middle</h2> <p>The difficult part was in the pruning of the large number of very good and excellent applications to fit the number of slots we would get. Many excellent applications were rejected in the process. We could not take everyone, and Google allocated a finite amount of money for the program (they did double the amount from $1M to $2M when they saw how many applications there were). This is not a good place to be: on one hand, you've invested quite a bit into your application and have well-grounded hopes that it succeeds; on the other hand, the success is far from being guaranteed. Did we make any choices that we would have reversed, given more complete information about the students' abilities? I'm sure. So, as an applicant, you need to give us this more complete information. In a high-profile program such as anything run by Google, the entire scale is shifted towards excellence. Excellence is the norm, once you quickly filter out the really bad applications. Excellence, thus, isn't enough. Aim higher. <hr> I would very much like to thank everyone who applied to the Summer of Code program with Internet2 as their mentoring organization. It is a privilege to be part of the Summer of Code program for us. <hr> <h2>Other Mentors' Advice</h2> <ul> <li><a href="http://summer.cs.pdx.edu/?q=node/12" rel="nofollow">Portland State University guidelines on writing proposals</a> <li><a href="http://venge.net/monotone/summerofcode.html" rel="nofollow">Guidelines from Monotone</a> <li><a href="../tmp/rwatson-soc.txt">Advice from Robert Watson</a> </ul> Stanislav Shalunov http://shlang.com/writing/protocol-design.html <p>This rant considers a few points of protocol design for computer communications networks. My opinion only. <h2>Text vs binary</h2> According to Padlipsky, the first protocol to use ASCII strings, terminated by newlines, as commands was FTP. This was so that you could, after logging onto a terminal server, send files directly from a file server to a printer. This is also the reason the FTP protocol is layered on top of TELNET (you would use the telnet program to connect to the printer and the file server and issue FTP commands manually). This was a reasonable design for the time when people actually typed FTP protocol messages; those who are used to modern command-line interfaces would find it a bit too complicated, but usable. <p>Since then, a large number of protocols that use difficult-to-parse pseudo-human-friendly formats of messages have been foisted upon us. Enough already. When humans talk to computers, the protocol needs to be human-friendly, because the computer is the human's servant; when computers talk to each other, or when different programs talk to each other within the same computer, the protocol has no reason, whatsoever, to be human-friendly. Debugging (which is often given as an excuse for unwieldy protocol design) of unnecessarily complex protocols is more, not less, difficult than debugging of protocols that are only complex enough to do what they need to. <h2>Round-trip times</h2> Consider the case of email transmission between strangers on a non-secure packet network. (This is the problem SMTP/TCP/IP solves.) How many round-trip times are necessary to send a short message (less than a maximum packet size) when no packets are lost? Here's what needs to happen: <ol> <li>sender --&gt; receiver: Message. <li>receiver --&gt; sender: Did you send message X? I got it. <li>sender --&gt; receiver: Yes, I did send message X. </ol> That's three messages for 1.5 round-trip times. How does SMTP compare? (The following exchange has some silly places where several messages in a row are sent by one side; this is how it works on FreeBSD 5.3. It doesn't affect the number of round-trip times, but is shown here for realism. If these messages were merged and sent in a single packet, the exchange could have taken 20 or even 18 packets. Note that I am not complaining about the number of packets, but about the number of round-trip times.) <ol> <li>sender --&gt; receiver: SYN. <li>receiver --&gt; sender: SYN+ACK. <li>sender --&gt; receiver: ACK. <li>receiver --&gt; sender: ACK. <li>receiver --&gt; sender: 220. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: HELO. <li>receiver --&gt; sender: 250. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: MAIL. <li>receiver --&gt; sender: 250. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: RCPT. <li>receiver --&gt; sender: 250. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: DATA. <li>receiver --&gt; sender: 354. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: Message. <li>receiver --&gt; sender: 250. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: QUIT. <li>receiver --&gt; sender: 221. <li>receiver --&gt; sender: FIN. <li>sender --&gt; receiver: ACK. <li>sender --&gt; receiver: FIN+ACK. <li>receiver --&gt; sender: ACK. </ol> That's nine round-trip times, or six times more than necessary. <p>I use SMTP as an example only here. If one were to try to analyse ESMTP as it is deployed today for the number of round-trip times, two considerations would need to be taken into view: the PIPELINING extension, which can reduce the number of round-trip times by two, and DNS queries, which, since they are all done serially in this case, can increase the number of round-trip times quite a bit. <h2>Bit economy</h2> Perversely, the same crowd happy with wasting round-trip times can easily come down on anyone whose protocol is ``wasteful.'' Bits are cheap. Every year, they get twice cheaper. Round-trip times are fundamental and bounded from below by distance divided by speed of light. <p>Use as many bits as needed. Worry about bits only when something that would fit into a single packet stops doing so. <h2>Layers</h2> Layers are a really clever idea. They let you trade off effort for efficiency. When the efficiency that you lose is in bits alone, using layers liberally makes sense. <p>Other ways to lose efficiency could be in round-trip times, CPU use, protocol complexity, and implementation size (and, thus, the number of bugs). In addition, at some point, layers start interacting in ways that are difficult to predict; such emergent behaviors can be surprising to protocol designers. These are arguments in favor of fewer layers (it is, perhaps, not surprising that the more successful Internet has fewer layers than the deadborn ISO network design). An even more important argument is that layers can be misleading; for example, most applications using TLS or SSH as a layer between transport and application don't get what their designers might think they are getting from the security layer: existential forgery is trivially possible with most deployed applications; such state of affairs would rightly be considered disastrous by most designers if they considered the system as a whole. <h2>Be liberal---or not</h2> This is the most harmful part of the conventional wisdom. This is not how you make protocols interoperate; this is how you make them interoperate poorly, while burying booby traps for the future. <p>Be pedantic in what you send and fascist in what you accept. <p>The good protocols (IPv4, IPv6, BGP, etc.) are actually not at all liberal in what they accept. Being liberal in what you accept gives you HTML/HTTP. <h2>Protocol versions and feature negotiation</h2> Protocols evolve. How does one provide for the evolution in the beginning? Different implementations of an evolving protocol provide different sets of features. How does one implementation learn what the other supports? Here's how: <ol> <li>IP version; <li>IP protocol; <li>Port number (for IP protocols 6 and 17); <li>Protocol version; <li>Feature negotiation; <li>Trial and error; <li>Wild optimistic guess. </ol> There are too many ways to negotiate that interact in unpredictable ways. <p>Particularly useless are protocol version numbers (with possible exception of the stablest, the most basic protocols, such as IP). Version numbers provide for linear ordering of sets of supported features, but set inclusion relationship is not a linear order. Not only is protocol version mechanism limited, it is redundant, too; protocol versions don't buy you anything you don't already have with port numbers. Better in that respect is feature negotiation. Even feature negotiation is too often wasteful and unnecessarily complicated. <p>Where it gets harmful is security. There's a push to make everything---every little primitive, every little option---negotiable. The resulting monsters are impossible to analyze and are almost certain to contain security holes. In many cases, the security holes are inserted intentionally for ``debugging purposes.'' <p>There's far too much versioning and feature negotiation. Port numbers do the job quite often. When they don't, capability strings or feature negotiation work better than protocol versions. <hr> <h2>Acknowledgments</h2> I would like to thank Simon Leinen for his comments and for discussion. Stanislav Shalunov http://shlang.com/writing/ink-test.html <p><blockquote><strong>Graphite pencils and gel pens are the most archival, while color Sharpie markers are the worst. Mechanical pencils, ballpoint pens, and all children's pencils and crayons are in between.</strong></blockquote></p> <p>An accelerated aging test of archival properties of several writing implements was performed as follows: the implements were used to write (or print) their own names on three separate sheets of alkaline Strathmore Pure Cotton paper; one of the sheets was placed in a plastic sleeve and was kept in a metal filing cabinet; one was taped from the inside to a second-floor window facing south (the window is made of two layers of ordinary float glass); the remaining sheet was tacked at four corners to a guard rail on the back porch, facing south. The first sheet was not intentionally exposed to any environmental factors and was kept for comparison purposes. The second sheet received ample light (including ultraviolet light that passes through float glass); thus, light-fastness was tested. The third sheet was exposed to unfiltered sunlight, acidic rain, snow, and repeated cycles of freezing and thawing in Ann Arbor. The test started on 2003-11-05 and was discontinued on 2004-04-02 when the sheet outside was torn by the wind; the experiment thus lasted through Michigan winter for 150 days. <p>The results are presented in the following table. <p> <table align="center" border> <thead> <tr><th>Writing Implement<th>Exposed to Light<th>Exposed to Light, Rain, etc. <caption>Ink Test Results</caption> <tbody> <tr><td>HP LaserJet 4100 laser printer</td><td>No change</td><td>No change</td> <tr><td>Pigma Micron 0.5 black marker</td><td>No change</td><td>No change</td> <tr><td>Higgins India Ink</td><td>No change</td><td>No change</td> <tr><td>Pentel esharp 0.5 pencil</td><td>No change</td><td>No change</td> <tr><td>General's Kimberly 2H pencil</td><td>No change</td><td>No change</td> <tr><td>Conte graphite 2B pencil</td><td>No change</td><td>No change</td> <tr><td>Uniball Vision pen (blue)</td><td>No change</td><td>No change</td> <tr><td>Uniball gelgrip pen (black)</td><td>No change</td><td>No change</td> <tr><td>Uniball UM-100 pen (black)</td><td>No change</td><td>No change</td> <tr><td>Zebra rollerball pen (black)</td><td>No change</td><td>No change</td> <tr><td>Zebra rollerball pen (blue)</td><td>No change</td><td>No change</td> <tr><td>Zebra Jimnie Gel Roller Ball Medium pen (red)</td><td>No change</td><td>Light fading</td> <tr><td>Panasonic FP-7824 photocopy machine</td><td>No change</td><td>Small chunks of black missing</td> <tr><td>General's charcoal 4B pencil</td><td>No change</td><td>Smudged</td> <tr><td>Cheap felt tip pen (black)</td><td>No change</td><td>Almost imperceptible smudging</td> <tr><td>Bic Great Erase 0.7 pencil, Japan</td><td>No change</td><td>Smudged</td> <tr><td>Conte carbon HB pencil</td><td>No change</td><td>Light smudging</td> <tr><td>Conte sepia</td><td>No change</td><td>Light smudging</td> <tr><td>Conte sanguine</td><td>No change</td><td>Smudged</td> <tr><td>Faber 2663 china marker</td><td>No change</td><td>Possibly light smudging</td> <tr><td>Sharpie twin tip (either end, black)</td><td>No change</td><td>Light smudging</td> <tr><td>Super Sharpie (black)</td><td>No change (seeped through)</td><td>Light smudging</td> <tr><td>Cheap ballpoint pen (black)</td><td>Significantly faded</td><td>Barely legible</td> <tr><td>Pentel RSVP BK91 ballpoint pen (black)</td><td>Significantly faded</td><td>Almost no trace left</td> <tr><td>Cheap ballpoint pen (black)</td><td>Significantly faded</td><td>Barely legible</td> <tr><td>Bic Atlantis ballpoint pen (blue)</td><td>Faded</td><td>Barely legible</td> <tr><td>Pilot PB-S fine ballpoint pen (blue)</td><td>Barely legible</td><td>Almost no trace left</td> <tr><td>General's watercolor pencil (blue)</td><td>Light fading</td><td>Light fading, light smudging</td> <tr><td>General's watercolor pencil (black)</td><td>Light fading</td><td>Light fading, light smudging</td> <tr><td>General's watercolor pencil (red)</td><td>Light fading</td><td>Significantly washed out</td> <tr><td>General's watercolor pencil (brown)</td><td>Light fading</td><td>Light fading, light smudging</td> <tr><td>General's watercolor pencil (orange)</td><td>Significant fading</td><td>Very significantly washed out and faded</td> <tr><td>Red wax crayon</td><td>Barely legible</td><td>Little trace left</td> <tr><td>Cheap ballpoint pen (black)</td><td>Lightly faded</td><td>Significantly faded</td> <tr><td>Papermate ballpoint pen (red)</td><td>Light fading</td><td>No visible trace left</td> <tr><td>Bic Atlantis ballpoint pen (red)</td><td>Significantly faded</td><td>No visible trace left</td> <tr><td>Super Sharpie (red)</td><td>Significantly faded</td><td>No visible trace left</td> <tr><td>Super Sharpie (blue)</td><td>Significantly faded and fully discolored</td><td>No visible trace left</td> <tr><td>Cheap ballpoint pen (rose)</td><td>Barely legible</td><td>No visible trace left</td> <tr><td>Sharpie extra fine (blue)</td><td>Very significantly faded and discolored</td><td>No visible trace left</td> </table> <p>Note: I have identified the writing implements as fully as I could; in the case of giveaways and hotel pens, where manufacturer is not specified and reproducibly obtaining the same kind of pen is not possible, I simply specified ``cheap'' instead of the name of the (unknown) manufacturer. <p>Salient points: <ul> <li>Of all commonly available writing implements tested, hard graphite pencils performed the best. <li>Gel pens generally performed well, but one had some fading. <li>The popular belief that most or all ballpoint black pens use carbon pigment is false. Most appear to use unstable dyes. <li>Sharpie markers other than black are surprisingly unstable. Black Sharpie markers are useful, but inferior to precise and pleasant Pigma Micron markers (cost is not substantially different if ordering Pigma by mail). <li>While carbon-based toner in laser printers and copy machines is archival, adhesion can be a problem. Copy machine output deteriorated. (One can test toner adhesion quickly with an eraser.) <li>The archival Strathmore Pure Cotton ultra-white paper itself faded to about the tone of Strathmore natural white paper. </ul> <hr> <p>The test is unscientific (I could have exerted different pressure on the three different sheets of paper; controlled source of light would have been preferable) and the findings should be viewed as my personal opinion. Stanislav Shalunov http://shlang.com/writing/press-freedom.html <p align="center">A comment on ``Third Annual Worldwide Press Freedom Index'' by Reporters Without Borders, 2004.</p> <p>The press freedom index counts events of abuse against journalists and freedom of press and ranks countries based on the number of such events. This method is flawed, because it penalizes large countries and makes small countries appear better than they are. For example, Trinidad and Tobago, with 2 events, is in 11th place of the ranking, tying with Estonia, Germany, and Sweden; yet the population of Trinidad and Tobago is just a little over a million while Germany has 82M people. Clearly Germany has the better record of press freedom than Trinidad and Tobago with the same number events and population that's eighty times larger. <p>Based on the original index, I generate different ranking, based on the number of events per million of population (rather than unscaled events). <p>Methodological note: The original method is clearly flawed because it heavily penalizes large countries. The method used here is more fair, but it still could be improved. Namely, scaling events based on population favors countries that have lower percentage of their population working as reporters and journalists in general. A better method would use the number of reporters and journalists (both foreign and domestic, professional and amateur) working in a given country. Unfortunately, I have no data for the number of journalists in different countries. I suspect that if such data were available, countries such as China and India (and perhaps Brazil) would score worse than they do in my table, while countries with large percentages of journalists (some bloggers should qualify as amateur journalists), such as United States and Western European countries would score better. <p>Data sources: The population data comes from the CIA factbook for 2004. Timor-Leste data was lacking from that source, so an alternative source was used. Similarly, an alternative source was used for the disputed territories in Israel; further, press freedom abuses committed by the Israeli army and by the Palestinian authority were merged to better characterize the degree of press freedom on that territory (after all, a murdered journalist is a murdered journalist, regardless of the murderer). <p> <table> <tr><td>Rank</td><td>Events per million</td><td>Country</td><td>Events</td><td>Population</td> <tr><td>1</td><td>0.0137</td><td>United States of America (American territory)</td><td>4.00</td><td>293027571</td> <tr><td>2</td><td>0.0243</td><td>Germany</td><td>2.00</td><td>82424609</td> <tr><td>3</td><td>0.0306</td><td>Netherlands</td><td>0.50</td><td>16318199</td> <tr><td>4</td><td>0.0361</td><td>India</td><td>38.50</td><td>1065070607</td> <tr><td>5</td><td>0.0579</td><td>France</td><td>3.50</td><td>60424213</td> <tr><td>6</td><td>0.0671</td><td>Switzerland</td><td>0.50</td><td>7450867</td> <tr><td>7</td><td>0.0711</td><td>China</td><td>92.33</td><td>1298847624</td> <tr><td>8</td><td>0.0785</td><td>Japan</td><td>10.00</td><td>127333002</td> <tr><td>9</td><td>0.0896</td><td>Brazil</td><td>16.50</td><td>184101109</td> <tr><td>10</td><td>0.0922</td><td>Slovakia</td><td>0.50</td><td>5423567</td> <tr><td>11</td><td>0.0924</td><td>Denmark</td><td>0.50</td><td>5413392</td> <tr><td>12</td><td>0.0959</td><td>Finland</td><td>0.50</td><td>5214512</td> <tr><td>13</td><td>0.0996</td><td>United Kingdom</td><td>6.00</td><td>60270708</td> <tr><td>14</td><td>0.1024</td><td>Canada</td><td>3.33</td><td>32507874</td> <tr><td>15</td><td>0.1093</td><td>Norway</td><td>0.50</td><td>4574560</td> <tr><td>16</td><td>0.1170</td><td>South Africa</td><td>5.00</td><td>42718530</td> <tr><td>17</td><td>0.1260</td><td>Ireland</td><td>0.50</td><td>3969558</td> <tr><td>18</td><td>0.1550</td><td>Italy</td><td>9.00</td><td>58057477</td> <tr><td>19</td><td>0.1583</td><td>Indonesia</td><td>37.75</td><td>238452952</td> <tr><td>20</td><td>0.1678</td><td>New Zealand</td><td>0.67</td><td>3993817</td> <tr><td>21</td><td>0.1768</td><td>Poland</td><td>6.83</td><td>38626349</td> <tr><td>22</td><td>0.2158</td><td>Thailand</td><td>14.00</td><td>64865523</td> <tr><td>23</td><td>0.2226</td><td>Sweden</td><td>2.00</td><td>8986400</td> <tr><td>24</td><td>0.2234</td><td>Spain</td><td>9.00</td><td>40280780</td> <tr><td>25</td><td>0.2290</td><td>South Korea</td><td>11.13</td><td>48598175</td> <tr><td>26</td><td>0.2651</td><td>Mexico</td><td>27.83</td><td>104959594</td> <tr><td>27</td><td>0.2750</td><td>Nigeria</td><td>37.75</td><td>137253133</td> <tr><td>28</td><td>0.3416</td><td>Czech Republic</td><td>3.50</td><td>10246178</td> <tr><td>29</td><td>0.3573</td><td>Russia</td><td>51.38</td><td>143782338</td> <tr><td>30</td><td>0.3865</td><td>Belgium</td><td>4.00</td><td>10348276</td> <tr><td>31</td><td>0.3879</td><td>Pakistan</td><td>61.75</td><td>159196336</td> <tr><td>32</td><td>0.3963</td><td>Tanzania</td><td>14.50</td><td>36588225</td> <tr><td>33</td><td>0.3976</td><td>Austria</td><td>3.25</td><td>8174762</td> <tr><td>34</td><td>0.4247</td><td>Philippines</td><td>36.63</td><td>86241697</td> <tr><td>35</td><td>0.4276</td><td>Portugal</td><td>4.50</td><td>10524145</td> <tr><td>36</td><td>0.4336</td><td>Latvia</td><td>1.00</td><td>2306306</td> <tr><td>37</td><td>0.4422</td><td>Bangladesh</td><td>62.50</td><td>141340476</td> <tr><td>38</td><td>0.4771</td><td>Australia</td><td>9.50</td><td>19913144</td> <tr><td>39</td><td>0.5407</td><td>Turkey</td><td>37.25</td><td>68893918</td> <tr><td>40</td><td>0.5449</td><td>Argentina</td><td>21.33</td><td>39144753</td> <tr><td>41</td><td>0.5453</td><td>Ethiopia</td><td>37.00</td><td>67851281</td> <tr><td>42</td><td>0.5715</td><td>Egypt</td><td>43.50</td><td>76117421</td> <tr><td>43</td><td>0.5981</td><td>Hungary</td><td>6.00</td><td>10032375</td> <tr><td>44</td><td>0.6264</td><td>Taiwan</td><td>14.25</td><td>22749838</td> <tr><td>45</td><td>0.6320</td><td>Chile</td><td>10.00</td><td>15823957</td> <tr><td>46</td><td>0.6504</td><td>Ghana</td><td>13.50</td><td>20757032</td> <tr><td>47</td><td>0.6574</td><td>Greece</td><td>7.00</td><td>10647529</td> <tr><td>48</td><td>0.6948</td><td>Kenya</td><td>22.25</td><td>32021856</td> <tr><td>49</td><td>0.7586</td><td>Benin</td><td>5.50</td><td>7250033</td> <tr><td>50</td><td>0.7641</td><td>Dominican Republic</td><td>6.75</td><td>8833634</td> <tr><td>51</td><td>0.7976</td><td>Romania</td><td>17.83</td><td>22355551</td> <tr><td>52</td><td>0.8315</td><td>Lithuania</td><td>3.00</td><td>3607899</td> <tr><td>53</td><td>0.8638</td><td>Mozambique</td><td>16.25</td><td>18811731</td> <tr><td>54</td><td>0.8831</td><td>Democratic Republic of Congo</td><td>51.50</td><td>58317930</td> <tr><td>55</td><td>0.9089</td><td>Uganda</td><td>24.00</td><td>26404543</td> <tr><td>56</td><td>0.9108</td><td>El Salvador</td><td>6.00</td><td>6587541</td> <tr><td>57</td><td>0.9158</td><td>Bosnia and Herzegovina</td><td>3.67</td><td>4007608</td> <tr><td>58</td><td>0.9845</td><td>Venezuela</td><td>24.63</td><td>25017387</td> <tr><td>59</td><td>0.9908</td><td>Afghanistan</td><td>28.25</td><td>28513677</td> <tr><td>60</td><td>1.0507</td><td>Vietnam</td><td>86.88</td><td>82689518</td> <tr><td>61</td><td>1.0570</td><td>Madagascar</td><td>18.50</td><td>17501871</td> <tr><td>62</td><td>1.0641</td><td>Bulgaria</td><td>8.00</td><td>7517973</td> <tr><td>63</td><td>1.0685</td><td>Ukraine</td><td>51.00</td><td>47732079</td> <tr><td>64</td><td>1.0730</td><td>Mali</td><td>12.83</td><td>11956788</td> <tr><td>65</td><td>1.0941</td><td>Hong-Kong</td><td>7.50</td><td>6855125</td> <tr><td>66</td><td>1.1186</td><td>Slovenia</td><td>2.25</td><td>2011473</td> <tr><td>67</td><td>1.1198</td><td>Colombia</td><td>47.38</td><td>42310775</td> <tr><td>68</td><td>1.1303</td><td>Sudan</td><td>44.25</td><td>39148162</td> <tr><td>69</td><td>1.1345</td><td>Iran</td><td>78.30</td><td>69018924</td> <tr><td>70</td><td>1.1554</td><td>Guatemala</td><td>16.50</td><td>14280596</td> <tr><td>71</td><td>1.1971</td><td>Burkina Faso</td><td>16.25</td><td>13574820</td> <tr><td>72</td><td>1.2488</td><td>Ecuador</td><td>16.50</td><td>13212742</td> <tr><td>73</td><td>1.2905</td><td>Israel (Israeli territory)</td><td>8.00</td><td>6199008</td> <tr><td>74</td><td>1.3350</td><td>Morocco</td><td>43.00</td><td>32209101</td> <tr><td>75</td><td>1.3539</td><td>Algeria</td><td>43.50</td><td>32129324</td> <tr><td>76</td><td>1.4187</td><td>United States of America (in Iraq)</td><td>36.00</td><td>25374691</td> <tr><td>77</td><td>1.4522</td><td>Peru</td><td>40.00</td><td>27544305</td> <tr><td>78</td><td>1.4907</td><td>Estonia</td><td>2.00</td><td>1341664</td> <tr><td>79</td><td>1.5370</td><td>Jamaica</td><td>4.17</td><td>2713130</td> <tr><td>80</td><td>1.6135</td><td>Niger</td><td>18.33</td><td>11360538</td> <tr><td>81</td><td>1.6808</td><td>Cameroon</td><td>27.00</td><td>16063678</td> <tr><td>82</td><td>1.6933</td><td>Malaysia</td><td>39.83</td><td>23522482</td> <tr><td>83</td><td>1.6959</td><td>Paraguay</td><td>10.50</td><td>6191368</td> <tr><td>84</td><td>1.7009</td><td>Iceland</td><td>0.50</td><td>293966</td> <tr><td>85</td><td>1.7220</td><td>Honduras</td><td>11.75</td><td>6823568</td> <tr><td>86</td><td>1.8238</td><td>Trinidad and Tobago</td><td>2.00</td><td>1096585</td> <tr><td>87</td><td>1.8337</td><td>Sri Lanka</td><td>36.50</td><td>19905165</td> <tr><td>88</td><td>1.8594</td><td>Serbia and Montenegro</td><td>20.13</td><td>10825900</td> <tr><td>89</td><td>1.9285</td><td>Costa Rica</td><td>7.63</td><td>3956507</td> <tr><td>90</td><td>1.9738</td><td>Uzbekistan</td><td>52.13</td><td>26410416</td> <tr><td>91</td><td>1.9812</td><td>Senegal</td><td>21.50</td><td>10852147</td> <tr><td>92</td><td>2.1773</td><td>Nicaragua</td><td>11.67</td><td>5359759</td> <tr><td>93</td><td>2.2925</td><td>Bolivia</td><td>20.00</td><td>8724156</td> <tr><td>94</td><td>2.3054</td><td>Iraq</td><td>58.50</td><td>25374691</td> <tr><td>95</td><td>2.3970</td><td>Yemen</td><td>48.00</td><td>20024867</td> <tr><td>96</td><td>2.4138</td><td>Angola</td><td>26.50</td><td>10978552</td> <tr><td>97</td><td>2.4258</td><td>Burma</td><td>103.63</td><td>42720196</td> <tr><td>98</td><td>2.6035</td><td>Malawi</td><td>31.00</td><td>11906855</td> <tr><td>99</td><td>2.6307</td><td>Croatia</td><td>11.83</td><td>4496869</td> <tr><td>100</td><td>2.6497</td><td>Guinea</td><td>24.50</td><td>9246462</td> <tr><td>101</td><td>2.7313</td><td>Cambodia</td><td>36.50</td><td>13363421</td> <tr><td>102</td><td>2.8435</td><td>Zambia</td><td>29.75</td><td>10462436</td> <tr><td>103</td><td>2.9167</td><td>Kazakhstan</td><td>44.17</td><td>15143704</td> <tr><td>104</td><td>2.9418</td><td>Uruguay</td><td>10.00</td><td>3399237</td> <tr><td>105</td><td>3.0492</td><td>Burundi</td><td>19.00</td><td>6231221</td> <tr><td>106</td><td>3.0691</td><td>Saudi Arabia</td><td>79.17</td><td>25795938</td> <tr><td>107</td><td>3.1030</td><td>Nepal</td><td>84.00</td><td>27070666</td> <tr><td>108</td><td>3.2442</td><td>Albania</td><td>11.50</td><td>3544808</td> <tr><td>109</td><td>3.4846</td><td>Cote d'Ivoire</td><td>60.38</td><td>17327724</td> <tr><td>110</td><td>3.4859</td><td>Chad</td><td>33.25</td><td>9538544</td> <tr><td>111</td><td>3.5092</td><td>Togo</td><td>19.50</td><td>5556812</td> <tr><td>112</td><td>3.7465</td><td>Syria</td><td>67.50</td><td>18016874</td> <tr><td>113</td><td>3.9578</td><td>Tajikistan</td><td>27.75</td><td>7011556</td> <tr><td>114</td><td>4.1639</td><td>Sierra Leone</td><td>24.50</td><td>5883889</td> <tr><td>115</td><td>4.6104</td><td>Moldova</td><td>20.50</td><td>4446455</td> <tr><td>116</td><td>4.6832</td><td>Rwanda</td><td>37.25</td><td>7954013</td> <tr><td>117</td><td>4.7362</td><td>North Korea</td><td>107.50</td><td>22697553</td> <tr><td>118</td><td>4.8326</td><td>Panama</td><td>14.50</td><td>3000463</td> <tr><td>119</td><td>5.1176</td><td>Namibia</td><td>10.00</td><td>1954033</td> <tr><td>120</td><td>5.2381</td><td>Somalia</td><td>43.50</td><td>8304601</td> <tr><td>121</td><td>5.2471</td><td>Belarus</td><td>54.10</td><td>10310520</td> <tr><td>122</td><td>5.3268</td><td>Zimbabwe</td><td>67.50</td><td>12671860</td> <tr><td>123</td><td>5.4316</td><td>Macedonia</td><td>11.25</td><td>2071210</td> <tr><td>124</td><td>5.5028</td><td>Haiti</td><td>42.13</td><td>7656166</td> <tr><td>125</td><td>5.8371</td><td>Congo</td><td>17.50</td><td>2998040</td> <tr><td>126</td><td>5.8587</td><td>Georgia</td><td>27.50</td><td>4693892</td> <tr><td>127</td><td>6.2829</td><td>Tunisia</td><td>62.67</td><td>9974722</td> <tr><td>128</td><td>6.3126</td><td>Azerbaijan</td><td>49.67</td><td>7868385</td> <tr><td>129</td><td>6.4545</td><td>Lebanon</td><td>24.38</td><td>3777218</td> <tr><td>130</td><td>6.9058</td><td>Mongolia</td><td>19.00</td><td>2751314</td> <tr><td>131</td><td>6.9370</td><td>Kyrgyzstan</td><td>35.25</td><td>5081429</td> <tr><td>132</td><td>6.9736</td><td>Jordan</td><td>39.13</td><td>5611202</td> <tr><td>133</td><td>7.3625</td><td>Botswana</td><td>11.50</td><td>1561973</td> <tr><td>134</td><td>7.8560</td><td>Armenia</td><td>23.50</td><td>2991360</td> <tr><td>135</td><td>8.6032</td><td>Mauritius</td><td>10.50</td><td>1220481</td> <tr><td>136</td><td>8.6841</td><td>Central African Republic</td><td>32.50</td><td>3742482</td> <tr><td>137</td><td>9.4467</td><td>Cuba</td><td>106.83</td><td>11308764</td> <tr><td>138</td><td>10.6013</td><td>Laos</td><td>64.33</td><td>6068117</td> <tr><td>139</td><td>11.5420</td><td>Libya</td><td>65.00</td><td>5631585</td> <tr><td>140</td><td>11.7972</td><td>Liberia</td><td>40.00</td><td>3390635</td> <tr><td>141</td><td>13.0917</td><td>Singapore</td><td>57.00</td><td>4353893</td> <tr><td>142</td><td>14.0285</td><td>Kuwait</td><td>31.67</td><td>2257549</td> <tr><td>143</td><td>15.8174</td><td>Lesotho</td><td>29.50</td><td>1865040</td> <tr><td>144</td><td>15.8824</td><td>Timor-Leste</td><td>13.50</td><td>850000</td> <tr><td>145</td><td>16.9264</td><td>Guinea-Bissau</td><td>23.50</td><td>1388363</td> <tr><td>146</td><td>17.0081</td><td>Mauritania</td><td>51.00</td><td>2998563</td> <tr><td>147</td><td>18.1638</td><td>Fiji</td><td>16.00</td><td>880874</td> <tr><td>148</td><td>19.0710</td><td>Gambia</td><td>29.50</td><td>1546848</td> <tr><td>149</td><td>19.9095</td><td>United Arab Emirates</td><td>50.25</td><td>2523915</td> <tr><td>150</td><td>20.5278</td><td>Turkmenistan</td><td>99.83</td><td>4863169</td> <tr><td>151</td><td>20.6737</td><td>Israel and PA (Occupied Territories)</td><td>80.67</td><td>3902062</td> <tr><td>152</td><td>20.9677</td><td>Eritrea</td><td>93.25</td><td>4447307</td> <tr><td>153</td><td>21.0694</td><td>Cape Verde</td><td>8.75</td><td>415294</td> <tr><td>154</td><td>25.5448</td><td>Bhutan</td><td>55.83</td><td>2185569</td> <tr><td>155</td><td>26.5129</td><td>Swaziland</td><td>31.00</td><td>1169241</td> <tr><td>156</td><td>27.6703</td><td>Gabon</td><td>37.50</td><td>1355246</td> <tr><td>157</td><td>28.3532</td><td>Cyprus (North)</td><td>22.00</td><td>775927</td> <tr><td>158</td><td>38.6771</td><td>Qatar</td><td>32.50</td><td>840290</td> <tr><td>159</td><td>40.6503</td><td>Comoros</td><td>26.50</td><td>651901</td> <tr><td>160</td><td>77.4467</td><td>Bahrein</td><td>52.50</td><td>677886</td> <tr><td>161</td><td>88.4235</td><td>Equatorial Guinea</td><td>46.25</td><td>523051</td> <tr><td>162</td><td>117.7982</td><td>Djibouti</td><td>55.00</td><td>466900</td> <tr><td>163</td><td>134.2928</td><td>Grenade</td><td>12.00</td><td>89357</td> <tr><td>164</td><td>203.8429</td><td>Maldives</td><td>69.17</td><td>339330</td> <tr><td>165</td><td>290.7264</td><td>Seychelles</td><td>23.50</td><td>80832</td> <tr><td>166</td><td>346.2540</td><td>Tonga</td><td>38.17</td><td>110237</td> </table> Stanislav Shalunov http://shlang.com/writing/second-foot.html <a href="../">Stanislav Shalunov</a> <p>The Second Foot Principle (TSFP): The first software system that implements some concept or specification will have to be thrown away. It's best to be prepared to throw it away as the second system will be superior anyway. <p align="center">The Second Foot Principle: When You Already Shot Yourself in the First</p> <h2>Acknowledgments</h2> The necessity of separation of the prototype from the product in software development is well known. I just propose a name for the rule. I acknowledge: <ul> 35959" rel="nofollow">Frederick P. Brooks, `The Mythical Man-Month: Essays on Software Engineering'</a> for coining `prepare to throw one away' <li>Guy Almes for coining `the second system principle' <li>Anatoly Karp for coining (in Russian) `<em>printsip vtoroi sobaki</em>' </ul> Stanislav Shalunov http://shlang.com/writing/tcp-perf.html <a href="../">Stanislav Shalunov</a> /09/06 19:18:29 shalunov Exp $</code> <p><blockquote> <strong>Summary</strong>: End-to-end TCP performance, its tuning and troubleshooting are discussed. Common problems and ways to identify them are presented. An algorithm to follow when starting to troubleshoot end-to-end performance is suggested. </blockquote> <h2>Intended Audience</h2> This document is intended for users of high-speed networks (i.e., those with workstations that have switched or dedicated connections to a wide-area network that is at least 100Mb/s) and will be of little use for home users with cable and DSL modems and of no use for people with dial-up modems. It also does not apply to people who are behind network address translators (you are behind a NAT if your IPv4 address begins with `10.' or `192.168.'), SOCKS proxies, application-level gateways, firewalls that maintain state for each TCP connection that passes through them, and QoS appliances (various kinds of shapers and droppers that go in the middle) -- the vast majority of problems experienced by people in this category would stem from the middlebox rather than the hosts or the network. It does not apply to wireless environments either -- these have problems peculiar to them. Finally, it does not apply to connections that do not go over a wide-area network (WAN), i.e., connections between hosts separated by less than 5ms (milliseconds) of round-trip time (RTT) when the network is unloaded. <p>Note that most commodity Internet users (i.e., virtually all home users and corporate users) are excluded from the applicability scope by the preceding paragraph. However, most Internet2 users (mostly people at U.S. research universities) are covered -- at least when workstations are involved (rather than computers at dormitories, which many network administrators put behind a shared half-duplex 10Mb/s Ethernet or worse behind a NAT or a QoS appliance). <p>Sometimes, we delve into details that might be helpful to an interested reader by can confuse someone who is only looking for guidance to solve a specific TCP performance problem. Those parts of the document that might be fairly technical or are only of interest to a subset of the intended readers and that are not always necessary for the understanding of the procedure to be applied are presented in a <small>smaller type</small>. The author apologizes if the document is presented to you in a form that does not distinguish between the two sizes (<small>a mitigating circumstance is that if you are reading this in Lynx or w3m, you are likely to be a geek who would want to know the gory details anyway</small>). <h2>Expectations for Excluded Audience</h2> At home or behind a corporate firewall, you get what you get and usually cannot complain about it. If you get more than 500kb/s (this is kilobits per second; it is about 63 kilobytes per second) you get more than the commodity Internet median performance, and should consider yourself lucky. [The 500kb/s figure is derived from observation of a major Tier 1 backbone network made in late 2001; contact the author if you're a network engineer or researcher and want further information on this.] If you need better performance at work (in a corporate environment), you're probably stuck with the choices made by the network administrator and can only improve your TCP performance by (i) changing employers; (ii) increasing funding for Internet connectivity; (iii) getting better network engineers and administrators (options ii and iii aren't available for most employees except top managers who cannot read anything longer than an executive summary and therefore are not reading this document and the author has yet to hear of someone resigning because the net was too slow). If you need better performance at home rather than at work and are prepared to pay a little extra for it, I know of at least one DSL provider (the one I use) that will provide SDSL lines with a service level agreement (SLA) that guarantees 80% nominal capacity available to a single TCP connection when measured to a well-connected site. <small> <p>If you're a student at a university in a dormitory, you should check what kind of outside Internet2 connectivity your university provides to the dormitory. Internet2 doesn't charge per bit, so there's no good excuse for a university not to provide a 100Mb/s full-duplex connection except for cost of switches, which is a one-time expense of only tens of dollars per capita. You might friendly ask your overworked and underpaid campus network administrator (who stays because of nice work environment) about the Internet2 connection that the dormitory has. If it's less than a switched full-duplex unlimited 100Mb/s connection, you might petition the university to get a better connection. Note that your institution pays per bit for commodity Internet traffic (essentially, those packets that don't go to other universities); don't expect them to work too hard to make it possible for you to run up their bills too much if you're paying a flat monthly networking fee -- on the other hand, if you are willing to let them pass the costs of commodity Internet transit to you, let them know: the administration probably thinks that there will be riots if they depart from a flat-fee policy. (If you think that you're entitled to eat as much commodity Internet capacity as you want because you already pay your tuition, you might also think that you are entitled to spending arbitrary amounts of printer paper and toner.) </small> <p>If you are stuck with 10Mb/s half-duplex shared Ethernet for your connection, you might get 2-3Mb/s from a single TCP connection and almost no increase from any connection banding. <p>If you are not a part of the intended audience of this document, you should stop reading now unless you want to turn green with envy. <h2>Expectations for Intended Audience (or, Definition of Poor Performance)</h2> If this document applies to you, and you're not a TCP expert, you likely will not get very good TCP performance right from the start: this is because computer operating systems and edge network technology are optimized for the commodity Internet. <strong>You should be able to get tens of megabits per second (or units of megabytes per second) in a single TCP connection to another well-connected Internet2 site.</strong> When sending files, you should typically be limited by disk speed unless using a RAID array, since typical IDE disk throughput through a file-system is 15-20Mb/s. If you do not get that performance, and get less than 10Mb/s instead, you have a TCP performance problem, which this document might help you understand, diagnose, and fix yourself or -- probably only if you're important enough -- with the help of someone from the networking support on campus. <h2>Causes of Problems</h2> <p>There are three most common causes for TCP performance problems, in the order of frequency: <ol> <li>Too small TCP window size; <li>Fast Ethernet duplex mismatch; <li>Bad patch cable. </ol> Mostly anecdotal evidence suggests that these are responsible for at least 80% of all TCP performance problems (with other problems including bad network interface cards and poor application design). <small> <p>In two years with Internet2 [as of summer 2002] the author has not encountered a single TCP performance problem that would be traced to the core network. He routinely obtains hundreds of megabits per second across the North America with a single TCP connection between specifically well-tuned and well-connected hosts. Internet2's measurement devices do not register any packet loss in the core and jitter only in microseconds. </small> <p>If you have a TCP performance problem, a problem in the core probably has about the same probability as little green men twisting the wires in your patch cable -- you are likely better off spending your time investigating more likely causes, starting with the three listed above. <p>To troubleshoot the problem, follow these three steps (briefly; background for each step is given below, followed by a detailed troubleshooting algorithm): <ol> <li>For a 100Mb/s connection, make sure RFC1323 extensions are enabled and TCP window size is at least 1 megabyte (and perhaps 2 megabytes if the machine in question has plenty of memory and a very good network interface card) on both sides; default TCP send and receive buffers should be equal and set to the desired value; <li>Make sure there is no duplex mismatch at either host's side; <li>Replace patch cables for both computers with different ones (preferably new, not ones that somebody else has thrown away and you found in the networking closet). </ol> <h3>Too Small TCP Window Size</h3> TCP window size is usually equal to the TCP buffer space and should usually be the same for both sides of a connection (otherwise, the minimum of the two will often be in effect). <small> <p>In traditional TCP, the window size cannot be larger than 65,535 bytes (because the unsigned integer field that holds it is only 16 bits wide). You will probably need a bigger window than 64KB. This is achieved by turning on TCP extensions specified in <a rel="nofollow" href="http://www.rfc-editor.org/rfc/rfc1323.txt">RFC1323, `TCP Extensions for High Performance', published in 1992</a> and now supported by most operating systems. The TCP window scaling option allows one to use TCP window sizes of up to 1,073,741,823 bytes, which should be good until speeds that approach 1Tb/s (with RTT of 100ms). Turn on this option. (It might already be on. No operating-system-specific instructions are provided in this document, but there are some links at the end that might help. If you have or know of more publicly available documents describing operating-systems-specific steps to set TCP options and window sizes, propose them to be included.) Once you figure out how to do this, you should check if your TCP stack supports selective acknowledgment (SACK) or explicit congestion notification (ECN) options; if it does, turn these on, too (SACK and ECN and not strictly speaking necessary to get good performance but they might help in certain environments). </small> <p>The sure way to learn the TCP window sizes in question is to perform a packet capture of your connection (e.g., with tcpdump, snoop, Etherpeek, or equivalent tool for your OS) and examine it. <small> <p>You will need a few packets from the beginning of the connection, including the SYN packets. (If you don't know what is a SYN packet or cannot do packet capture yourself, find someone local who has some networking expertise -- such as your network support technician. Otherwise, prepare to spend some time figuring this stuff out, but you must be the type that likes to fix car engines by yourself.) For each direction, the window scaling option will be in the SYN packet (if it is not, it is implicitly 0). For each direction, take the actual window size number from the first data or pure ACK packet and multiply it by 2^{window scaling}; this is the actual window size at the start of the connection. (For example, if window scale in the SYN packet is 3 and the window size in the first data or pure ACK packet is specified as 32768, the actual window size for that direction is 32768*2^3 = 262144; this figure is in bytes.) Make note of the window size numbers in both directions. To make matters more confusing, some packet sniffers will already have done the multiplication for you; it happened if and only if the window size advertised in the first data or pure ACK packet is already presented as being larger than 65535; if this is the case, you do not need to multiply the number by the exponentiated window scale (it is already usable as is). Having established the window sizes, make note of the minimum of the two numbers for each direction and use it in subsequent paragraphs. </small> <p>Measure the round-trip time. The simplest way is to use the <code>ping</code> utility. <small> <p>Make it send 60 packets slowly (e.g., 1 packet/second) while the network does not carry your traffic and take the minimum and the average of the 60 numbers (these are normally printed at the bottom along with some other numbers). On a properly functioning high-speed network, the numbers should both resemble within a factor of two your propagation delay (to be estimated in seconds as distance between hosts in kilometers divided by 200,000km/s, which is approximately equal to the speed of light in fiber; the delay should be fudged 50% up for network path not being a part of a great circle of the planet). Take the minimum as your RTT, to be used subsequently. </small> <p>The theoretical maximum TCP throughput that you can obtain with your tuning parameters is window/RTT. Rule of thumb: For a realistic number, divide the theoretic hard limit by two. <p>With 70ms RTT (typical cross-country delay), you might see the following throughputs for different window sizes (assumed equal at both sides): <p> <table border align="center"> <caption>TCP throughput <it>vs.</it> window size for RTT=70ms</caption> <thead valign="top"> <tr> <th>Window Size <th>Theoretical max throughput <th>Realistic throughput <tbody> <tr align="right"><td> 8KB <td> 0.9Mb/s <td> 0.8Mb/s</tr> <tr align="right"><td> 16KB <td> 1.9Mb/s <td> 1.8Mb/s</tr> <tr align="right"><td> 32KB <td> 3.7Mb/s <td> 2-3.5Mb/s</tr> <tr align="right"><td> 64KB <td> 7.5Mb/s <td> 3-7Mb/s</tr> <tr align="right"><td>128KB <td> 15.0Mb/s <td> 6-14Mb/s</tr> <tr align="right"><td>256KB <td> 30.0Mb/s <td> 10-25Mb/s</tr> <tr align="right"><td>512KB <td> 59.9Mb/s <td> 20-40Mb/s</tr> <tr align="right"><td> 1MB <td>119.8Mb/s <td> 30-60Mb/s</tr> <tr align="right"><td> 2MB <td>239.7Mb/s <td>60-100Mb/s</tr> </table> <small> <p>(I.e., for what is currently considered large values of window size, perhaps a factor of two of safety margin is required; for small window sizes, the safety margin is largely determined simply by the remainder of the division of allocated buffer space by TCP Maximum Segment Size (MSS); MSS is simply the maximum number of payload bytes that can be sent inside a single TCP packet on a given link.) <p>The theoretical maximum is a hard limit of sustained performance with arbitrary TCP options, only a small number of non-consecutive drops per RTT, and a perfect network. The realistic values are what the author would personally expect in a typical case, based on experience, from hosts, TCP implementation, and networks from the real world (in particular, it assumes no SACK, no ECN, standard aggressiveness, 1500-byte MTU, and no RED at the bottleneck). </small> <p>For example, if a 64KB window size is used on both sides and one sees 3Mb/s actual throughput with 70ms RTT, this is nothing out of the ordinary: it's only slightly less than twice the hard limit one would see in a perfect world. <p>To make the long story short: <strong>On a high-speed wide-area network, hosts should have RFC1323 extensions turned on and default TCP send and receive buffers that are equal to each other and are at least 1 megabyte</strong>. <h3>Fast Ethernet Duplex Mismatch</h3> <blockquote> <strong>NB (added in 2005):</strong> This description was a work-in-progress and is preserved on this web page for historical record. If you want to learn about duplex mismatch (rather than how I understood it in 2002), you should read the following paper: <p><a href="http://www.pam2005.org/PDF/34310138.pdf">Stanislav Shalunov, Richard Carlson, ``Detecting Duplex Mismatch on Ethernet,'' PAM 2005, Boston, Springer, LNCS 3431, pp.135--148.</a> </blockquote> <p>Fast Ethernet (the 100Mb/s version of Ethernet) can operate in two modes: half-duplex and full-duplex. In full-duplex mode, both sides can send simultaneously. In half-duplex mode, only one network interface card at a time can send. <small> <p>Some older (or cheaper) network interface cards (NICs) support only half-duplex mode; most modern cards from a brand-name manufacturer support both half-duplex and full-duplex modes. Each NIC can either have duplex mode hard-coded by the host computer's operating system, or have no preset mode. If a NIC is configured to use a certain mode, it will use it without regard for the other side of the connection. If a NIC does not have a hard-set mode, it will attempt to engage in auto-duplex negotiations with the other side. The process attempts to establish a common duplex of communication, with preference to full duplex if both sides support it; on a connection that is not fully switched, full-duplex communication makes no sense, so the auto-negotiation process attempts to discover if more than one other NIC is present and uses half-duplex mode if it senses more than one other NIC. This process works fairly well for modern cards, but for old cards it used to have success rate of 52% (anecdotally), in other words, the results were essentially random. </small> <p>If the two sides of a switched Fast Ethernet connection disagree on the duplex mode they use (i.e., one is using full-duplex mode and the other is using half-duplex mode), a <em>duplex mismatch</em> is said to occur. <small> <p>A Fast Ethernet duplex mismatch can happen in one of the following ways (not meant to be an exhaustive list of causes): <ol> <li>One card is hard-coded to use full-duplex (often the switch side but it migh