Text transcript of show #292. November 10, History of HTTP and the World Wide Web with Henrik Frystyk Nielsen

Transcription

1 Hanselminutes is a weekly audio talk show with noted web developer and technologist Scott Hanselman and hosted by Carl Franklin. Scott discusses utilities and tools, gives practical how-to advice, and discusses ASP.NET or Windows issues and workarounds. Text transcript of show #292 History of HTTP and the World Wide Web with Henrik Frystyk Nielsen One day Henrik Frystyk Nielsen met Tim Berners-Lee at CERN and became his first graduate student. He joined the W3C and worked on HTTP and some of the first browsers. Henrik is one of the primary authors of the HTTP specification. He sits down with Scott and they chat about the history of the Web from HTTP to the mysterious HTTP Status Code 418. (Transcription services provided by PWOP Productions) Our Sponsors Copyright PWOP Productions Inc. Page 1 of 7

2 Lawrence Ryan: From hanselminutes.com, it's Hanselminutes, a weekly discussion with web developer and technologist, Scott Hanselman. This is Lawrence Ryan, announcing show #292, recorded live Thursday,. Support for Hanselminutes is provided by Telerik RadControls, the most comprehensive suite of components for Windows Forms and ASP.NET web applications, online at In this episode, Scott talks with Henrik Nielsen about the history of HTTP and the World-Wide Web. Scott Hanselman: Hi, this is Scott Hanselman and this is another episode of Hanselminutes. Today we're sitting down with a pretty interesting person, Henrik Nielsen. He is a Danish engineer and computer scientist, and he did some really interesting work early on in the World-Wide Web. How are you, sir? I'm pretty good. Scott Hanselman: You and I are within five years of each other's age, but I think while I was fooling around in high school, you were getting your master s degree in engineering in Denmark and then you met an interesting person when you went to work at CERN. Can you tell us a little bit about that? Yes. A long, long time ago when I was student I figured out that it is very convenient to travel out and do things as a student because you can sort of just drop in a place and then be there for a little while and leave again and have some great experiences. So I looked around actually to where to do my master s while I was at my Danish university, and I talked to somebody who talked to somebody who knew a guy who was sitting at CERN and said, "Well, this guy is doing some interesting stuff. We don't know exactly what it is. He's calling it something, something, something web, but it is very interesting so you should go talk to him." So I went down, and CERN is on the Swiss-French border just outside of Geneva. So I went down there and visited and I met Tim Berners-Lee. That was back in '93. He had formerly proposed the Web as a concept in 1989 I believe and was working with -- there were two or three students sort of associated roughly with his group plus a couple of CERN people plus Tim, but that was about it. There were about five people all in all. So he was very enthusiastic. He said, "Yes, great. Come on in and sit down." That's how I effectively started working on the Web. So I had to go back and pack my stuff and do other kinds of stuff. But I came down in basically January 94 and moved to CERN as what's called a technical student and then started working on Web technology as it was being developed. And very specifically I was working on some library, a code library called libwww. That has the old sort of UNIX naming style that had the first HTML browser, it has some HTTP implementation. Most of it was actually done by Tim himself plus another guy that helped doing some of the stuff, and I helped and continued developing this together with another student from Finland. That's how the CERN server came to be. There was something called a line mode browser and other kinds of things. And then after a little while, I think it was in March where the first, March of '94, where the first Web conference ever was held. It was held at CERN and there were about 300 people and that was the entire community who knew about the Web in the world effectively. Marc Andreessen was there. A few other people from around the world who had been looking, there were some other browsers at that time. There were people who had been putting up websites; SLAC, a physics lab had also put up a website. It was very exciting to sort of be of the part that gang of 300 or so people, basically with everybody who had been either putting on a web server or starting writing browsers or working on servers and those kinds of stuff. That's how it pretty much started. Scott Hanselman: So at that time, you said there were about 300 people in the room and that's basically everyone on the planet that knows everything about the web. Are there less than 300 websites at this point? Oh, yes. At that time, Tim basically had a little web page where he wrote -- there was the first index of web pages and it was there on a single page. So there were not a lot. Scott Hanselman: So it was just a page that was like a list of all the other pages on the internet. Yes, yes. And that website was called info.cern.ch which was the place where the initial specs, where this page of websites and a bunch of other information was posted for a long time. Scott Hanselman: When you first heard about this idea of the Web, I assume you were using Telnet. I mean I remember at that time I was probably using something like CompuServe and I was dialing in to universities and telneting around. Even before that I was using Bulletin Board systems. Was there a sense or were you already drawn to the idea of an interconnected group of computers? I didn't quite realize the power of the interconnectedness on a global scale because it's hard to get a grasp until you sort of see it. I was immediately taken by the idea of this information space that everybody can participate to produce information, and to consume information, and to exchange information, and to have a common place for talking about things. I think that was a very exciting idea. I think one of the times where it kind of clicked sort of personally was really when you started Page 2 of 7

3 with the website but also just with using and things, that you could send s. For example, one of the times (and now it sounds, of course, naïve) is that there was a UNIX guru called Stevens who wrote some very good UNIX books for networking and for the TCP stack and what-not. I was reading that because I was programming some of the stuff in this library and there was a piece of DNS that I couldn't understand what it was doing. So he had his in one of his books and so I sent him an , which was kind of like I might as well send off a probe to Mars because I had no idea sort of what the person was and where he was. He answered and that was kind of saying, oh, my goodness, this is really powerful. This being able to communicate, obviously this was an , the Web sort of took that to another level. It was something that I think many people sort of have to, they have an internal realization saying, oh my, this is really powerful. And then you get on board. Then there's just no going back. Scott Hanselman: So that's one of those moments where you realized that this is going to be a thing. Yes, absolutely. Scott Hanselman: So you worked with Tim Berners-Lee and the W3C, the World Wide Web Consortium. Yes. Scott Hanselman: When I think of the World Wide Web Consortium, I think of it rarely. I think of it when I have to go and dig deep into a specification to find out whether or not something works the way that it's supposed to work, and I often find myself poking around inside of the HTTP specification and saying was this intended to do this. You're one of the primary authors of that. How does that work? You just sit in a room and think it up or is it a series of meetings? When you're off inventing something like HTTP and writing it down to the byte detail, where you are you and what are you doing when this is happening? Well, the way I'd say it is that it's an understanding very early on that this is a collective effort by a lot of people and that is not very much somebody sitting and dreaming up the idea. Obviously Tim dreamt up the initial idea dan how the protocol can work, but there's a lot of details for whether you should be using, what basics should you use, what language should you use to describe, what is the format, what is the syntax, what are the semantics around it, and there are a lot of choices and effectively they are always engineering compromises. It takes early on sort of -style headers, it s called RCA-22; headers would have some name: value as being the language of HTTP. That was introduced in HTTP 1.0. Before that, it actually was just one line which said GET, and then the URI that you want to get. There was no metadata around it. So all those kinds of things I'd say evolved. It was not somebody sitting locked in a room just thinking about it. It was writing code, interacting with people seeing what the scenarios were, very much being part of a vibrant community where people wanted to do, and then, for example, HTML forms came about, so post was dealt with. I remember one detail, for example doing PUT. You can obviously do a PUT content up to a Web server. In 1.0, what we had was that you just sent the document right with a PUT upfront. And I was doing some experiments where I was sitting. At that time I was sitting, I moved to MIT with the Web Consortium but we were still playing around with a server back at CERN. So I did PUT cross Atlantic, and we realized that sometimes if you do an authentication and you got something back that said you're not allowed to post a document over here, you could actually lose the response with the way that HTTP was formulated so we had to add something called a second-level -- this was when these 101 temporary or 100 continue messages was introduced to HTTP in order to get around those kinds of things. So many of those things were really, obviously there was some fundamental -- the way I'd say it is that you come up with some basic principles that you would like to adhere to. Some of them were document-oriented that is hyper-text driven, that is based on URIs, and that is stateless. That means that you don't have sessions, you don't have anything like that. Once you sort of picked your set of principles upfront, then the rest of it is about filling in the gaps and figuring out how does this work, how does this work, and how does that work, and build code. One of the things that I think has always been at heart of both IETF and also W3C, W3C is sort of inherited from IETF, is... Scott Hanselman: IET,F of course, being the Internet Engineering Task Force, for folks that may not be familiar with that acronym. Yes, exactly. Exactly. That's where the typical sort of RFC acronym comes from. Those are documents that are published by that organization. They also have published TCP and IP and DNS, and effectively they are to the Internet what the World Wide Consortium has been to the Web in terms of driving the core specifications. The key piece has always been what is called rough consensus and running code. That means that you get into a room, you don't agree on everything but you have rough consensus for saying this is roughly the right thing, and then you basically have two or more implementations that work together. So you have to demonstrate that things work. That is the great way for getting beyond my thing is prettier than your thing kind of discussions where you can discuss that until Page 3 of 7

4 the cows come home. But at the end of the day, really the focus has always been, yes, that's great but we need to have running code, we need to demonstrate that it runs not just in your lab or between two machines but can run globally. That's where you can't just sit down and think that through. There are too many variables and too many unknowns and too many quirks that you have to work with. It really takes the community to come out with a spec, and I think HTTP was very much developed as a community. There were the basic principles that they started out, with what Tim had in terms of the basic things that we've just talked about. But on top of that, it's really about the community pitching in saying this is great but why don't we add this, why don't we add that, and then it grows from there. So effectively the most productive and most successful specification work that I've been part of has always been where the working group has been working in tandem with developed implementations. So it's not that the working group goes out and dreams about something up and then the implementations come later. It's really the implementations and the specification go hand in hand. Scott Hanselman: That's really a fascinating idea. It should be a guiding concept for everything that we design. I think there are a lot of times when people spend hours and hours in front of a whiteboard brainstorming things and then being very congratulatory once the white-boarding is done. But you're right. Once you have an implementation, you're only half done. But two complete working implementations presumably written by different people talking to each other is a really great way to confirm that things are working as you specified, and as you imagined them, and certainly no better way than to find the bugs in the specification and to actually try to implement it. Yes, yes. And in fact, the HTTP specification work has actually just recently undergone, right now as we speak, undergoing a big editorial revision. It's called the HTTPbis work which is updating the HTTP spec with a lot of clarifications about how things work by deployed implementations already. There might be places where the spec is unclear so there are a lot of things that are being cleared up. But in a sense the critical aspect of it is that the spec becomes more of a historic document because it kind of captures what is working, what is out there, what is available today. That, I think, is the critical aspect of what a spec should be. It documents what is working. That is what it should be doing. If it defines and says every time you do something you have to go to that document to figure out what the idea was, then you don't have this interaction between deployed implementations and the specification. Scott Hanselman: This episode of Hanselminutes is brought to you by Careers 2.0. Careers 2.0 is a new service by our friends at Stack Overflow. You probably are all familiar with Stack Overflow, the online Q&A resource dedicated specifically to programmers and programming related topics. Well, the team at Stack Overflow created Careers 2.0 to provide you with access to great jobs and introduce you to a bunch of great companies that you might consider working for even if you're not currently looking for a job. Think of Careers 2.0 as a programmer profile. It gives you a platform to show that you're awesome by featuring your proudest contributions to Stack Overflow, GitHub, SourceForge, Bitbucket, anything programming related. You can even add your favorite programming books from amazon.com. Profiles on Careers 2.0 are free. They're easy to get started especially by importing your LinkedIn profile. However, there's one catch. Profiles on Career 2.0 are invite only. They did this to keep out the spam and have a high-quality environment. Fortunately for you, as a Hanselminutes listener I've got your back. Head on over to to accept your invitation today. Once again that's I hope you like it. Now you probably know that specification backward and forward because even though, while it was a huge community effort, you re kind of the note taker and the facilitator and you're actually typing-up, you know, you're one of the principal authors for typing it up. How many times have you discovered that you thought you had it completely solid and you thought the spec was clear and there's a section of the spec that is completely and totally unclear and then is being fixed 10, 12, 15 years later? I'd say that actually is by far the most common. It so happens that what somebody feels is clear turns out always to be unclear to somebody else. In fact, one of the key pieces -- and this is one thing that is rarely appreciated as part of the working group -- is that as part of going through a working group (and the HTTP working group was big, at some point I think it has probably 100, 200 people, not all participating but sort of reading and sometimes pitching in and doing sort of having good suggestions and other kinds of things), one of the key aspects is that it's a collective education of that group of people for getting shared context for what this document actually is supposed to mean because you can only write it up to a certain point in English. There will always be questions about, well, how does this work if you do this and the face of the moon it's exactly that and those kinds of things. What happens is that the group of people of a hundred or so through this process have solved part of the specification and then they go out and implement it also, and it kind of grows from there. We try, of course, to capture a lot of that in the document but there's also a tremendous set of knowledge in the aliases and just in how people get a better understanding on what's going on. Page 4 of 7

5 Scott Hanselman: This might be an ignorant question on my part. Forgive me. But do you ever feel like you're surprised that it works? Well, the way I d say it is that I think some of the design...in a sense I was very lucky because I did not have any Internet experience. I did not have any distributed computing experience when I started working on the Web so I really didn't come in with a pre-conceived notion of saying, well, but this is not a distributed object system, but this is not a distributed file system, or this is not transactional. I didn't have any of those sort of preconceived notions about what would work or not, and I don't think there was ever any question that it would work. It was clear that there were a lot of discussions about whether the protocol was clunky and whether it was wasteful, and I had lots of discussions with other people who had developed other solutions that were very technically, you can say more elegant in certain ways. They were separated better out. For example, one of the things we never did in HTTP is you cannot have sent responses out of order. So when you send a request on a single TCP connection, the response has to come back before you can send another request. We never sort of said, well, I can send two requests and then I can get the last request, the response first, and so it gets them out of order which can have some significant performance optimizations. We never did that because that would break existing HTTP implementations. But they were all the solutions that went down this route and went that way. You could say from a network optimization, from a usability point and sort of from a network usage point of view, they were better in certain ways. What I learned though is that that's not really what matters most. What matters most is that you provide value to what people want to do. This idea that you could just start a Web server, point it at a file share and now serve those documents, whether those were HTML documents or PDF documents or whatever else, or images, you could just serve them, was a tremendous value to the community and to people who wanted to expose information and to create information. So this idea that you could build things that were more elegant, more efficient, all those kinds of things, is true. But what I learned is that never try and sit in front of to have this efficiency argument if the solution is just to put in another server. If you sit in front of the brute forces or if I just put in another server, I'm fine. It's a very hard argument to say "Well, but if you change your infrastructure completely, you don't have to do that." That is a very hard argument often to do because the value of the network goes up exponentially with the number of nodes participating in it. As soon as one combination takes off, the value of the network becomes huge, and so changing out from that way of communicating is almost insurmountable. So in a sense you can say, well, it is fine because I was at many conferences where I heard, especially from TCP transport people who were objecting to how especially HTTP 1.0 used TCP, which was kind of wasteful in many ways, and they said that, I had many arguments saying, well, it is a stupid protocol because it doesn't use TCP in the right way and it doesn't do this and it has big request headers of many, many hundreds of bytes per request and response is also wasteful and those kinds of things. While that's all true, the alternatives were not able to break the momentum of HTTP simply because the value was too high. That, I think, is a very useful lesson. Engineering is a practical way of building stuff, and sometimes you have the luxury of doing things multiple times, but often when you come up with something it really is a balance between the level of entry. How complicated it is to get up and running your system is a huge factor in the system taking off. If it has the right properties, then performance is something that could come down the line. Scott Hanselman: That is a powerful concept, and the idea that people want to be agile, and people talk about agile methods, but sometimes doing a big spec upfront and really thinking deeply about it and doing small prototypes is a very valuable way to get something done. It may come out looking like a big design upfront, but it is actually a number of small iterative things. Like you said, you can do performance a little bit later but I wouldn't... Would you say that these kinds of specifications in the Internet itself were done in an agile way? Because if you look at something like HTTP, it hasn't really changed in a while. I mean things sit on top of it but the essence is still there 14 years later. Yes, the essence is absolutely there still and also all the quirks are there. For example, even today there are five or six ways that you can in fact, the limit an HTTP method how long it is,which from a sort of a pure protocol point of view seems excessive. Normally you will say, well, can I just have a byte count that I need to read into one place and then I can understand when one message ends and the other one starts. But HTTP doesn t have it. It has five ways and the reason for that is because, well, it started out not having any way whatsoever. And then we had to add ways to it and there were different ways of adding it. Some were successful and some were sort of quasi-successful, but they also stuck and so now we have this aggregation of various features, and you could come up with something from a clean slate that is much more efficient in many ways from that perspective but it's going to be an uphill battle to convince everybody to change. Scott Hanselman: So you went to work for Microsoft in '99 after you left the W3C. Yes. Page 5 of 7

6 Scott Hanselman: on lately? What have you been working Well, the Web stayed with me, or I stayed with the Web, throughout my career in one way or the other because I'm fascinated by the potential, and the overall idea and the ability for people to communicate and exchange information I think is extraordinary. What I'm doing at different times has been kind of expanding where you can use the Web technologies. One aspect that I was working on a couple of years ago was building robotic frameworks where you can say, well, what has robots to do with Web technology? But it turns out actually you can use a lot of Web technology or a lot of ideas behind how document-oriented, using URIs, being able to inspect by putting a browser and looking at what's going on. It matches very, very well with something like robotics. So I've been trying to push Web technology into new areas where traditionally it hasn't been part of it. Robots are great fun. They are just near and dear to my heart because the fact that they move around makes you laugh. It looks so much less interesting. Things that move around are just interesting, more interesting. And they told me a bunch of interesting things about other aspects that Microsoft has been looking into a lot like enterprise computing with reliable messaging and transactions and those kinds of things. Some of the things that we learn is that, well, if a robot rolls off a cliff, no amount of transactions or reliable messaging or any of those kinds of things that typically have been deployed in enterprises work because the robot is gone and you have to figure out what are you going to do now. That actually works great with Web technology because the Web technologists always say, well, we're going to do the best effort. But we're not just going to stop working if a server is down. We're going to try and do something else. What I'm doing now is trying to take the same idea for building Web platforms and merging it into Cloud scenarios where we can provide a much easier way for you to build Web APIs in particular. One of the things we've been focusing on is that the initial sets of Web servers were either file servers or they were HTML. They were wrapping HTML around backend processes. They could either use CGI which was like this a way of communicating with the backend process. But the backend process generated some data and now the Web server's responsibility was to put HTML's angle brackets around that data and then present it so that it could be consumed by a browser. HTML5 and JavaScript have changed that in the sense that what happens now is that a lot of the rendering can happen on the browser side so the server doesn't have to put HTML around the data. It just exposes the data as what we call Web APIs where you can now program directly against these APIs and then build all the rendering as part of your HTML5 application. So that means that there's a big push towards building these kind of new applications because they can be consumed both by browsers, by mobile devices and across the board, across a set of devices that might render things differently but now they can do the rendering locally rather than having the data coming at any, and the presentation coming straight from the server being tailored to that device. So what I'm working on now is to provide a new framework which hopefully will be available in the nottoo-distant future where we can show some of the ways that you can build very lightweight, very efficient Web APIs using the full force of HTTP where you have sort of great access to headers and to methods and you can set caching semantics, you can set whatever you want to set all around, and has just an overall great experience for you to build these APIs and deploy them into the Cloud. Scott Hanselman: One last question. I always wanted to ask someone who works so closely on the Internet is if you could explain to myself and to our listeners what HTTP Status Code 418 is. I don't even remember what that one is. [laughter] Scott Hanselman: You might be familiar with it as related to RFC 2324, the Hyper Text Coffeepot Control Protocol. Oh yes, yes indeed. Yes. So Larry Messenger who was the working group chair, followed up on a proud tradition in IETF for writing what is called April Fool's specifications. I think it goes all the way back to the 1970s, 1975 or 1976, when the first of those were published. They had, over the years, they actually had great sense of humor. One of the ones I remember is to send electricity over IP, which of course is entirely an impossible thing to do. But it was all about sort of doing things that at the front actually look plausible, but did something that ultimately didn't make sense. Messenger had this great idea of writing the Coffeepot Control Protocol as an HTTP protocol, and he published it as an April Fool's joke and I think it was actually great. I believe it is, at least in RFC 2616, it is one of the formal references. If you look very carefully at the back of the document, it is actually referenced as a formal document supporting the HTTP 1.1 specification. Scott Hanselman: And the 418 Status Code is in the spec right now; I should search for it. Oh, it's in the spec right now. Great. Perfect. I love it. Scott Hanselman: Yeah. People can check those out at April Fool's Day RFC. Page 6 of 7

7 Yeah. And the thing was it actually was so that you could in fact build it as a protocol. It was not just a thought experiment. You could actually do it, and I believe somebody has done it. I seem to remember somebody posting that they actually implemented it. It had a little webcam so you could show the state of the coffeepot. Scott Hanselman: Yeah, it's much more fun. If you're going to do something like this, you need to actually implement it. Someone in 2009 proposed doing IPv6 over a social network. So someone ended up implementing it. There's no reason not to. So they ended up implementing IPv6 over Facebook itself. Yes, yes. All those things are absolutely...the thing is on one hand they are great fun because they poke, sort of poke fun at the layering and using modules for thinking about the protocol stack because, of course, you can always take something that sits at the bottom, say IP, and then put it at the top, at the very top of the stack. But it actually is a testament, if you think about it, to how the layering kind of works because it works because you can take something that sits at the bottom and it's independent of what is both below and above, and you can then put in another, you can reshuffle the stack and put it in another spot in the stack and still make it work. Of course, from a practical point of view and performance point of view, it doesn't work. But it's a good lesson in that it kind of tells indirectly that the layering works. Scott Hanselman: It does, and that really is the testament to the web itself. It works and we appreciate your role in helping make that. Henrik Nielsen, thanks so much for chatting with us today. Thank you so much. It was great fun. Scott Hanselman: This has been another episode of Hanselminutes, and we'll see you again next week. Page 7 of 7