For the past 4 years or so, in my spare time, I have been working with a small start-up company, Encitra, whose goal is to help cities and real estate developers make sustainable urban plans come to life in the minds and hearts of stakeholders and the general public. We go at it with virtual reality. Not just computer animation movies; we develop complete multi-user interactive virtual environments that are built and re-built over time by multiple people, and that simulate urban areas — both structural and dynamic aspects — as faithfully as possible. Recently, we accomplished an important milestone: we were able to simulate an area of 3km x 1.5km of the city of Uppsala, Sweden. This includes the actual terrain, the major landmarks of the city, several hundred assorted buildings, as well as traffic and pedestrians. It’s all live and accessible on the Internet, although not on the Web browser. This post explains the technology behind it. For the most part, it’s all based on open source software!

The Server Side

Being a developer of OpenSimulator, it’s no surprise that the core of the infrastructure is OpenSimulator. I seriously considered other alternatives, though, but I leave the technology decision process for another post.

 The area is divided into a 2D grid of 256-meter “regions”, meaning that we have 6×12=72 regions total. In turn, the regions are grouped in “sectors” of 3×3 regions, each sector running in a separate simulator. So we have 8 simulators for this area, like the picture on the left (not to scale, it’s just an illustration).

We use OpenSimulator in a standard grid configuration, with one central “Robust” server, and with the simulators all sharing the grid resources. During different phases of this project, we placed the simulators in a variety of servers, from our own machines to AWS EC2. We have developed a simple systems administration layer on top of OpenSim that allows us to very quickly deploy simulators in whatever servers we need them to be on.

We have configured the simulators using advanced options that are available in OpenSim, but that aren’t well known. For example, we open child agents in a span of 6 regions all around, instead of the default 1. This allows us to view almost the entire area independent of where our avatars are.

One characteristic of the Encitra virtual reality environment vs. Second Life and the OpenSim grids out there is that our environment has very few visitors. During the most active phases of the build, there are 3-4 people logged in, and when we have meetings, there are 7-10 of us. We don’t expect ever having to cope with large numbers of users in the same simulators, since this is not a social environment, it’s just a collaborative one. We do, however, produce relatively large builds in terms of numbers of prims, meshes and textures.

The Viewers

We use primarily 2 viewers: Imprudence and the Zen viewer. Imprudence is a great utility viewer, as it allows us to create very large structures, as well as importing/exporting objects. More recently, we have all come to use and love the Zen viewer for doing routine builds and for capturing video and pictures. Zen is based on the Linden Lab V3 code base, meaning that it supports all the latest eye candy like media-on-a-prim, shadows, etc. It also comes with presets for creating stunning environments regarding the sky, the water and the light. We have been very pleased with the videos that Zen allows us to produce!

Terrain

With an external toolset that I developed, we are able to generate terrains from GIS data, embedded with aerial images. (One of these tools is based on old code from Adam Frisby.) These realistic aerial-image terrains, by themselves, provide a fair amount of immersion, even before any building is modeled. They also give us the footprint for placing the buildings in the scene and for visualizing the roads and vegetation.

These terrains allow us to work under them. Even though this was not part of the requirements for the Uppsala simulation project,  I simply had to place something underground… after all, cities are not just what’s visible, they are also what’s invisible.

 The Build Process

The build process is exactly the same as any build in Second Life / OpenSim. We have been using primarily prim buildings, but we also have sculpties and meshes. We have taken a fair amount of pictures of the building façades, and used them to texture the buildings. Here are pictures of some of the Uppsala landmarks:

For the urban plans that are being considered but that don’t yet exist, like the podcar system, for example, we do something special with them so that we can add them to or remove them from the scene with the click of a button.

Traffic Simulation

We developed a traffic simulation addon that is capable of driving thousands of vehicles all over the simulated area without this negatively impacting the visiting user’s experience. The traffic simulation runs on a separate server. The vehicles go from one end to the other without regards for “region” or simulator borders.

The current traffic simulation/visualization has been developed by me from scratch, but the intention is to hook up this technology to external traffic simulators, of which there are a few out there. That will be a future milestone. But since I developed this one, I developed an appreciation for traffic simulators. They’re fun pieces of software! My traffic simulation includes working traffic lights and stops to which the vehicles react, as well as collision detection, all wrapped up in very simple, boid-like rules.

Currently, the painful part is the establishment of routes. We have no way to infer the routes other than by visual inspection and by insider’s knowledge of the traffic flow and signs in Uppsala. The picture on the left shows the way points for cars and buses around the station area. With areas like this one, 3km wide, setting up the paths that the vehicles use, and the speed limits, is a daunting task, almost as daunting as doing it in real life! Perhaps when we hook this up to real traffic simulators that information will already be available.

On the positive side, the podcar system, given that it doesn’t yet exist, has been much more amenable to automation. We have automatically generated both the tracks and the routes from an existing plan.

Pedestrians

Since 0.7.3, OpenSim supports server-side bots that are scriptable inworld. We have used that facility to create over 60 standing/sitting bots and a dozen of walking ones. The standing/sitting bots are very lightweight; we could easily have hundreds of them in every simulator. The walking bots, on the other hand, are relatively heavy, as they are part of the physics scene.

Additionally to contributing to lag, and similarly to traffic, one of the major hassles associated with walking bots is establishing the routes that they walk. Our bot developer has scripted a very pretty system that allows us to visualize the routes of the bots using particle systems. This can’t exist in real life, but it’s so pretty that I wish it could!

Can Cities be Simulated, Really?

You may very well be asking that question. Clearly, this simulation ignores an enormous amount of things that happen in the real city of Uppsala. In fact, it focuses only on a very small number of aspects — the buildings, the roads, the traffic, the pedestrians around the stations and, most importantly in this case, the podcar system connecting the main station to the Hospital and the University. As with any computational model, big chunks of reality are discarded. That’s how these models work.

In this particular case, the main question at hand is the viability of the podcar system — its concept, its look & feel, its utility with respect to the alternative (buses), its layout throughout the city, and the positive and negative interference with existing infrastructure. Since there aren’t many podcar systems in the world, people are’t used to this urban transportation concept. The decision to have one is part of a long process of technical, political and strategic deliberation involving many stakeholders. This simulation and visualization is part of a larger set of artifacts that are being produced. It will be interesting to see what the outcome of those deliberations will be…

My answer is: the vast amount of accessible documentation and knowledge out there. For example, I can’t program in an airplane without Internet; I roll for 10 minutes until I get stuck on something for which I don’t know the answer, or I know that there’s a much better way of doing it, I just don’t remember the API. This is followed by a feeling of frustration that makes me quit the IDE/emacs/vi and write a blog post or start working on the PowerPoint slides for a talk. Forget about types, objects, and whatnot. On-demand documentation, examples and question answering `a-la Stackoverflow is the single most important thing for me. I bet for others too.

This is quite a dramatic change from how things were back in the 20th century. The Web and search engines have raised the bar really high, a bar that we didn’t even know existed: the acquisition of “brick knowledge.” Brick knowledge is knowledge about specificities of how to do things. For example, if I want to add a feature to my program that uses gzip compression/decompression of certain data, the non-brick knowledge is knowledge about compression in general, about gzip in particular, and what it does to the performance of my program; while the brick knowledge is knowledge about the specific APIs or example code that implements compression in the various programming systems (say, .NET or Python or Haskell). Once I know what I want to do, and why, I just pull up a search engine and type “.net gzip” or “python gzip” or “haskell gzip” and the brick knowledge magically appears in seconds. Without the Web, I’m stuck.

In the old days, people coped with this by specializing on specific programming systems. This would allow them to acquire vast amounts of brick knowledge, so they had it when they needed it. The Web has changed the game. These days, I feel I can program in just about any language/system out there… I CAN HAZ POWER…  as long as there’s searchable brick knowledge about it.

The more the better. So the best programming systems, for me, are those for which all my information needs are satisfied with a search using reasonable keywords.

One of these days, I’d like to prototype a programming system that lets me write programs as a sequence of search queries

PLs are dear to my heart, and a considerable chunk of my career was made in that area. As a designer, there is something fundamentally interesting in designing a language of any kind. It’s even more interesting and gratifying when people actually start exercising those languages to create non-trivial software systems. As a user, I love to use programming languages that I haven’t used before, even when the languages in question make me curse every other line.

But the truth of the matter is that ever since I finished my Ph.D. in the late 90s, and especially since I joined the ranks of Academia, I have been having a hard time convincing myself that research in PLs is a worthy endeavor. I feel really bad about my rational arguments against it, though. Hence this essay. Perhaps by the time I am done with it I will have come to terms with this dilemma.

Back in the 50s, 60s and 70s, programming languages were a BigDeal, with large investments, upfront planning, and big drama on standardization committees (Ada was the epitome of that model). Things have changed dramatically during the 80s. Since the 90s, a considerable percentage of new languages that ended up being very popular were designed by lone programmers, some of them kids with no research inclination, some as a side hobby, and without any grand goal other than either making some routine activities easier or for plain hacking fun. Examples:

• PHP, by Rasmus Lerdorf circa 1994, “originally used for tracking visits to his online resume, he named the suite of scripts ‘Personal Home Page Tools,’ more frequently referenced as ‘PHP Tools.’ ” [1] PHP is a marvel of how a horrible language can become the foundation of large numbers of applications… for a second time! Worse is Better redux. According one informal but interesting survey, PHP is now the 4th most popular programming language out there, losing only to C, Java and C++.
• JavaScript, by Brendan Eich circa 1995, “Plus, I had to be done in ten days or something worse than JS would have happened.” [2] According to that same survey, JavaScript is the 5th most popular language, and I suspect it is climbing up that rank really fast. It may be #1 by now.
• Python, by Guido van Rossum circa 1990, “I was looking for a ‘hobby’ programming project that would keep me occupied during the week around Christmas.” [3] Python comes at #6, and its strong adoption by scientific computing communities is well know.
• Ruby, by Yukihiro “Matz” Matsumoto circa 1994, “I wanted a scripting language that was more powerful than Perl, and more object-oriented than Python. That’s why I decided to design my own language.” [4] At #10 in that survey.

Compare this mindset with the context in which the the older well-known programming languages emerged:

• Fortran, 50s, originally developed by IBM as part of their core business in computing machines.
• Cobol, late 50s, designed by a large committee from the onset, sponsored by the DoD.
• Lisp, late 50s, main project occupying 2 professors at MIT and their students, with the grand goal of producing an algebraic list processing language for artificial intelligence work, also funded by the DoD.
• C, early 70s, part of the large investment that Bell Labs was doing in the development of Unix.
• Smalltalk, early 70s, part of a large investment that Xerox did in “inventing the future” of computers.

Back then, developing a language processor was, indeed, a very big deal. Computers were slow, didn’t have a lot of memory, the language processors had to be written in low-level assembly languages… it wasn’t something someone would do in their rooms as a hobby, to put it mildly. Since the 90s, however, with the emergence of PCs and of decent low-level languages like C, developing a language processor is no longer a BigDeal. Hence, languages like PHP and JavaScript.

There is a lot of fun in designing new languages, but this fun is not an exclusive right of researchers with, or working towards, Ph.Ds. Given all the knowledge about programming languages these days, anyone can do it. And many do. And here’s the first itchy point: there appears to be no correlation between the success of a programming language and its emergence in the form of someone’s doctoral or post-doctoral work. This bothers me a lot, as an academic. It appears that deep thoughts, consistency, rigor and all other things we value as scientists aren’t that important for mass adoption of programming languages. But then again, I’m not the first to say it. It’s just that this phenomenon is hard to digest, and if you really grasp it, it has tremendous consequences. If people (the potential users) don’t care about conceptual consistency, why do we keep on trying to achieve that?

To be fair, some of those languages designed in the 90s as side projects, as they became important, eventually became more rigorous and consistent, and attracted a fair amount of academic attention and industry investment. For example, the Netscape JavaScript hacks quickly fell on Guy Steele’s lap resulting in the ECMAScript specification. Python was never a hack even if it started as a Christmas hobby. Ruby is a fun language and quite elegant from the beginning. PHP… well… it’s fun for possibly the wrong reasons. But the core of the matter is that “the right thing” was not the goal. It seems that a reliable implementation of a language that addresses an important practical need is the key for the popularity of a programming language. But being opportunistic isn’t what research is supposed to be about… (or is it?)

Also to be fair, not all languages designed in the 90s and later started as side projects. For example, Java was a relatively large investment by Sun Microsystems. So was .NET later by Microsoft.

And, finally, all of these new languages, even when created over a week as someone’s pet project, sit on the shoulders of all things that existed before. This leads me to the second itch: one striking commonality in all modern programming languages, especially the popular ones, is how little innovation there is in them! Without exception, including the languages developed in research groups, they all feel like mashups of concepts that already existed in programming languages in 1979, wrapped up in their own idiosyncratic syntax. (I lied: exceptions go to aspects and monads both of which came in the 90s)

So one pertinent question is: given that not much seems to have emerged since 1979 (that’s 30+ years!), is there still anything to innovate in programming languages? Or have we reached the asymptotic plateau of innovation in this area?

I need to make an important detour here on the nature of research.

<Begin Detour>

Perhaps I’m completely off; perhaps producing innovative new software is not a goal of [STEM] research. Under this approach, any software work is dismissed from STEM pursuits, unless it is necessary for some specific goal — like if you want to study some far-off galaxy and you need an IT infrastructure to collect the data and make simulations (S for Science); or if you need some glue code for piecing existing systems together (T for Technology); or if you need to improve the performance of something that already exists (E for Engineering); or if you are a working on some Mathematical model of computation and want to make your ideas come to life in the form of a language (M for Mathematics). This is an extreme submissive view of software systems, one that places software in the back sit of STEM and that denies the existence of value in research in/by software itself. If we want to lead something on our own, let’s just… do empirical studies of technology or become biologists/physicists/chemists/mathematicians or make existing things perform better or do theoretical/statistical models of universes that already exist or that are created by others. Right?

I confess I have a dysfunctional relationship with this idea. Personally, I can’t be happy without creating software things, but I have been able to make my scientist-self function both as a cold-minded analyst and, at times, as an expert passenger in someone else’s research project. The design work, for me, has moved to sabbatical time, evenings and weekends; I don’t publish it [much] other than the code itself and some informal descriptions. And yet, I loathe this situation.

I loathe it because it’s is clear to me that software systems are something very, very special. Software revolutionized everything in unexpected ways, including the methods and practices that our esteemed colleagues in the “hard” sciences hold near and dear for a very long time. The evolution of information technology in the past 60 years has been _way_ off from what our colleagues thought they needed. Over and over again, software systems have been created that weren’t part of any scientific project, as such, and that ended up playing a central role in Science. Instead of trying to mimic our colleagues’ traditional practices, “computer scientists” ought to be showing the way to a new kind of science — maybe that new kind of science or that one or maybe something else. I dare to suggest that the something else is related to the design of things that have software in them. It should not be called Science. It is a bit like Engineering, but it’s not it either because we’re not dealing [just] with physical things. Technology doesn’t cut it either. It needs a new name, something that denotes “the design of things with software in them.” I will call it Design for short, even though that word is so abused that it has lost its meaning.

<Suspend Detour>

Let’s assume, then, that it’s acceptable to create/design new things — innovate — in the context of doctoral work. Now comes the real hard question.

If anyone — researchers, engineers, talented kids, summer interns — can design and  implement programming languages, what are the actual hard goals that doctoral research work in programming languages seeks that distinguishes it from what anyone can do?

Let me attempt to answer these questions, first, with some well-known goals of language design:

• Performance — one can always have more of this; certain application domains need it more than others. This usually involves having to come up with interesting data structures and algorithms for the implementation of PLs that weren’t easy to devise.
• Human Productivity — one can always want more of this. There is no ending to trying to make development activities easier/faster.
• Verifiability — in some domains this is important.

There are other goals, but they are second-order. For example, languages may also need to catch up with innovations in hardware design — multi-core comes to mind. This is a second-order goal, the real goal behind it is to increase performance by taking advantage of potentially higher-performing hardware architectures.

In other words, someone wanting to do doctoral research work in programming languages ought to have one or more of these goals in mind, and — very important — ought to be ready to demonstrate how his/her ideas meet those goals. If you tell me that your language makes something run faster, consume less energy, makes some task easier or results in programs with less bugs, the scientist in me demands that you show me the data that supports such claims.

A lot of research activity in programming languages falls under the performance goal, the Engineering side of things. I think everyone in our field understands what this entails, and is able to differentiate good work from bad work under that goal. But a considerable amount of research activities in programming languages invoke the human productivity argument; entire sub-fields have emerged focusing on the engineering of languages that are believed to increase human productivity. So I’m going to focus on the human productivity goal. The human productivity argument touches on the core of what attracts most of us to creating things: having a direct positive effect on other people. It has been carelessly invoked since the beginning of Computer Science. (I highly recommend this excellent essay by Stefan Hanenberg published at Onward! 2010 with a critique of software science’s neglect of human factors)

Unfortunately, this argument is the hardest to defend. In fact, I am yet to see the first study that convincingly demonstrates that a programming language, or a certain feature of programming languages, makes software development a more productive process. If you know of such study, please point me to it. I have seen many observational studies and controlled experiments that try to do it [5, 6, 7, 8, 9, 10, among many]. I think those studies are really important, there ought to be more of them, but they are always very difficult to do [well]. Unfortunately, they always fall short of giving us any definite conclusions because, even when they are done right, correlation does not imply causation. Hence the never-ending ping-pong between studies that focus on the same thing and seem to reach opposite conclusions, best known in the health sciences. We are starting to see that ping-pong in software science too, for example 7 vs 9. But at least these studies show some correlations, or lack thereof, given specific experimental conditions, and they open the healthy discussion about what conditions should be used in order to get meaningful results.

I have seen even more research and informal articles about programming languages that claim benefits to human productivity without providing any evidence for it whatsoever, other than the authors’ or the community’s intuition, at best based on rational deductions from abstract beliefs that have never been empirically verified. Here is one that surprised me because I have the highest respect for the academic soundness of Haskell. Statements like this “Haskell programs have fewer bugs because Haskell is: pure [...], strongly typed [...], high-level [...], memory managed [...], modular [...] [...] There just isn’t any room for bugs!” are nothing but wishful thinking. Without the data to support this claim, this statement is deceptive; while it can be made informally in a blog post designed to evangelize the crowd, it definitely should not be made in the context of doctoral work unless that work provides solid evidence for such a strong statement.

That article is not an outlier. The Internets are full of articles claiming improved software development productivity for just about every other language. No evidence is ever provided, the argumentation is always either (a) deducted from principles that are supposed to be true but that have never been verified, or (b) extrapolated from ad-hoc, highly biased, severely skewed personal experiences.

This is the main reason why I stopped doing research in Programming Languages in any official capacity. Back when I was one of the main evangelists for AOP I realized at some point that I had crossed the line to saying things for which I had very little evidence. I was simply… evangelizing, i.e. convincing others of an idea that I believed strongly. At some point I felt I needed empirical evidence for what I was saying. But providing evidence for the human productivity argument is damn hard! My scientist self cannot lead doctoral students into that trap, a trap that I know too well.

Moreover, designing and executing the experiments that lead to uncovering such evidence requires a lot of time and a whole other set of skills that have absolutely nothing to do with the time and skills for actually designing programming languages. We need to learn the methods that experimental psychologists use. And, in the end of all that work, we will be lucky if we unveil correlations but we will not be able to draw any definite conclusions, which is… depressing.

But without empirical evidence of any kind, and from a scientific perspective, unsubstantiated claims pertaining to, say, Haskell  or AspectJ (which are mostly developed and used by academics and have been the topic of many PhD dissertations) are as good as unsubstantiated claims pertaining to, say, PHP (which is mostly developed and used by non-academics).  The PHP community is actually very honest when it comes to stating the benefits of using the language. For example, here is an honest-to-god set of reasons for using PHP. Notice that there are no claims whatsoever about PHP leading to less bugs or higher programmer productivity (as if anyone would dare to state that!); they’re just pragmatic reasons. (Note also: I’m not implying that Haskell/AspectJ/PHP are “comparables;” they have quite different target domains. I’m just comparing the narratives surrounding those languages, the “stories” that the communities tell within themselves and to others)

OK, now that I made 823 enemies by pointing out that the claims about human productivity  surrounding languages that have emerged in academic communities — and therefore ought to know better — are unsubstantiated, PLUS 865 enemies by saying that empirical user studies are inconclusive and depressing… let me try to turn my argument around.

Is the high bar of scientific evidence killing innovation in programming languages? Is this what’s causing the asymptotic behavior? It certainly is what’s keeping me away from that topic, but I’m just a grain of sand. What about the work of many who propose intriguing new design ideas that are then shot down in peer-review committees because of the lack of evidence?

This ties back to my detour on the nature of research.

<Join Detour> Design experimentation vs. Scientific evidence

So, we’re back to whether design innovation per se is an admissible first-order goal of doctoral work or not. And now that question is joined by a counterpart: is the provision of scientific evidence really required for doctoral work in programming languages?

If what we have in hand is not Science, we need to be careful not to blindly adopt methods that work well for Science, because that may kill the essence of our discipline. In my view, that essence has been the radical, fast-paced, off the mark design experimentation enabled by software. This rush is fairly incompatible with the need to provide scientific evidence for the design “hopes.”

I’ll try a parallel: drug design, the modern-day equivalent of alchemy. In terms of research it is similar to software: partly based on rigor, partly on intuitions, and now also on automated tools that simply perform an enormous amount of logical combinations of molecules and determine some objective function. When it comes to deployment, whoever is driving that work better put in place a plan for actually testing the theoretical expectations in the context of actual people. Does the drug really do what it is supposed to do without any harmful side effects? We require scientific evidence for the claimed value of experimental drugs. Should we require scientific evidence for the value of experimental software?

The parallel diverges significantly with respect to the consequences of failure. A failure in drug design experimentation may lead to people dying or getting even more sick. A failure in software design experimentation is only a big deal if the experiment had a huge investment from the beginning and/or pertains to safety-critical systems. There are still some projects like that, and for those, seeking solid evidence of their benefits before deploying the production version of the experiment is a good thing. But not all software systems are like that. Therefore the burden of scientific evidence may be too much to bear. It is also often the case that over time, the enormous amount of testing by real use is enough to provide assurances of all kinds.

One good example of design experimentation being at odds with scientific evidence is the proposal that Tim Berners-Lee made to CERN regarding the implementation of the hypertext system that became the Web. Nowhere in that proposal do we find a plan for verification of claims. That’s just a solid good proposal for an intriguing “linked information system.” I can imagine TB-L’s manager thinking: “hmm, ok, this is intriguing, he’s a smart guy, he’s not asking that many resources, let’s have him do it and see what comes of it. If nothing comes of it, no big deal.” Had TB-L have to devise a scientific or engineering assessment plan for that system beyond “in the second phase, we’ll install it on many machines” maybe the world would be very different today, because he might have gotten caught in the black hole of trying to find quantifiable evidence for something that didn’t need that kind of validation.

Granted, this was not a doctoral topic proposal; it was a proposal for the design and implementation of a very concrete system with software in it, one that (1) clearly identified the problem, (2) built on previous ideas, including the author’s own experience, (3) had some intriguing insights in it, (4) stated expected benefits and potential applications — down to the prediction of search engines and graph-based data analysis. Should a proposal like TB-L’s be rejected if it were to be a doctoral topic proposal? When is an unproven design idea doctoral material and other isn’t? If we are to accept design ideas without validation plans as doctoral material, how do we assess them?

Towards the discipline of Design

In order to do experimental design research AND be scientifically honest at the same time, one needs to let go of claims altogether. In that dreadful part of a topic proposal where the committee asks the student “what are your claims?” the student should probably answer “none of interest.” In experimental design research, one can have hopes or expectations about the effects of the system, and those must be clearly articulated, but very few certainties will likely come out of such type of work. And that’s ok! It’s very important to be honest. For example, it’s not ok to claim “my language produces bug-free programs” and then defend this with a deductive argument based on unproven assumptions; but it’s ok to state “I expect that my language produces programs with fewer bugs [but I don't have data to prove it].” TB-L’s proposal was really good at being honest.

Finally, here is an attempt at establishing a rigorous criteria for design assessment in the context of doctoral and post-doctoral research:

• Problem: how important and surprising is the problem and how good is its description? The problem space is, perhaps, the most important component for a piece of design research work. If the design is not well grounded in an interesting and important problem, then perhaps it’s not worth pursuing as research work. If it’s a old hard problem, it should be formulated in a surprising manner. Very often, the novelty of a design lies not in the design itself but in its designer seeing the problem differently. So — surprise me with the problem. Show me insights on the nature of the problem that we don’t already know.
• Potential: what intriguing possibilities are unveiled by the design? Good design research work should open up doors for new avenues of exploration.
• Feasibility: good design research work should be grounded on what is possible to do. The ideas should be demonstrated in the form of a working system.
• Additionally, design research work, like any other research work, needs to be placed in a solid context of what already exists.

This criteria has two consequences that I really like: first, it substantiates our intuitions about proposals such as TB-L’s “linked information system” being a fine piece of [design] research work; second, it substantiates our intuitions on the difference of languages like Haskell vs. languages like PHP. I leave that as an exercise to the reader!

Coming to terms

I would love to bring design back to my daytime activities. I would love to let my students engage in designing new things such as new programming languages and environments — I have lots of ideas for what I would like to do in that area! I believe there is a path to establishing a set of rigorous criteria regarding the assessment of design that is different from scientific/quantitative validation. All this, however, doesn’t depend on me alone. If my students’ papers are going to be shot down in program committees because of the lack of validation, then my wish is a curse for them. If my grant proposals are going to be rejected because they have no validation plan other than “and then we install it in many machines” or “and then we make the software open source and free of charge” then my wish is a curse for me. We need buy-in from a much larger community — in a way, reverse the trend of placing software research under the auspices of science and engineering [alone].

This, however, should only be done after the community understands what science and scientific methods are all about (the engineering ones — everyone knows about them). At this point there is still a severe lack of understanding of science within the CS community. Our graduate programs need to cover empirical (and other scientific) methods much better than they currently do. If we simply continue to ignore the workings of science and the burden of scientific proof, we end up continuing to make careless religious statements about our programming languages and systems that simply will lead nowhere, under the misguided impression that we are scientists because the name says so.

Copyright © Crista Videira Lopes. All rights reserved.
Note: this is a work-in-progress essay. I may update it from time to time. Feedback welcome.

Dear @bby,

I am being asked to write a recommendation letter for someone who has been working with me for 3 years and who I think sucks. What should I do? Should I simply decline to do it? Or should I say what I honestly think about that person and his work? — because he deserves it!

Sincerely, Conflicted Recommender

Dear Conflicted Recommender: By by now you probably realized that recommendation letters are the currency of your trade. A negative recommendation may stop that person’s career on its tracks. You have the power! Before you indulge yourself in dishing that poor soul and pulling the rug from under him so unelegantly, ask yourself the following questions: (a) Does that person’s work really suck, or is your dislike of him, and it, a consequence of a bad fit, mismatched expectations and/or personal issues? (b) Did you take the time to tell that person how displeased you were with his work, how he needed to change if he wanted to continue to work for you, and, most importantly, how his poor performance would influence your recommendation to future employers?

Your answers to these questions will determine the right course of action.

If you can’t guess, the no-no situation here is for you to write a letter that sounds like a Youtube comment. That can actually work against you, and can thwart your plans to end that person’s career. You accumulate Youtube commenter vitriol (a form of black energy recently discovered by astrophysicists) when you let things roll with that person without ever expressing your concerns directly to him, or take a passive-aggressive attitude, until the time comes to write that killer letter that will avenge your frustration with that person. The professor reading that letter on the other end will likely think you, not him, are the jerk. Let’s be honest: you might have been the jerk here. How could you possibly waste 3 years supervising someone who you think sucks? You should have gotten rid of him long ago.

So make sure that during those 3 years of unpleasant interactions you made your concerns known to that poor soul. If he was foolish enough to stay, and even more foolish to ask you for a recommendation letter, then you have gained the right to dish him. Do so carefully. Start by saying that you made all you could to get rid of him, and even told him the letter wouldn’t be good, but that he didn’t understand, so you are in the unfortunate situation of having to write this letter.

Better yet: don’t write it at all. If you have nothing nice to say, decline to give your recommendation. The professor on the other end will get the smoke signals and will interpret them accordingly. This is a manner of conveying what you think without having to say a word, so don’t let your emotions get in the way.

Dear @bby is written by @bb1941l v@n Bµr3n. @bby channels uncommon common sense for cynic academics. Send her your difficult questions by email to abby at this web site’s domain name.

As you deploy your experiment among those women, you observe that not only it fails to produce the results you expected, but it also accelerates the process of aging: within 9 months, those 40-something women suddenly look like they are 60 years old. You scratch your head, and go back to the drawing board trying to understand where your beautiful model went wrong. You don’t feel bad for those women, after all they were willful volunteers; but your ego is bruised because your wonderful idea didn’t work, and the possibility of a Nobel prize is farther than ever.

What I just described to you is a typical scenario in routine Ethics courses that all of us academics in the US need to go through every few years.

Now imagine this. You have a brilliant idea for how to generate value in economic markets, a clever combination of bets on future values that you have been developing in your lab with your graduate students, and that will open the possibility of riches to a whole breed of entrepreneurial financiers, as well as to entire economies by trickle-down effects. You have done your Math, you have developed a model that explains why your idea works. You have simulated it using powerful computers. You got 90% success. You have very little doubt that it works in real economies too. If only you could test it. Well, it’s 1995, there are no Institutional Review Boards for Economics, no Ethics committees to go through — the issue of Ethics doesn’t even register in your mind as something to worry about. There are only some political barriers to your ideas, some old laws from post-Great-Depression years that vaguely prevent those ideas from being put to practice. With help from powerful friends who believe that your ideas will make them even richer, those laws are quickly and quietly deactivated, one by one. In order to test your ideas, you simply need to recruit governments, organizations and individuals who are cash-strapped. Your confidence in your Mathematical model plus those countries’, organizations’ and individuals’ desperation are a powerful combination for testing your ideas, they will try anything! — especially a financial ‘cure’ that is undersigned by such world-renowned economics professor such as yourself. You dream of a Nobel prize or maybe you already won one.

As you deploy your experiment, you observe a bubble that first gives the impression of economic growth but that then comes crashing down, leaving behind it millions of people, organizations and governments with debts they can’t pay. Entire countries’ economies are in ruins, social unrest is everywhere. You scratch your head and go back to the drawing board trying to understand where your beautiful model went wrong. You don’t feel bad for the economic disaster; you feel proud for trying such a clever, innovative model, and for observing that your ideas had a strong effect in real life. You don’t empathize with the millions of people who got screwed out of their hopes and desires; after all, they were willful participants in your experiment, just like the women who were too old to conceive. It’s really fun to see your mathematical model come to life at the macro-economic scale! Perhaps you need to tweak a few parameters for next time. You back up your pride with the assurance that the crisis was not predicted, because economic theory predicts that such events cannot be predicted. Trivial mathematical truth that applies to just about everything that is complex and stochastic but that, said by you with your awards and recognitions to back you up, sound like something complex that lay people are unable to understand.

Θ

I don’t know about you, reader, but this state of affairs in Economics research, what is going on in world-famous Business Schools and their involvement with high-finance companies, bothers me a lot. Some of my esteemed academic colleagues seem to be completely out of control and out of touch with reality — not unlike the state of affairs in Medical research until that community felt the need to establish Institutional Review Boards in the second half of the 20th century. I have no doubt they have good intentions — most people don’t do evil on purpose, not even those physicians in the past who committed atrocities in the name of advancing knowledge. Economists happen to live and work in the sciences of the artificial, a place of mind that has very little contact with people and empirical data (how I understand the appeal!). They dwell in their stochastic, abstract models of how markets work, and, in the absence of the need for empirical validation, it’s easy to get enamored by such models. Economists would be as harmless as theoretical computer scientists… if it wasn’t for the real damage that comes from their collaboration with greedy financiers, who see in those models additional chances to make a buck.

Perhaps it’s time for the Economics research community to start having an internal conversation about their trade, and for the rest of us academics to put some pressure on them. Here is a start — an eye-opening essay by one of them. This essay may be hard to digest, so here’s a documentary that deconstructs the financial meltdown in simple bottom-line punches and puts some of the spotlight on some of those academics — watch it on Netflix.

I’m chairing SPLASH/OOPSLA this year. That means that I’m like a Producer, I get to do all the work behind the scenes in order to make the conference come to life. And it’s finally coming to life. After one year and a half of “programming,” I just pressed “Run.” It’s a little crazy if you believe in agile. A whole year and a half of designing and “programming,” with no testing whatsoever, no small chunks, just a long process of envisioning, estimating, guessing, coordinating, signing contracts, making decisions; then we unleash the event during 5 days over almost 600 people and hope for the best!

So what’s involved in producing a conference like SPLASH? Read on if you want to know.

For the most part, there is a huge amount of coordination work that needs to be done. The producer needs to coordinate with the ACM (the sponsors of the conference), the hotel(s), the conference venue, the A/V and Internet people, external restaurants/entertainment, the industry supporters, and the registration people, among other 6 or so miscellaneous services. This is the administrative, logistic and operational side of the conference. I had produced one conference before SPLASH; for that one, I had no idea what I was getting myself into, so I ended up doing all this work myself, because I realized (too late) that if I didn’t do it, it wouldn’t happen — a duh moment for 1st timers. For SPLASH, I knew better. So from the onset I brought in one wonderful person from the Institute for Software Research, Debi Brodbeck, who is an absolute maniac when it comes to getting things done. She is an “animal” in the sense that Paul Graham uses that word in his essay “How to start a startup.” If I ever would start a startup and had to hire a COO, I would hire Debi in an instant. We’re so very lucky to have her at UC Irvine…! It really speaks to the environment we have at the University; that we have these absolutely fantastic staff people who like to work there, when they probably would make a lot more money if they would jump to positions in companies. Not unlike us faculty, I guess… except that we’re usually not nice.

Anyway, besides the administrative, logistic and operational aspects of the conference, there are also strategic and content aspects to it. These are the ones I called upon myself to take care of, again, with the help of Debi who also knows a lot about these issues, and the Steering Committee of SPLASH. When I accepted to do this, I realized that the conference was in flux trying to find its position in a context that is quite different from what we had in the 90s, when OOP was the big thing and everyone wanted to go to OOPSLA. It was this strategic challenge that made me accept to produce SPLASH 2011.

There are other big things now, and some of them have their own conferences; there are many developer-oriented conferences that took the ideas from OOPSLA, matured them, and made them even better for those audiences. For about two decades or so, OOPSLA has been right on the edge where academic and industrial research meets advanced development. It’s a balancing act at that edge. I didn’t think I could bring SPLASH back to the golden days of 2,000-people OOPSLA, and that wasn’t even a goal for me. Things have changed, and I loathe staring into the past. My goal here was to try to formulate a mission statement for SPLASH that goes beyond catchy, meaningless groups of words (we know OOP is not hot anymore, it’s everywhere), and that truly captures the uniqueness of this community — because I believe there’s something really unique here that we don’t find in any other conference.

So, what is the uniqueness of SPLASH? Why would someone attend SPLASH as opposed to, say, RubiConf, EclipseCon, Goto, Qcon, or academic conferences like PLDI, POPL, and the like? As I said, SPLASH sits right at the edge of these two types of conferences. Look at the program this year: the first keynote speaker is Turing award winner Ivan Sutherland; the third keynote speaker is Mr. JavaScript Brendan Eich; the keynote speaker in the middle is a Swiss academic, Markus Puschel, with some pretty wacky ideas on performance/productivity. Where else could we possibly find this combination of keynote speakers in one conference?!

This edge is not for everyone. Many people are better served if they go to conferences that have only Brendan Eich type of speakers or Ivan Sutherland type of speakers or Markus Puschel types of speakers, but not the combination of the three. And that’s ok. But this is the uniqueness of SPLASH: it’s a hybrid, a melting pot of software development approaches. As you go from session to session you may have the impression that you are traveling between distant planets!

The rest of the [vast] program reflects this hybrid combination, with academic research papers woven with experience reports, idea-papers and demonstrations. Even the 3 TechTalk speakers are a hybrid bunch: Jesper will geek out on “How to handle 1M daily users without a cache;” Dave will entertain us with a rant on “Why modern application development sucks!;” and Kresten will tell a more personal story of his involvement with Erlang.  Let’s not forget the self-hybrid that is the incredible Guy Steele doing a live demonstration of singing calls! The days preceding the main conference are also full of interesting talks and events with the same hybrid characteristic: from the Dart people (at DLS and VMIL), who have just unleashed one of the largest programming language design experiments ever over all of us, to Brad Myers, who studies the human aspects of programming in relatively controlled environments, to the AWS Hackathon.

You can probably sense the pride that I have in being the producer of this conference. I’m not going to hide it, I am proud of being part of this wacky hybrid! And I am extremely grateful to all the people who have helped put this conference together. As many said before me, the most important thing for a team leader to do is to put in place a great team and move him/herself out of the way!

Now, let me go monitor the execution of this test-less program… I have a few amulets in my pocket!

Let me explain briefly how we detect aspects in this study. We don’t; we detect topics using an unsupervised topic modeling technique called Latent Dirichlet Allocation (LDA). This technique is used very successfully for topic modeling  in text. It works by finding co-occurrences of words in bags of words (files/texts/whatever); the topics emerge given the most frequent co-occurrences of groups of words. It’s a probabilistic model, meaning that it has some nice smoothing properties and a few other goodies such as the distribution (aka entropy) of those topics across files and within files — a direct measure of scattering and tangling of topics.

Let me back track a little. This work came to be because Erik, who, at the time, knew nothing about AOP, decided to run LDA on a few hundred Java projects we had laying around; then one day he showed me the latent topics that emerged from that experiment, and that’s when it hit me — holy crap!, these are aspects! So yeah, I wish I could tell this was all part of a grand plan, but it wasn’t; it was an accidental finding.

Scattering of topics in the entire repository

After that first experiment, we tuned a few things in our method of using LDA (like, for example, carefully selecting which words to use) and ran it again over a much larger collection of Java projects. Again, the surprising finding was that the topics that emerged with high entropy across that very large collection of Java programs included, among many others, the concerns that the AOP community had been using as examples of aspects in the first place: persistence, authentication, exception handling, concurrency, etc. This was very interesting, and immediately made me want to explore how the concept of latent topic related with the concept of aspect.

But something was still a bit off: we ran LDA on the entire collection of 5,000 projects or so; we were treating the entire collection as one big project. A such, and on second thoughts, it was not surprising to find that things like concurrency cut across many projects/files. The latent topics with high entropy read almost like a “list of things you need to deal with if you program in Java.” Perhaps if we ran it on a per project basis, things would look different.

Scattering of topics in some projects

Indeed, things looked different on a per-project basis. There was some overlap on generic things like string manipulation; but the interesting thing is that some of those topics were very specific to the projects in question. For example, for JHotDraw (a graphics editor) we got topics that are clearly related to drawing; for Jikes (a Java compiler) we got groups of words that are quite reasonable to find in a compiler; for CoffeeMud (a MUD game engine) we got this mysterious word “mob” prominently represented in several topics; we had to look into the documentation of that project to solve the mystery: it stands for Mobile OBject, a central concept in that engine — basically NPCs.

Scattering curves for selected projects

Then we plotted the topic distributions for 5 projects on the same plot and saw an interesting fact: some projects show more topic scattering than others.  CoffeeMud is noticeably more scattered. JHotDraw has a handful of very scattered topics, but most of the topics in it are a lot less scattered than all topics in all other projects. Jikes is the most regular one, with less variation.

Could this be a good metric for measuring the accidental complexity of projects? I.e. is the CoffeeMud code base a mess compared to, for example, the code base of Jikes? This is definitely an interesting conjecture, but it’s nothing more than a conjecture at this point. That relates to the second never-challenged assumption underlying AOP: that tangling and scattering are “bad” for software development. Our study here shows that there is scattering and tangling of topics, and gives us a nice mathematical framework for quantifying those effects, but it says absolutely nothing about whether that is a good thing or a bad thing.

The results raise a lot of questions for AOP, and the paper has a lengthy discussion regarding the findings. I’m going to end this summary with the issue that I think is more itchy.

A lot of scattered topics we saw emerging were clearly uses of specific APIs — string manipulation, list manipulation, xml, io, etc. This resonates with my experience; some things are used everywhere. “String manipulation,” for example, could be seen as an aspect of programs; it would be perfectly reasonable to want to understand what the entire program is doing wrt that. If AOP says that scattering is bad, does it mean that scattering of string manipulations is bad? Should we localize string manipulations in programs? Should we extract them out into a separate module and do reverse binding?

If that would be odd, then a question is raised about the reason for doing that for concerns such as logging and concurrency. At the very least, that practice, advocated by AOP from early on, requires a better justification.

I have my own view on these questions, but that’s a topic for another paper/post.

This work was a collaboration between myself, Pierre Baldi and Erik Linstead, with help from Suhil Bajracharya. Work in part supported by National Science Foundation grants EIA-0321390, CCF-0347902 and CCF-0725370 and a Microsoft Faculty Research Award.

The ultimate goal of this line of work in my group is to get a better understanding of the social dynamics that emerged on the Web with user-generated content sites — wikis, social networks, virtual worlds, etc. Unlike interactions in the real world, these sites collect an enormous amount of data regarding the interactions that people have with each other and with the content. As such, one can now feed that data into computing machinery and hope to gain insights on social dynamics [on the Web, at least, but maybe beyond]. Wikipedia happens to be a wonderful playground for that line of investigation, because all the data is available.

Vandalism is a very strong word but with a clear definition within Wikipedia. Wikipidians have strict policies and guidelines for editing articles, and they spend a lot of time fighting editorial “crimes.” They have some bots that do basic housekeeping; some of those bots [1] trigger alarms and/or revert edits in obvious cases of vandalism. But not all vandalism is obvious. Improving automatic vandalism detection is therefore a goal that many people set out to accomplish — including one of my former students, Sara, who just graduated last month and is now working for Microsoft. There is a research community around this topic that curates data, organizes workshops and puts up competitions from time to time — that’s PAN. Sara participated in this community, and won 3rd price in the PAN competition in 2010.

The paper I’m focusing on here is the culmination of Sara’s work. Here is what we did, in a nutshell: first, we collected a relatively large number of features that had been known to be of value for vandalism detection in Wikipedia — a total of 66 features. Most of these features had been proposed and tested by other people, others were our own. These features all fell nicely into four groups: user features, textual features, metadata features and language model features. Then we trained a random forest classifier with all those 66 features using the PAN corpus train set. Finally we run that classifier on the PAN corpus test set. With such a feature-rich model we were able to achieve  the highest performance ever reported for that corpus, an AUC of 0.9553 — the previous record was 0.9218.

However, feature-rich models aren’t very practical; they are slow to compute. For all practical usages of ML, especially the ones that run online, we need models with few and cheap features. Machine learning approaches tend to suffer from this problem: we throw a large number of intuitions about what matters for classification, and let the machine figure it out, but the machine doesn’t ever tell us what’s really important, what’s not so important, or how the features correlate. So we need to do extra work in order to find that out. Here’s what we did.

In order to detect and eliminate redundant features, we performed two sets of experiments. First, we studied the contribution of each of the 4 groups of features to determine if any of those groups could be eliminated without a significant drop in AUC. Then we studied the contribution of each feature individually and used the results for eliminating redundant features, using a technique called Lasso (Least Absolute Shrinkage and Selection Operator).

The first set of experiments told us that the User features were the most important group — by a lot. With the User features alone, we obtained an AUC of 0.9225. In the second set of experiments, we were able to reduce the model to 28 features (down from 66) and still obtain an AUC of 0.9505. These 28 features include features from all groups, but the User features have a strong presence.

And there you have it, this is how we answered the question “If you have to develop a classifier for detecting vandalism in Wikipedia with just a small number of features, what kind of features do best?” But I think the result we obtained is more interesting that its practical application on vandalism detection in Wikipedia. What the result suggests is that there are very strong signals associated with the users’ actions within a system, i.e. who the user is, as given by the sequence of actions of that user within the system. It’s not just that someone added text; it’s who that person is. This gets at the concept of reputation, but goes at it from an implicit, within-the-system perspective, rather than with an explicit thumbs-up-thumbs-down kind of approach. It suggests that it is possible to automatically build extremely accurate models of users’ reputations without explicit endorsements from other users.

This work was a collaboration with David McDonald, and it was supported by the National Science Foundation under grant No. OCI-074806.

The clones

In order to understand the nature of these clones we looked into a large number of projects that had clones in them in order to identify the circumstances under which the files ended up being clones. Here’s a categorization of what we found:

• Demos/tutorials: these are files that were specifically intended to be used as examples of functionality, usually for a specific library or framework. These
  files usually originate from example or demo programs, or are working code fragments included with tutorials. We found files for SWT, JBoss, Java Servlets, Swing, and a number of other well-known libraries.
  We think this category is a perfectly legitimate case for file cloning. These are files designed to be copied and executed in new projects, and they do not make up any integral part of the system.
• Small library / utility files: these are clones that appeared to come from small third-party libraries or self-contained utility files.  Some examples include a single-file Java port of GNU Getopt, a tool for encoding PNG files, a Java connector for Spidermonkey, and a file for converting CVS date strings.
  It is difficult to classify this category as strictly good or bad. On the one hand, libraries are being reused through copying, which eliminates the connection to their original source and carries with it a whole host of maintenance issues. On the other hand, these libraries are all rather small and their functionality not overly complicated. Furthermore, especially in the case of copied utility files, there may not have been any reasonable way to reuse the functionality without copying the files. And developers might be hesitant to introduce an external dependency for the use of a handful of files.
• Library: this category differs from the previous one only in that the copied libraries are larger and more well known. Examples of this category are split between those projects that copied significant portions of common libraries, and those that copied only a handful of files. The copied libraries include Jython, Apache Beanutils, SWT, JUnit, and the Java Excel API.
  The most interesting example we found in this category was this: the version of Apache Lenya in our dataset contained a complete copy of Apache Cocoon, a spring-based framework that Lenya uses. In trying to discover why this copy existed, we looked at the most recent version of Lenya, which no longer contains a copy of Cocoon. Instead, it has been replaced by a script to check out Cocoon and then automatically apply a number of patches. It appears the developers of Lenya needed to modify portions of Cocoon, and originally did this by copying the entire library. Only later did they settle upon the patching mechanism to achieve the same result.
  Clones in this category are a bit uglier than those in the previous category. These libraries are larger and more complex, and so are more likely to contain bugs. As seen with Lenya, one possible motivation behind this copying is  developers wishing to modify portions of the library. They might also wish to remove aspects that they don’t need. While understandable, we would hesitate to recommend such action except when absolutely necessary. Lenya’s current  solution is clearly preferable.
• Related Projects: this category includes those clones that occurred between two projects that are related in some way. This includes project forks, sub-projects, and renamed projects. Roughly 1/3 of the cases in this category were due to a developer beginning a new project by copying the entirety of an older project.
  This type of cloning can clearly impact the maintainability of a system, but, if handled properly, forms a reasonable part of an open source project’s lifecycle.
• Duplicated Project: This category is similar to the previous category, except instead of the projects simply being related, they are exact copies. The most common cause of this is that the project has been simultaneously placed in multiple open source repositories. Usually the actual version control is handled by one repository, while the other contains a package distribution. While projects in this category are not clones in the same sense as the other categories, their duplication can be a source of confusion to those looking to find a project’s real home.
• Java Standard Library: this category was quite unexpected. We were surprised to find so many projects containing Java itself. On further investigation, we discovered a large number of applications designed to transform Java code that contained their own, often modified, versions. We found tools for converting Java to Javascript, multiple implementations of Java Virtual Machines, and a few Java compilers. Cloning of this type is necessary, but also very limited in the scope of projects that require it.
  We also found a large number of projects including files from org.xml.sax and org.w3c.dom, despite their being included with the JDK. These libraries are something of a special case, they are on a more frequent release schedule than the JDK itself, and were not included in older versions.
• Java Extensions: this category contains copies of files from common Java extensions, such as JAXP or JMX. While these extensions are now packaged with the JDK, this has not historically been the case. The sheer number of times that slightly different versions of these extensions appear as source in different projects suggests the importance of a better mechanism for handling extensions before their inclusion in the JDK.
• Other: the common theme of this category is their being extremely ugly examples of file cloning. There were two related projects that copied an Apache library, yet renamed every package to something slightly different. In an extreme case, a developer copied someone else’s Java application for animating images and uploaded it to a new repository under his name.

The data

The data consisted of a universe of 13,241 Java projects collected from repositories such as Apache, Java.net, Google Code, and Sourceforge, in a total of 3,237,910 files. The picture on the right shows the distribution of project size in this collection. Read it like this: there are about 5,000 projects with 1 to 50 files (left-most column), etc., all the way to there are about 90 projects with 2001 to 17,893 files (right-most column). Indeed, there are a few massive projects out there!

I’d like to hear people’s personal experiences with file cloning in their projects. Do you copy-and-paste code a lot? At the file level, smaller or bigger?

This study was led by Joel Ossher, with help from Hitesh Sajnani and with me supervising. The work was supported by the National Science Foundation under Grant No. CCF-1018374.

The Insider Scoop

Here’s the secret: if your resume is just plain good, and not insanely awesome, the single most important pieces of the application process are (1) your direct contact with a prospective advisor, and (2) your recommendation letters. Forget about GPAs and GREs. You and the interactions you have with others are _it_. Professors like me are looking for good people who are driven to produce great work and who can grow into independent intellectuals under their supervision. GPAs and GREs give some information about you, but they miss to capture the potential you have for doing independent work. Besides the signs of potential for good, independent work (usually given by papers and/or projects), the impression that you make on your prospective advisor is the deal maker or breaker. As such, it is really important that you prepare for that, as much as (or more than) you prepare for the GRE.

Matt has some good advice on how to get into grad school. My post here focuses on the interactions that you will have with prospective advisors.

Your Context

So, you are sitting there in front of your computer — in India, China, Iran, or even in the U.S. You just graduated, or are about to graduate, from an undergraduate institution, and you have no idea what you want to do with the rest of your life. The prospect of getting a job doesn’t entice you. Or maybe you tried it, and you think that there ought to be more intellectual stimulation in your life. You like being in school, you admired some of your professors, you enjoyed doing some projects, you have good grades, and your family values Higher Education. Some of your friends and acquaintances are doing a PhD, and you have a secret envy for them — not the least of it because they seem to continue to have the relatively care-free lifestyle that characterizes student life, and that is very different from the lifestyle that your other friends have, those who got a job. And then it hits you — you could do it too! I’ll leave the reasons for choosing to go to graduate school for another post. Let’s just say you made up your mind. But where to start?

You frantically Google for information about graduate schools. You hit the so-called “top 10″ Universities, and you day-dream about being a nerd-wiz at MIT. Then reality hits you, and you start looking at other options. You exhaustively browse through faculty web pages to check out what they do and imagine how it would feel working under their supervision. Finally you feel ready to shoot up the first contact messages. And this the first opportunity you have… for your plans to fail. Right here, in the beginning.

The Many Ways of Succeeding in Making a Bad Impression

Want to make sure your application is ignored? Send an email that reads like spam. Here is the worst possible email you can send:

Dear  Sir,

I am XX YY from UUU University in CCC. I am seeking an opportunity to use my background to do research in your prestigious lab. I have seen your publications and research work and they are of great interest to me. I am attaching my resume. I had acquired programming skills on C, C++, VC++, Java, Oracle, and MS SQL. Also I had conducted relevant projects during that period.

I am rather sorry to trouble you. It would be very helpful to me if you could convey to me the chances I have of getting into the PhD program in your university.

Sincerely yours,
XX YY

OK, stop right there. If this is the best you can do, don’t even bother doing it! And if the reason isn’t clear, let me explain what’s wrong with this email. There is nothing in this email that differentiates prospective University A from prospective University B, much less prospective advisor L from prospective advisor M. Why should I pay attention? It’s really easy for you, because it’s the exact same message, but what message are you really sending me, the receiver? Essentially, you are saying “I’m so lazy, I’m not bothering to write down anything that will force me to manually type a few differentiating parts in the 100 emails that I am about to spam professors with.” Professors don’t like spam and they also don’t want lazy people in their labs — at least not uninterestingly lazy like this. It’s a #SureFail.

(There is a slightly worse variation of this email, one that discloses your background in, say, signal processing, while spamming professors whose research areas have nothing to do with that…)

Here is the second-to-worst email that you can send: (notice the typesetting)

Dear    Professor Lopes   ,

My name is XX YY from UUU University in CCC. I am writing to explore the possibility of becoming a PhD student in your lab at University of California Irvine .

I am very interested in pursuing research in    Mining Software Repositories . My previous educational and work experience has given me solid background in analysis/design/programming.

Attached please find my resume in word format. I am looking forward to your reply. Thank you.

XX YY

This is slightly better than the first because at least you went to the trouble of writing down some differentiating bits. Unfortunately the typesetting reveals the spamming nature of this message. The differentiating parts were pasted using a different font, revealing that you have composed this message from a template with blanks on it. Then you simply went through a list that you may have compiled in Excel or something, listing professors names and research areas, and voila! – insta-spam again. #SureFail.

You may feel that I’m being unfair in calling this spam, because you have done your homework in researching professors and their research interests. But I’m not being unfair. This is still spam from where I stand. This message is telling me: “Hey, I really want to be admitted… somewhere, working with… someone, doing… something. I don’t care where, who or what, I just want to go to graduate school.” Well, professors like to think that their work matters. They aren’t just looking for generic brain power, they are looking for people who are a good fit and who show signs of being able to find out what they want.

What Your First Email Should Look Like

What I am about to tell you is about quality and is fairly incompatible with sheer quantity. If you do what I say, there’s just not enough time for you to do this for 100 professors. As such, narrow your search to just a few and target them wisely.

First of all, understand what kind of work goes on under that professor’s supervision. This means that you need to figure out what projects are going on in that lab, what publications are being produced, who that professor’s current graduate students are, and who the past graduate students were and where they are working. You need to read through a few of those papers. If you care about your career, you absolutely need to do this thorough background research, not just look at the professor’s home page to pick up a few keywords. You are about to embark on a 5-6 year engagement that will have a profound effect on the rest of your life. Caring about who, what and where you are committing yourself to, respecting yourself, is the basis for making a good impression in others.

Then write emails specifically for each professor you are interested in working with. Don’t send the application essay in that email, leave that to where it belongs — in the application materials. In the emails, be short, but be specific for each of them. Just cover the main points. Tell them what it is of their labs’ work that got your attention, and why you’d like to join. If you refer to a paper or two written by the professors and their students, and make some intelligent comments about them, I guarantee you you will get the professors’ attention.

As I said above, this takes time and engagement on your part. But here’s the thing: in the absence of substantial research experience, the time you take to study the prospective professors’ work and the intellectual engagement you show with that work are the first real indicators of your potential as a researcher, and how good of a fit you are to the professors’ labs.

Coda

If you read the entire post, feel free to email me if you’re considering applying to UCI/ICS for graduate work, and especially, if you are interested in my lab. Besides the ‘old’ program in Information and Computer Sciences, we have a shining new Ph.D. program in Software Engineering.