Fabulous slide show, showing the problems with peer-review in CS today

If you are interested in peer review, see this presentation by Jeffrey Naughton.

Slides 14-20, 28-30 and 35 onward are particularly relevant.

A key point is that over-tough reviewing is often caused by reviewers themselves wanting to treat others just like they have been treated. There needs to be a way to break this cycle.

Some ideas in the presentation:

• All reviewers in a conference should be required to propose accepting a certain percentage of papers
• Reviewing where the reviewer can't see the author's name, but the author can see the reviewer
• No reviewing at all - let the market determine the best papers based on links and citations

However there is still a lot of 'bad science' that gets submitted to conferences. This could be solved by having reviews published along with each paper. Authors would want to avoid bad comments, and would also be able to respond to them and update their paper. See my earlier post on liquid publication.

Strikes over pensions: Surely there is a middle ground between defined benefit and defined contribution

A lot of labour disputes in Canada centre around the desire of some corporations to move away from defined-benefit (DB) pension plans, and towards defined-contribution (DC) plans.

DB plans pay an amount during retirement that can be calculated in advance, and depends on the employee's years of service and salary. The problem is, that corporations have a problem when the market turns down, since they rely on investment returns being at predicted levels (averaged over many years) in order to have enough money for the anticipated retirements. They rely on actuaries to tell them how much money they need to have in their fund; actuaries make actuarial assumptions. Some of these assumptions, such as the amount of time people will live, can be calculated with reasonable confidence using statistics and demographic data. However nobody can predict market performance, so corporations have a lot of difficulty in the years after each recession, since their plans go into 'deficit', meaning there is a need to contribute more for a while, and this affects the corporation's profitability, or even its solvency. Sometimes pension plans have a surplus (more than is expected to be needed), in which case the corporation can take a breather, making lower payments for a while, but this situation seems to rarely last.

The DC plans get rid of all this complexity by simply setting aside a defined amount of money each month the employee works. Employees then get a pension at retirement that depends on how well the investments have done over the years. Sometimes employees can get really great pensions with a DC plan, but after a recession is not a good time to retire, as the expected pension would be considerably lower. And that is the crux of the problem with DC plans: They transfer the uncertainty of retirement planning from employer to employee. If an employee has to retire, e.g. due to disability, when the plan has lost value, then the employee loses out. Similarly, this means that corporations are likely to lose a lot of good employees in times when the market is doing really well.

In the recent Air Canada settlement, we are told that current employees get to keep their DB plan, but new employees will get a plan that is to be determined by an arbitrator – most likely a DC plan. This will create different economic classes of employees, which it seems to me will be divisive.

It seems to me that negotiators could find ways to better split the burden: If a DB plan is in deficit, they could make an agreement that says employees who retire in future will have benefits reduced by X% while the deficit is above a certain level, with those benefits coming back to normal when the deficit goes down below a threshold, and with a commitment to increase the promised benefits above the current level by Y% if and when a surplus occurs at some point in the future.

It is unfortunate that few new DB plans are being created. It is good for society for pensioners to have come confidence in their retirement income. However I also think that splitting the burden of deficits between the employees and employer makes a lot of sense, as long as the benefits of surplus are also shared. In some sense this is blending the notions of DB and DC. I also don't mind if corporations put new employees on a plan that has a higher portion of DC than current employees. Just don't get rid of DB element entirely. Perhaps the plan could be 50% DB and 50% DC.

I have heard news reports that call DB plans 'generous'. This entirely misses the point. The two plan types simply put the risks on different parties; conceptually, the same amount of money is contributed and received, if the actuarial assumptions end up being accurate.

Some people also think that retirement savings should be left to individuals (e.g. using RRSP's in Canada). The trouble with that is that the risk is then transferred not just to the employees, but to society at large: A considerable portion of people won't have the self-discipline to save enough, so will end up needing various forms of taxpayer-funded assistance when they retire.

Surveys regarding CS and SE knowledge: Please participate

CIPS CBOK Survey: I am chair of the Body of Knowledge committee of CIPS; we are in the middle of conducting a survey to ask which topics should every computing and IT professional know, and at what level. This would include computer scientists, software engineers, business technology managers (BTM), project managers, data centre managers, database administrators, CIO's etc. If you are interested in participating in the CIPS CBOK survey, please click this link to access the survey. You may skip the questions on each page regarding references if you want to save time; if you do that, the remainder of the questions take about 35 minutes. The deadline is July 15th, 2011.

CS2013 Survey: Also the ACM and IEEE are looking to revise their recommendations on computer science education. They would be interested in having you fill out their survey of the characteristics that computer science graduates require.

SE2004 Review Task Force survey: Similarly, another ACM/IEEE group is reviewing the recommendations for software engineering education that were published in 2004. If you are interested in helping them assess the state of SE education, please fill out their survey.

Umple tutorial 3: Mixins

This is the third series of Umple tutorial posts on my blog. The first two were about 1) basic attributes and associations, and 2) state machines. In this post I introduce the concept of Umple mixins.

A mixin, a concept popularized in the Ada language, but now available also in languages such as Ruby, allows bits of code to be incorporated into other programming elements.

Let us imagine the following Umple code that you put in file1.ump:

// Contents of file1.ump

class X {

  singleton;

  lazy name;

}

class Y {

  Integer id;

}

This could be a small self-contained system with two unconnected classes. However let us imagine that you also created the following Umple code in file2.ump:

// Contents of file2.ump

use file1.ump;

class X {

   Date dueDate;

}

association {1 X -- * Y;}

If you compile file1.ump and file2.ump together, the result is that class X now contains two attributes, name and id.  Furthermore there is also an association between classes X and Y. The second declaration is class X is not redefining the class, it is adding to the class.

It wasn't necessary to put the above code snippets in separate files. Any time Umple encounters class X { ... } it will incrementally add the contents of the curly brackets to class X.

This can be useful to allow you to keep the 'pure' class diagram part of the model separate from the methods that belong in each class. In Umple you always have a choice: keep these elements together, or keep them separate.

Another use of mixins is to create various versions of a product line that have different features. So for example if we have the following code:

// Contents of file3.ump

class X {

   Time dueTime;

}

association {0..1 X -- 1..* Y;}

One product could be built using file1.ump + file2.ump and another, slightly different product could be built using file1.ump + file3.ump.

Incidentally. This is not the only way to create variants in Umple. Another concept is VML4Umple. I will provide a tutorial for that at a later date. For the eager, you can look at Jenya Levin's masters thesis.

Note that class X is declared to be a singleton. This means that code will be generated following the standard "Gang of Four" singleton pattern. Umple has several features that help implement patterns, and more are planned as Umple is developed further.

Apple's growing penchant for not supporting customers by changing services too radically

I admit I am an Apple fan, but they are gaining a reputation for abandoning customers too easily.

I am a Mobile Me user. I used to be a .mac user. They forced me off .mac some time ago and onto Mobile Me.

Then they forced me to update Mobile Me a few months ago, and this resulted in 'old' calendar data not being brought forward. They somehow thought that I would only care about 'future' and 'last month' calendar data. Wrong! I want to be able to look back at such things as past medical appointments, and other things I was doing on certain dates for many good reasons. I had backed up my calendars, but it was not easy to selectively pick the 'old' events and restore them; you have to manually edit the .ics files, which is something I would expect only a computer professional to be able to do.

Now we are told that Mobile Me will be replaced by iCloud, and that we will be given instructions to migrate later this year.  But they have left everybody in trepidation about what services will continue to exist in iCloud.

Surely they could have managed this in a better way. If I was their customer-relations department, the message would have been: 'Dear loyal Mobile Me subscriber, you have heard that Mobile Me will be replaced by iCloud. Don't worry; none of the functionality or data will be lost, this is primarily a rebranding exercise. When we change the technology and move to iCloud, your calendars and other synced data will continue to be synced, your galleries of photos will remain, your iDisk will still be accessible, and the Backup program will continue to make backups.'

Maybe it will be this way. If so they should have said it. If not they should be forthcoming and tell us which data will have to be moved somewhere else.

Maybe not a lot of customers are using Backup, iDisk or photo galleries so it is not cost-effective to maintain them. But at least they should move the existing data to some default place on iCloud. But my biggest plea is that they make calendar syncing transparent, without yet again forcing people to lose their historical data.

Usability improvements that I have requested in my blog: Some results

Here are updates on two usability advances that I had discussed in earlier blog posts:

The BBC has made one important improvement: Back in November I complained about automatically running videos launched from RSS feed items. Now the BBC RSS feed identifies such items as 'VIDEO'. However I would still prefer that videos don't play by default, so you can 'stack up' tabs with videos ready to play. Also, there is still too much category clutter on the BBC site; my complaints about that still stand.

Apple, in its announcement of iOS 5 has responded to my November suggestion (and that of countless others) that there be quick access to the camera app. The beta of iOS 5 now allows the camera to be accessed by double-clicking the home key when the device is locked. If the device is not locked, you would presumably press 'off', then double-click 'home'. In either case this can be done quickly and with gloved hands, which can be important. Taking pictures in the camera app becomes possible with the volume key, a good idea, despite overloading the functionality of that key. Thank heavens they have not yet removed the "home" button, as had been rumoured. The camera app in iOS

Usability Blooper: The operation can't be completed in Mac OS X copying

Here's a blooper in Mac OS X that has driven me crazy for several years.

The following 'copy failure' modal dialog can occur when trying to copy a large collection of files from one place to another using Finder. The dialog states: "The operation can't be completed because you don't have permission to access <file> - OK".

This breaks several key usability guidelines for the design of error messages:

1. It doesn't give the user sufficient information to readily solve the problem: The user might not recognize the file, or know in which folder it is stored. There should be an option to open the enclosing folder so the user can look at the file and fix the permissions.
2. It leaves the user half way through a incomplete task. There should be an option to continue to copy all other files. After the whole copying task is complete, information about which files were not copied should then be displayed, with the options to ignore each or to open its enclosing folder as in my point 1. Ideally, there also ought also be the option to revert to the state before the copying started (i.e. undo the entire copy process)

Here is a scenario in which it occurs for me: I like to back up my Mac OS X files in various different ways. Sometimes I copy my entire 'Library' folder to a backup location. The above error occurs in the middle of the process (the problem is with a particular Real Networks file that I actually don't care about). Interestingly enough, Spotlight hasn't even indexed this file (probably because of the same permission problems) so I am not able to locate it straightforwardly.




There is a workaround, but it is annoying. I look in the destination to see how much of the copying has been completed. Luckily copying proceeds alphabetically by file name, so I can then just initiate new copies to finish off copying the top level folder, and each level of subfolder, skipping the problem file. This workaround is not, however, something I would want to inflict on the non-expert. And it becomes very tricky when more than one file fails to be copied.

Other workarounds are to use tools other than the Finder for backup, and indeed I do that too. However, my inclination is to back up in a variety of ways in case one method fails.

These workarounds won't work in some contexts. For example,

• If you have command-clicked on an arbitrary set of files and folders and have dragged them all to a destination, when the copying task fails you might have already lost track of  which items you carefully selected to copy.
• If you initiate many copy operations, having them all proceed concurrently, when one or more of them fail in the above manner, you may be completely at a loss to know what parts of your work have been completed, and which have not been completed. Ideally then, the 'copy failure' dialog should also give you information about the complete set of files being copied, and allow you to resume or complete the process after fixing the permissions.

Phones ringing in events annoy everyone: Suggestion for flash reminders

At conferences I attend, there will always be somebody whose phone rings loudly. The same happens in meetings and in class. It has probably happened to you. It happens to me every few months. Very embarrassing.

Sometimes organizers remind people, don't forget to turn off your phones. But how many times have you reached for your phone when somebody else's phone rings, just to check if yours is off?

Here's the idea: All people need is a quick 'flash' reminder to check their phone. It need only take a couple of seconds. When introducing a session, the organizer could flash a slide for a few seconds with: "remember to check your phone is on vibrate!" The speaker doesn't have to say anything. Everybody knows what to do; hardly anybody wants their phone to ring.

Of course the better answer is to have calendar applications build in vibrate mode automatically, as I discussed in my previous blog post, but we may have to wait a while for that.

Live Blog from ICSE: Peer review in open source projects

This is my final live blog from ICSE. See recent posts for others.

This is a discussion of the presentation of Understanding Broadcast Based Peer Review in Open Source Software Projects by Peter C Rigby and Margaret-Anne Storey 

 

The presenter, Peter Rigby discussed 460 instances of peer review in open source projects, including interviews with 9top reviewers.

They examined scalability of the peer review process, techniques for finding patches to review, motivation, patch structure and norms, as well as the effect of too many opinions and ignored reviews.

The presenter discussed the fact that large projects can receive hundreds of peer review messages a day. One lead developer received over 2000 emails in a day, only two of which were personal emails.

Motivations include intrinsic interest and a sense of responsibility. People are invested in a particular code base; if they don't review changes, then the code quality will deteriorate.

They filtered messages from people and subsystems. This helps developers to not feel overwhelmed. However many messages are posted to multiple lists.

One developer said that when reviewing potential patches, "a good and descriptive subject line draws immediate attention". A change log in the message gives conceptual understanding.

The developer snips out excess detail in replies; this seems to help considerably to streamline the process. My opinion is that this should be the case in general email etiquette.

 

 Another practice is that in a back-and-forth email conversation, people tend to reply to individuals as well as the list, and gradually more cc recipients get added.

Some review discussions are purely technical (whether it works); others focus on scope, politics, necessity etc. There are thus often too many trivial opinions (Parkinson's law of triviality); people with trivial information tend to drown out more serious discussion. They measured outsider involvement and influence.

One problem they examined is that there are too few developers in open source development. Lack of time tends to result in postponement of reviewing; this puts the onus on the author of the patch to ensure it is eventually included.

Takeaway mssages:

• Reviewers are invested in doing reviews
• They tend to postpone rather than rush
• Asynchronous review processes facilitate discussion
• Politicized discussion is infrequent
• Scaling in large systems can occur through multiple mailing lists for different subssytems (other methods are discussed in the paper)

The presenter discussed the threats to the credibility of his work, and how these are mitigated. For example, the data is public.

An audience member asked about how this compares to traditional inspection. The presenter said there are many similarities and that the open source process seems to work as well as inspections. Key is early reviewing, and reviewing in small chunks. An audience asked about 'review then commit' and 'commit then review'; the presenter said that the processes were mostly the same. Another audience member asked a related question about status-quo bias: He said that patches that are committed first have a higher necessity for review.

An audience said that email for this is terrible and there should be better tools. The presenter said that the flexibility of email is underestimated. Several projects, however, have started to use tools, with more traceability. Forcing people to use tools has some drawbacks though. "You have to be careful to say that email is bad ... if you understand the norms of the community then it is really very efficient".