The end of the impact factor

Recently, both the American Society for Cell Biology (ASCB) and the journal Science both publicly proclaimed that the journal impact factor (IF) was bad for science. The ASCB statement argues that IFs limit meaningful assessment of scientific impact for both published articles and especially other scientific products. The Science statement goes further, and claims that assessments based on IFs lead researchers to alter research trajectories and try to game the system rather than focussing on the important questions that need answering.


Impact factors: tale of the tail
The impact factor was created by Thomson Reuters and is simply the number of citations a journal has received in the the previous two years, divided by the number of articles published over that time span. Thus it is a snapshot of a particular type of 'impact'. There are technical problems with this metric -for example, that citations accumulate at different rates across different subdisciplines. More importantly, and what all publishers and editors know, is that IFs generally rise and fall with the extreme tail of the distribution of the number of citations. For a smaller journal, it just takes one heavily cited paper to make the IF jump up. For example if a journal publishes one paper that accumulates 300 citations and it published just 300 articles over the two years, then its IF can jump up by 1, which can alter the optics. In ecology and evolution, IFs greater than 5 are usually are viewed as top journals.

Regardless of these issues, the main concern expressed by ACSB and Science is that a journal-level metric should not be used to assess an individual researcher's impact. Should a researcher publishing in a high IF journal be rewarded (promotion, raise, grant funded, etc.) if their paper is never cited? What about their colleague who publishes in the lower IF journal, but accrues a high number of citations?

Given that rewards are, in part, based on the journals we publish in, researchers try to game the system by writing articles for certain journals and journals try to attract papers that will accrue citations quickly. Journals with increasing IFs usually see large increases in the number of submissions, as researchers are desperate to have high IF papers on their CVs. Some researchers send papers to journals in the order of their IFs without regard for the actual fit of the paper to the journal. This results in an overloaded peer-review system.

Rise of the altmetric
Alternative metrics (altmetrics) movement means to replace journal and article assessment from one based on journal citation metrics to a composite of measures that include page views, downloads, citations, discussions on social media and blogs, and mainstream media stories. Altmetrics attempts to capture a more holistic picture of the impact of an article. Below is a screenshot from a PLoS ONE paper, showing an example of altmetrics:

By making such information available, the impact of an individual article is not the journal IF anymore, but rather how the article actually performs. Altmetrics are particularly important for subdisciplines where maximal impact is beyond the ivory towers of academia. For example, the journal I am an Editor for, the Journal of Applied Ecology, tries to reach out to practitioners, managers and policy makers. If an article is taken up by these groups, they do not return citations, but they do share and discuss these papers. Accounting for this type of impact has been an important issue for us. In fact, even though our IF may be equivalent to other, non-applied journals, our articles are viewed and downloaded at a much higher rate.

The future
Soon, how articles and journals are assessed for impact will be very different. Organizations such as Altmetric have developed new scoring systems that take into account all the different types of impact. Further, publishers have been experimenting with altmetrics and future online articles will be intimately linked to how they are being used (e.g., seeing tweets when viewing the article).

Once the culture shifts to one that bases assessment on individual article performance, where you publish should become less important, and journals can feel free to focus on an identity that is based on content and not citations. National systems that currently hire, fund and promote faculty based on the journals they publish in, need to carefully rethink their assessment schemes.

May 21st, 2013 Addendum:

You can sign the declaration against Impact Factors by clicking on the logo below:

Holding fast to a good(?) idea

One of my favourite lists on the internet is tucked away in the credits for the PHYLIP software. PHYLIP was authored by Joe Felsenstein, a professor at the University of Washington and expert on methods for phylogenetic inference. PHYLIP is a free package of programs for inferring phylogenies, and probably the first and oldest widely-distributed phylogenetic program. Programs like PHYLIP made phylogenetic approaches easily accessible to ecologists and evolutionary biologists. Apparently it took years to get from the idea for PHYLIP to funding, and Felsenstein memorializes this with his “No thanks to” list (below). The list includes reviewers and panels from the US Dept of Energy, NSF, and NIH that turned down his proposals and made comments like "The work has the potential to define the field for many years to come.... All agreed that the proposal is somewhat vague. There was also some concern that the proposed work is too ambitious.”

(Click to enlarge)

There are obvious responses to this list, mostly relating to the short-sightedness of funding agencies, meaningless requirements for ‘broader impacts’, the fact that proposals might be improved through the process of multiple failed applications, and of course the benefit of being long-established and respected when posting such lists on your website. But what I always wonder about is how long do you hold on to an idea, a proposal, or a manuscript that it is repeatedly rejected, before you give up on it? 

This question is interesting to me for a couple of reasons. Firstly, because personality is so intertwined with confidence about an idea’s success. We all know people who would argue that all their ideas are Nature-worthy, criticism be damned. Other people need to be convinced of the merit of their own ideas. Obviously past success probably helps with judgment – having experience in identifying good ideas builds confidence in your ability to do so again. But what is the line between self-confidence and self-delusion? Secondly, it is a reminder that lots of good ideas and good papers were rejected many times. In any case, I am curious whether people tend give up on an idea simply because they became discouraged at the prospects of getting it published, or because they lost faith in the idea, or a combination of both. 

Love the lab you’re with or find the lab you love? Being happy in grad school.

Every grad student is unhappy at some point; existential angst is basically required, hence the success of PhD comics. A surprisingly common reason for grad school unhappiness results when students feel they have diverged from the path they want to be on - that they are somehow in the wrong lab, learning the wrong thing, or working with the wrong person. That they dislike their research. Some people might argue that few people start in their dream job, but grad school is more like an apprenticeship than a 9-5 job: a place to obtain skills and experiences rather than a source of income.

Every unhappy student is different in their own way, but there are a few predictable causes. The path between undergrad and grad student is highly stochastic. Most undergraduates make choices about grad school while under-informed about their options and unclear on where their interests truly lay. Choosing a lab for a PhD or Masters is a huge commitment for an undergrad who has had comparatively limited interactions with ecological research. Academic labs tend to be so specialized that even if an undergrad has had the opportunity and motivation to interact with a number of labs as a student, they have experienced only a tiny fraction of the areas available for study. Students in schools with general biology programs, rather than specialized EEB departments, may be more limited again in the ecological experiences they can have. I can’t help but think that few undergrads are really equipped to make a definite, informed decision about what they want to spend the next 5 years (or more) of their lives doing. Even if they are, a successful graduate student should grow as a researcher and their interests will naturally expand or shift. Expanding interests and changing foci are part of a successful graduate experience, but what initially felt like a good fit may suddenly feel less comfortable. Students may also end up in uncomfortable fits simply because their choices for grad school were limited by geographical constraints or the availability of funded positions, causing them to compromise on their interests.

My own experience moving from undergrad to PhD student was pretty much in line with this. I knew I wanted to go to grad school and I felt reasonably prepared – I had good grades, three years of research experience in a lab, I researched and contacted a few potential supervisors – but I hadn’t specialized in ecology and didn’t exactly know what my interests were. It took more than a year as a PhD student, reading deep into the literature and taking classes, to realize what I really was interested in was completely different than what I was supposed to be doing. This was accompanied by a period of unhappiness and confusion – I had apparently gotten what I wanted (grad school, funding, etc), but it wasn’t what I wanted after all. No one prepared me for this possibility. Eventually, but with some hassle, I changed labs and was lucky to have the opportunity to get the skills I was really wanted.

I don’t think this outcome is anyone’s “fault”. I think most departments and many supervisors are sympathetic to these sort of graduate student issues. Formal advising of undergraduates at the department level, in addition to the usual informal advising that grad students and advisors provide, should be focused on guiding students in determining what they want from grad school (or if they really want grad school at all!), how to identify areas of interest and programs/supervisors that would suit their interests. In particular, they need to be empowered in how to contact potential supervisors and how to discuss the supervisor’s expectations and approach, what changing interests mean on a laboratory and department level, and what resources are available for student who may wish to obtain particular skills not available in the lab. Supervisors benefit too when their students are informed and more likely to be happy and engaged.

The lab rotation system, which some departments have, also seems like a good way to expose students to their options (although I have no personal experience with it). In addition, when grad student funding comes through the department, rather than from individual supervisors, students can change labs with less difficulty. Some supervisors have very relaxed approaches to grad student projects, allowing students to explore their interests well outside of the lab’s particular approach. But other supervisors (or funding sources) are very much organized around a particular project, making it difficult for students to do anything but the project they were hired to work on.

So what is a student who realizes they want to be working with a different system, approach, sub-discipline, or supervisor, supposed to do? How much does unhappiness really matter in the long run? This depends a lot on what a student wants to get out of grad school and what they need to achieve it. One thing students need to do is elucidate what they hope to achieve as a grad student. Though a student may ultimately be unsatisfied with some aspects of their position, they may be able to gain the experiences they want from grad school regardless. There are many tangible and intangible skills students learn in grad school. Students may decide that they want to obtain particular quantitative skills (statistics, ArcGIS, coding and modeling experience, etc, etc) that they want for the job market; if these aren’t available, change may be necessary. On the other hand, even if a student is less interested in the particular system they are working in, it may be possible to obtain experimental and technical skills that are transferrable elsewhere. If students wish to remain in academia, but realize they are interested in a different subdiscipline than where they work, one consideration is whether it will be easier to make the shift now compared when finding a postdoc and attempting to convince a potential employer that their knowledge is transferrable. This is a difficult question – having read a large number of Ecolog post-doctoral position ads, it seems that the request for system-specific experience occurs in about 50% of ads. The need to have a particular skill set (say, Python and R, or experimental design) tends to be mentioned in every ad. So if you want to go from a protist-microcosm PhD to a postdoc in kangaroo ecology, it seems difficult to predict how well your experimental design skills will trump your lack of understanding of Australian ecosystems.

Of course, there is no one-size-fits-all answer about what to do. Sometimes, unhappiness will pass, sometimes it won’t. Students need to be proactive above all. The truth is that sometimes it is better to be willing to drop out, to change labs, or take other drastic action. Students commonly fall victim to the sunk-cost fallacy, the idea that they’ve spent 2 years on this degree, so they might as well not “waste” it. Sometimes it is worth sticking it out, but there should be no stigma in deciding not to.

Testing the utility of trait databases

Cordlandwehr, Verena, Meredith, Rebecca L., Ozinga, Wim A., Bekker, Renée M., van Groenendael, Jan M., Bakker, Jan P. 2013. Do plant traits retrieved from a database accurately predict on-site measurements? Journal of Ecology. 101:1365-2745.

We are increasingly moving towards data-sharing and the development of online databases in ecology. Any scientist today can access trait data for thousands of species, global range maps, gene sequences, population time series, or fossil measurements. Regardless of arguments for or against, the fact that massive amounts of ecological data are widely available is changing how research is done.

For example, global trait databases (TRY is probably best known) allow researchers to explore trait-based measures in communities, habitats, or ecosystems without requiring that the researchers have actually measured the traits of interest in the field. And while few researchers would suggest that this is superior to making the measurements in situ, the reality is that there are many situations where trait data might be required without the researcher being able to make them. In these cases, online databases are like a one-stop shop for data. But despite the increasing frequency of citations for trait databases, until now there has been little attempt to quantify how well database values act as proxies for observed trait values. How much should we be relying on these databases?

There are many well-recorded reasons why an average trait value might differ from an individual value: intraspecific differences result from plasticity, genotype differences, and age or stage differences, all of which may vary meaningfully between habitats. How much this variation actually matters to trait-based questions is still up for debate, but clearly affects the value of such databases.  To look at this question, Cordlandwehr et al. (2013) examined how average trait values calculated with values from a North-west European trait database (LEDA) corresponded with average trait values calculated using in situ measurements. Average trait values were calculated across several spatial scales and habitat types. The authors looked plant communities growing in 70 2m x 2m plots in the Netherlands, divided between wet meadow and salt marsh habitats. In each community, they measure three very common plant traits: canopy height (CH), leaf dry matter content (LDMC), and specific leaf area (SLA).

In situ measurements were made such that the trait value for a given plot was the median value of all individuals measured; for each habitat it was the the median value of all individuals measured in the habitat. The authors calculated the average trait values (weighted by species abundance) across all species for each community (2m x 2m plot) and each habitat (wet meadow vs. salt marsh). They then compared the community or habitat average as calculated using the in situ values and the regional database values. 

From Cordlandwehr et al. 2013. Habitat-level traits at site scale plotted against habitat-level traits calculated using trait values retrieved from a database. 

The authors found the correspondence between average trait values measured using in situ or database values varied with the scale of aggregation, the type of trait and the particular habitat. For example, leaf dry matter content varied very little but SLA was variable. The mesic habitat (wet meadow) was easier to predict from database values than the salt marsh habitat, probably because salt marshes are stressful environments likely to impose a strong environmental filter on individuals, so that trait values are biased. While true that rank differences in species trait values tended to be maintained regardless of the source of data, intraspecific variation was high enough to lead to over- or under-prediction when database values were relied on. Most importantly, spatial scale mattered a lot. In general, database values at the habitat-scale were reasonable predictors of observed traits. However, the authors strongly cautioned against scaling such database values to the community level or indeed using averaged values of any type at that scale: “From the poor correspondence of community-level traits with respect to within-community trait variability, we conclude that neither average trait values of species measured at the site scale nor those retrieved from a database can be used to study processes operating at the plot scale, such as niche partitioning and competitive exclusion. For these questions, it is strongly recommended to rigorously sample individual plants at the plot scale to calculate functional traits per species and community.” 

There are two conclusions I take from this. First, that the correlation between sampling effort and payoff is still (as usual) high. It may be easier to get traits from a database, but it is not usually better. The second is that studies like this allow us to find a middle ground between unquestioning acceptance or automatic criticism of trait databases: they help scientists develop a nuanced view that acknowledges both strengths and weaknesses. And that's a valuable contribution for a study to make.