New version of the COMPADRE & COMADRE portal is coming up

Just like winter, a new version of COMPADRE & COMADRE is coming too. This work has been going on for a couple of years now, but we are starting to make some tangible progress and we are excited to share where the project stands at the moment. We assume that you are familiar with the COMPADRE Plant Matrix Database and the COMADRE Animal Matrix Database in their current format. Navigating to http://www.compadre-db.org/ presents you with information about the databases, the team behind them and a direct download link to a single R-data file for either COMPADRE or COMADRE. There is also a table for the ‘waiting list’ of species that are soon to be added to the sister databases, but the current format does not offer searchable menus to explore the data before downloading them.

Screenshot 2017-02-18 16.33.10.png

The current COMPADRE & COMADRE data portal

Behind the scenes, we have been building a new way of presenting and accessing COMPADRE and COMADRE, whereby you’ll be able to explore the data through the data portal. Here is a quick rundown on some of the features:

1. Query, then download:

You will be able to query the database so that before you download the data, you can work out whether the data are useful to you. We will have developed tables to search by species and publication and will be extending this to advanced filtering through all metadata so that the user gets just what s/he needs.

null

Table for searching by species, able to filter by growth type and search terms

2. Explore demographic info across the tree of animals and plants:

Come and explore among multiple kingdoms, orders and families, worm your way through the tree of life and see what species are related to one another, and where we have (and not) demographic data. Currently, we are finalising a click-through explorer gathering a short summary from Wikipedia, but there are plans to integrate the incredible OneZoom taxonomic explorer into the user interface.

null

Click-through taxonomic explorer

3. Educational pages for all ages, sizes and stages:

We are dedicated to teaching all st/ages about the wonders of demography and connecting maths with the living environment.

For every species page, we are developing a georeferenced interface to show the locations of the populations of the study species. We also include species occurrence data from GBIF to give a general view of the distribution of the species. In addition, we provide relevant metadata about the species to be able to contextualise the matrix model.

null

Map showing the locations of studied populations, the small yellow squares are GBIF occurrence data.

4. More hierarchy, but easier to manage and navigate:

The database’s hierarchical structure is being incorporated into how we present the data. Beneath the species information, each population is listed with summary statistics about how many matrices it contains, its ecoregion and for how long it was studied. Expanding this reveals each of the matrices that were produced from studying this population. Expanding the matrices shows the matrix model in full, with clickable tabs to select whether you wish to view the full matrix, fecundity matrix, survival matrix or clonal matrix.

null

The contents of a single population with multiple matrices. One matrix has been expanded to show the values in the matrix.

5. Giving credit where it is due:

At the bottom of the page, you can see details about the publications from which the MPMs were taken from, and directly link to the original journal article via DOI. We hope that the users of COMPADRE and COMADRE will continue to credit the tremendous work done by the original sources of the demographic information.

null

A list of all publications contributing data for Cirsium pitcheri

Intended users of the database

We hope that this project will make the database more accessible to a wider range of users. These improvements to the COMPADRE & COMPADRE data portal resource are aimed at:

  • Students and entry-level population modellers who want to explore matrix populations models (MPMs).
  • Scientists who aren’t primarily studying population dynamics or demography but are working with a study species and would like to know more about their study species.
  • Comparative demographers who are looking to use a large chunk of the data available but might want to examine a particular matrix or species as to whether they will include it in this analysis. The user interface will provide an easy way of viewing the data and linking to the literature source.
  • Ecologists who are looking to construct MPMs but would first look at previously constructed matrices for similar species or study motivations to help guide their study design.
  • School groups who are interested in life-cycles and we want to hook them into a life long obsession of COMPADRE and COMADRE.
  • Other managers of ecological databases that might want to share data and collaborate.

The Compadrino Zone

The compadrinos are the real workforce behind the demographic data. These are a group of MSc, BSc, PhD and postdocs students supervised by the core committee. They carefully inspect published papers, extract relevant information, email authors for extra information and clarification, and help implement the error-checking before the data go live on the open-access portal. The total COMPADRE & COMADRE workload is split amongst nodes located at different institutions across the globe. This poses a logistical challenge as to how to manage the data which have been currently been tackled with cumbersome spreadsheets.

null

A form for entering information about a publication, with an auto-fill from DOI javascript button.

The work we are doing will also improve the behind-the-scenes data digitisation workflow. Compadrinos will now have access to easy-to-use friendly web forms and be able to use the user interface to help identify errors. We will also be able to track changes to the database over time and make it a more streamlined, enjoyable, and less error-prone experience. That’s all well and good but as a user of the database, why should you care about this? These improvements should result in an increased flow of data from publications to database, leading to a greater quantity of data, expanding the taxonomic breadth of the database.

We hope that once the new portal has been instituted, we will move onto a new functionality: providing users with the ability to upload their data directly. The hope is for COMPADRE & COMADRE to become the DRYAD of stage-structured demography. Of course, all data that th users will upload will still need careful curation, and every datum will continue to go through our error-checking and data standardisation protocols.

Technical details

Finally, a few details of the project:

  • The newest iteration of the COMPADRE and COMADRE database will be hosted on a server at the University of Exeter and mirrored among all nodes of the COMPADRE/COMADRE digitalization network.
  • The web application is built on a python-based framework called Flask and the database is being migrated from the original R-data object into a MariaDB database.
  • Following the open-access philosophy behind COMPADRE & COMADRE, the present also is an open source project, and the source code is available on our Github repository: https://github.com/Spandex-at-Exeter/demography_database

Timeline

Now we’ve whetted your appetite, you’re probably wanting to know how long before you can delve into this exciting new resource? Our first milestone is to finalise the process by which compadrinos input and process data to make sure everything functions as expected and the data are flowing from publication to the database without a hiccup. This is our focus for the next couple of months. Following that, we will change focus to improving the user interface and data download tools for end users, hoping to launch late 2017.

Simon (@simon_rolph) and Danny (@dannylbuss)

Image uploaded from iOS.jpg
Danny and Simon hard at work

Follow COMPADRE (@compadreDB) and COMADRE (@comadreDB)

Have questions about COMPADRE & COMADRE, or would like to send us your data? Please contact us at compadrecontact@gmail.com

About the writers of this blog: Danny Buss is a Computing Development Officer (CDO) at the University of Exeter in Dave Hodgson’s research group, and Simon Rolph is PhD student at the University of Sheffield in Rob Salguero-Gómez’s research group. A few weeks, ago we convened in Sheffield to work together on an important upgrade of the COMPADRE data portal. We must also recognise Francesca Sargent’s huge contribution towards this project, previously a CDO at the University of Exeter.

Advertisements

A signature of life history in the stochastic dynamics of structured populations

Different species have different average life histories. Such a variation is apparent from key species characteristics like longevity, age of maturity, or the number of offspring produced per clutch. Matrix population models such as those in the COMADRE Animal Matrix Database allow us to calculate and explore the diversity of such characteristics across species, and understand how they are connected.

In a recent study published in Oikos, my colleague and I examined another key life history characteristic: demographic variance.  This trait provides a measure of the total amount of stochasticity in a given life history arising from inherent randomness in survival, offspring production, and other individual-level demographic processes. Typically, species with a high demographic variance are short-lived and can produce many offspring per reproductive event, such as mice. Long-lived species, who typically produce only one offspring at a time, such as elephants, tend to have a much lower demographic variance.

Using matrix population models from 24 mammal species from COMADRE, we calculated their demographic variance. We first considered how demographic variance is related to generation time (see Figure 1).  As expected, there was a strong correlation with generation time, where species whose populations take longer to renew its individuals had a much lower demographic variance. 

screen-shot-2017-03-05-at-05-01-38Figure 1. Demographic variance plotted against generation time, calculated using demographic information for mammals and birds from the COMADRE Animal Matrix Database.

We then considered the temporal correlation in the stochastic population dynamics, which arise because of short-term fluctuations in the demographic structure.  For each matrix model we estimated the autocorrelation function, which describes the degree to which two population growth increments tend to be similar for different time steps. This function is different for each model, and represents a signature of the life history and demographic stochasticity (see examples in Figure 2).  A main result from our study is that the sum of these correlations over time lags describes the impact of population structure on the demographic variance.

Screen Shot 2017-03-07 at 09.55.13

Figure 2. Autocorrelation functions for two of the mammal population models used in the study – a signature of the life history.  The sum of this function describes the impact of the demographic structure on demographic variance (which can be positive or negative, depending on the structure).

Thanks to COMADRE, we were able to demonstrate this result using the matrix models of different mammals.  Future research may also consider applications to other taxa (e.g. birds, as in the Figure 1) and other kinds of life histories.

Yngvild Vindenes

Science committee member of COMPADRE & COMADRE

Reference:

Vindenes, Y. and Engen, S. In press. Demographic stochasticity and temporal autocorrelation in the dynamics of structured populations. Oikos DOI: 10.111/0ik.03858