In the absence of a contextslide, CSIRO has a data repository ( It s

Transcription

1 In the absence of a contextslide, CSIRO has a data repository ( It s designed to be self-serve, so it s technically possible for a CSIRO researcher to deposit data, publish it, and then obtain a DOI all without talking to anybody in the library or IT. This system has been minting DOIs for a little over a year now, so following on from the other presentations, this is a case study of trying to actually locate these DOIs in the wild, so to speak. By this I mean using existing metrics and search tools to identify citations of these DOIs. 1

2 TheData Citation Index does not currently have data from Research Data Australia, but the ingest is scheduled for later this year. In principle, being able to search for DOIs, or truncated DOIs, will make this a valuable resource for citation metrics, as the entire purpose of the database is to provide metrics for data citations. This is a subscription product, and will not be accessible to the general public. Screenshot 1: This is what a search for the three ANDS DOI prefixes would look like, although it currently produces no results. Screenshot 2: This shows the number of results for the only current way to search for CSIRO datasets. It s likely that none of these results would appear as records in the Data Access Portal. 2

3 Scopusallows you to search the references field. Assuming that journal papers are citing DOIs, searching for a DOI in this field should, in principle, allow you to retrieve a list of papers that cite that DOI. The search syntax of Scopus allows you to be very precise about what you are searching for, including punctuation marks, which allows you to search for DOIs quite easily. There are some limiting factors to this, in that not all journals use a citation style that will include a DOI, and there is also the fact that not all authors will cite a dataset s DOI in the references section. Scopus is also a subscription product and is not publicly accessible Screenshot 1: This shows a naïve attempt to search for some DOI prefixes. The syntax is incorrect, thus the reasonable number of results is mostly false positives. Screenshot 2: This search uses the correct syntax. Here we see Scopus picks up 2 ANDS DOIs in the references field. While this is an underwhelming result right now, it demonstrates that the technique works, these two results genuinely do cite DOIs with the ANDS prefixes. 3

4 Imust confess to not being an expert with the search syntax of Google, but I m pretty sure that Google Scholar doesn t let you have true exact phrase searches. To be precise, punctuation in phrase searching is ignored. Thus a search for /08 (the CSIRO DOI prefix) will actually conduct a search like Worse, it seems that it may be searching in a similar way to 10 OR 4225 OR 08, which naturally retrieves irrelevant results. That said, the structure of a DOI is unique enough that a search for brings up a small enough set of results that it s possible to manually inspect all of them. This shows the advantage of searching across a dataset that is already limited to scholarly publications, rather than a wider internet search. 4

5 Conversely,here is the result from searching Google generally. Even refining the search to include DOI still brings up more results than is convenient to manually inspect. In this case you are forced to rely on the ranking algorithm. 5

6 Microsoft Academic Search allows you to search for DOIs, but the syntaxdemands complete DOIs, not prefixes or truncations. Nonetheless, searching for a DOI prefix generally brings up a manageable set of results. Screenshot 1: While there are some irrelevant results (e.g. Most results are there because they contain 10 ) we see that the first result is directly relevant. Screenshot 2: Exact phrase searching will probably produce some false results, but so far the example shown only produces one positive. Note that it is not finding either of the papers that were retrieved by Scopus. This is possibly a limitation of the content indexed, rather than the actual search features, and the website clearly states the product is still in beta. 6

7 For those new to the term, altmetrics are differentiated from conventional bibliometrics by the types of use and citation that a research product might have. So for example bibliometrics will examine things like how many times a paper has been cited and how many citations various papers in a journal get. Altmetrics will instead look at things like how many references to a paper have appeared in social media networks (e.g. Twitter, Facebook), how many times a paper has been viewed online, or downloaded, mentioned by the media or blogs. Screenshot1: This graph indicates the word altmetrics has a short history of being searched for in Google. 7

8 Per article it's possible to dig around various sources and find some altmetric information manually. Some sources aren't openly available, however. For example, say I wanted to find out how many times people have used bit.lyto link to a particular DOI, I might have to negotiate access to their data. On the other hand, if I set up a program to scan Twitter and follow bit.lylinks to see where they point, I could then look at the data where the final links use DOIs. There are a number of services available that try to take care of these details for you. At the moment they're still working on the details of actually gathering data, and what sort of data is relevant. This means they aren't always directly comparable, and the results have to be taken within their context. For example, comparing the number of times two articles get mentioned on Twitter where one article is from 2008 and another is from 2010 can't be a fair comparison due to the rapid growth of Twitter. Screenshot 1: The PLOS website shows some altmetrics about articles. Screenshot 2: Finer detail. We see that the views of the article vastly outweighs the citations. The ratio of views to citations can vary between fields, or even between papers in the same field with the same number of citations. 8

9 Impact Story ( is a free service that allows researcher to create profiles. The bibliography created will show details about citations and altmetrics in generalised terms, with options to see the numbers behind the generalisations. It distinguishes different publication types, e.g. papers, datasets, slides, software, websites. Datasets from Dryad and figshareare supported and will be displayed with metrics, but tracking DOIs from DataCitehasn't been implemented yet. This means that untracked datasets will appear in a section titled items with no metrics found. Tracking DOIs from DataCiteis planned, but currently there is no specific time frame for implementation. Screenshot 1: a CSIRO dataset appearing in the items with no metrics found section of Mark Pearce s profile ( Screenshot 2: Dryad and figsharedatasets as listed in Heather Piwowar s profile ( 9

10 Altmetric.comis a commercial service that offers a subset of its data for free. It calculates their arbitrarily defined score, where different metrics are weighted according to their own criteria. There is also information regarding the variety and proportion of the contributing sources. The score isnotnormalised, so difficult to use for comparisons between fields. Theyalso stress the limitations in comparing papers from different time periods, due to the rapid rise in various social media tools. As an easily implemented example, they provide embeddable badges displaying the altmetric score and donut, which indicates the diversity of sources. Full access to the Altmetricdataset requires negotiation with the organisation, but limited access to the API is available for assessment. An example search for one of the ANDS DOI prefixes would be: Screenshot 1 (Top Right): an embeddable badge for a CSIRO dataset ( The one Twitter result is from a test done by Sue Cook ( has produced the result at Screenshot 2 (Bottom Right):This is a section of a mocked up a page that has an Altmetic.com badge for every CSIROdataset that has a DOI -the only one that has any data tracked so far is the one we tested inscreenshot 1. Note: Sam Searleat Griffith Universitytweeted about datasets (e.g. but thiswas not tracked by Altmetric ( Interestingly, the paper that was linked to in the same Tweet was tracked ( Screenshot 3 (Bottom Left): An example of an embedded badge as viewable in Scopus (a subscription service) further details for this are available at Altmetric.com 10

11 PlumAnalytics is a commercial product that requires subscription access ( Trial periods can also be arranged. It appears to have the widest coverage of sources for altmetric data. Therewere some issues with how datasets were being displayed. It s not certain whether this is due to the software or they way it is being used, but in this example individual tables are being listed with nondescript labels. 11

12 Dryad download numbers are displayed for each dataset. Screenshot 1 (Top): browse by popular data. Screenshot 2 (Bottom): the collection record ( There are links to the original paper that concerns this data. 12

13 This refers to the previous slide. The journal paper to which the dataset relates only lists the data citation in the header of the paper ( Since the datacitation does not appear in the references list, any citation tracking service that indexes the bibliographies of papers may easily not count this as a citation. Thisraises the question, what should the criteria be for counting a data citation as a number? Does it have to occur in the paper s references list, or can it be anywhere, like the acknowledgements? 13

14 Project Columbus is an ANDS sponsored project to explore the conditions for successful data citation and citation indexing; it involves providing specific examples of data citations to organisations developing data citation tracking tools (Thomson Reuters (Data Citation Index) and Elsevier (Scopus)) so that they can then work with publishers to improve the capture of citations to data, which would then become the citation counts in those tracking tools. In CSIRO we are able to cross-reference a list of dataset authors with journal paper authors from our publication system, and identify authors who are still in the process of submitting a paper. 14

15 15

16 Figure 1: Thisgraph represents the total number of DOIs CSIRO has had minted at various points in time. The sharp rise after May 2013 was from ATNF updating existing collections in order to get DOIs minted (many of these collections had been published for more than a year at that point). These form the majority of CSIROs data DOIs, and it seems that these have been minted too recently for citations to be seen in the wild. 16

17 Additionally,the ATNF data specifies conditions for acknowledgment of the data. A link to this acknowledgment statement is clearly displayed to people using the ATNF specific search feature of the Data Access Portal. 17

18 This statement (found here: is quite general, and doesn t actually require the user to mention the attributionstatement for specific collections. So if we want to track the citations of this data, we can t rely on the DOI appearing in a paper s list of references. In fact, we need to search the full text of a paper for the attribution statement. Here we have two GoogleScholar searches. Google Scholar was used under the assumption that the full text of papers is indexed and searchable. Screenshot 1 (Left): The first is for the CSIRO DOI prefix. Screenshot 2 (Right): The second is for the attribution statement. Both have a small set of results. Also, both sets of results are different, meaning none of the results for the attribution statement search also appear in the CSIRO DOI prefix. Due to the relatively recent minting of the CSIRO DOIs, it s probably too early to read anything significant into these results. 18

19 Since there are limited results in resourcesaimed at scholarly publications, it here is a comparison Google search for the CSIRO data DOI prefix, excluding the domains csiro.au and ands.org.au. Screenshot 1 (Right): The set of results for a DOI prefix is small enough to go through manually ( If you look through the results you will find irrelevant results, but you will also see some items that have not come up in scholarly citations (e.g. the Zebedee mapping device, which has received news coverage). This shows some of the public interest in particular works. Screenshot 2 (Bottom Left): As a comparison, the search results for the ANDS Handle prefix (which is not specific to CSIRO) retrieves a larger set of results. There remains a good proportion of relevant results, but since Google ignores punctuation this is essentially searching for three numbers, which leads to a fair amount of irrelevant material. It s possible that the structure of a DOI makes it a more unique search string than a handle, which aids in identifying relevant results. 19

20 In the absence of tracking citation metrics throughdois, other information can be obtained from the distribution system itself. Some activity in the Data Access Portal is tracked by Google Analytics. Figure 1: an example of download behavioursplit into categories. Ignoring the spikes in activity from the ATNF data (which dwarfs what is shown here), this graph shows clear periods of increased use of the Web Services Interface (or REST interface). This activity is likely to have been conducted via programs or scripts that have been developed to obtain the data automatically. In June there is a clear period of activity for nonastronomy data. In August and September we see an increase in use of the REST interface for astronomy data. This graph uses data exported from Google Analytics, and is not part of the Google Analytics web interface. 20

21 For comparison, thisis a screenshot of download activity for ATNF data, as displayed in the Google Analytics web interface. These are the peaks that were off the scale in the previous slide. 21

22 An example ofdownload activity for a specific data collection ( We see that the scale is much smaller. Generally speaking, research groups may be interested in seeing the periods of activity for their data collections. 22

23 Screenshot1 (Left): Geographic distribution of visitors to the Data Access Portal in Australia. Screenshot 2 (Right): Geographic distribution of visitors to the Data Access Portal in China. 23

24 Note: this slide was not inthe actual presentation, but I have added it since they were the workshop challenges that I mentioned. 24

25 25

26 26