The Web of Science Research Assistant works as a chat interface for searching the content of the Web of Science Core, which offers 92 million references in the scientific field since the beginning of the 20th century. Clarivate invested nine months of testing the GenAI assistant's new search possibilities and heavily involved its user community, particularly its newly formed AI Academic Advisory Council, in the process.
Interface and search options
The home screen is the same as with the classic Web of Science search. Access is through a click on "Search Assistant," which opens the new interface. Users are presented with four options. Clarivate's take on why a searcher should choose each option follow in quotes.
The first, called "Start a new search by asking a question," allows users to launch a specific question in natural language. "This is an intuitive way to start an exploration, particularly useful for researchers unfamiliar with advanced search techniques."
The second, "Understand a topic," aims to quickly offer a structured overview of a specific subject. "It is an excellent starting point to quickly get acquainted with a new research area."
The third, "Literature review," is designed to help conduct a more in-depth literature review. "This feature is particularly useful for researchers preparing a detailed study or writing the introduction of a scientific article."
The fourth, "Find a journal," helps to identify a publication to which a researcher can submit an article presenting their research. We will not dwell on this, as our focus is resolutely on information retrieval.
Analysis of the "Start a New Search by Asking a Question" option
We asked a series of questions using this first option. Our first question with this option was, "What are the computer vision systems for defect detection?"
The first result is a summary of the content of the first references. It is worth noting that the same question posed a few days later did not generate exactly the same answers, which is common with AI tools.
Next, the system indicates it searched by combining the concepts "computer vision systems" AND "defect detection". It found.58,039 results. We will come back to this.
At this point, Research Assistant proposed two options:
- Suggested Questions
As soon as it served the first retrieved eight references, query suggestions related to the request appeared at the bottom of the screen. In our example, the following suggestions were made:
- A graph showing the evolution of publications in the field of computer vision over time—note that only one of the concepts is used.
- A "topic map" presenting, with links, the various concepts related to computer vision, again just one concept. A table can be viewed indicating the number of documents in which each concept (e.g., "feature extraction" or "image segmentation") appears. It is not possible to click on a term in the table, but it can be done on the map.
- The identification of the six most important authors, still using only the computer vision concept. Some of them stopped publishing a long time ago, but all of them have published extensively.
- Foundational or reference documents on the selected concept, once again just computer vision: "I want to know seminal papers about computer vision." These documents may have been cited tens of thousands of times.
These four types of suggested questions, adapted to the subject, are found regardless of the initial question. More specific questions related to the subject, typically two or three, are also proposed. For example, they could pertain to recent research, evolution over time, pioneers, the countries where the research has been most developed, or developments in specific industry sectors.
Occasionally, the system responded that it found nothing. Apparently, suggested questions are created as soon as the initial question is asked rather than being "canned questions".
- Access to References
The second option is to view these eight references in a reduced format. These documents, overall, are not very recent, and some have been cited numerous times (often hundreds). Each reference can be viewed in full, and you can also see documents citing them or "related documents", the latter being proposed in the "view more" option.
Full list of results
To go beyond these first eight references, you click on "view additional documents relevant to this response." Then, the start of the list of 58,039 documents, mentioned earlier, appears, which, as indicated, were obtained using the strategy: computer vision systems OR defect detection. The OR operator seems more in line with the number of responses than the AND operator previously mentioned.
We have carried out several searches like this, and each time the system breaks down the query into concepts, then systematically links them with an OR operator—which logically leads to a very large number of responses, prioritizing recall over precision.
It is not clear how the system searches to obtain such a large number of results. Out of curiosity, we searched in the classic interface using phrases with quotation marks. The results numbered 16 with the AND operator and 9,673 with the OR operator. Without quotation marks and without the "OR" term, the number of results becomes 790, corresponding to a search where all words are implicitly connected by an AND operator.
Returning to our list of 58,039 documents, they are sorted by default in decreasing relevance. The references are slightly more complete than the first eight since they also include the abstract, the number of citing and cited references with links, a link to the article's full text, and finally, a link to "related records" that leads to a list of documents displayed in the same format as the original list.
Exploitation of document lists
Let’s dwell for a moment on these document lists that occur at several stages of the research process. They are found:
- After the first eight responses when you ask to view the full list of responses;
- As a set of documents citing a reference,
- Or as "related documents," accessible directly or via one of the "view more" options in other cases.
These lists often offer a very large number of documents—it is not uncommon for them to include tens of thousands. They are sorted by default in decreasing relevance or by most recent date, but other sorting options are available.
Since it is obviously impossible to view them all, a first simple option is to select from the first few if they are sorted by date and/or by decreasing relevance.
We also noticed that for documents sorted by decreasing relevance, only the first 1,000 can be viewed, even if there are tens of thousands more.
However, many other selection options are available, accessible from the left side of the screen.
- You can first search within the documents by entering a term in the search box at the top left. This term is searched across all fields, including the publication title, which can sometimes be inconvenient.
- A large series of criteria are also available for selecting the most relevant references for your specific issue, particularly "Web of Science categories" and "research areas", from which you can include or exclude items.
- You can select by publication date. You can also filter by open access articles and/or reviews (noting that in "literature review" searches, which we will discuss later, the results often consist entirely of reviews).
The authors, as expected, are ranked in decreasing order of frequency. You can filter by classic criteria such as the article's language, the country of origin of the author(s), publishers, funders, affiliations, and publication titles.
Analysis of the "Understand a Topic" option
To test this option, which aims to offer a structured overview of a specific subject, we chose "CO2 capture from the atmosphere" as our research topic.
- First, we asked the question in French, "la capture du CO2 dans l’atmosphère." Clearly, there was a problem with the system's understanding of the question, as the French summary of the first eight references read: "The documents discuss the growing presence of methane in the atmosphere."
We repeated the same question, and the summary became: "The documents discuss the importance of methane as a greenhouse gas, its role in the Earth's atmosphere, and its impact on the climate system." A third attempt gave a similar, but not identical, response.
- We then posed the question in English: "CO2 capture in the atmosphere."
This time, the summary (in English) correctly discussed CO2 capture, but it was related to capturing CO2 at the output of a manufacturing process in a factory, which does not answer the question. In one abstract from the eight references, we found the term "direct air capture," which we found very interesting.
We then launched a search, still in the "understand a topic" section, with the question "direct air capture of CO2." This time, the first eight references were relevant, and the concept even appeared clearly in the title.
Once again, it is worth noting that the corresponding documents were published before 2020. Additionally, four of them had more than 500 citing references, and seven of the eight had more than 300 citing references.
From there, when viewing citing references, it is very useful, or even necessary, to limit the search to those containing the word atmosphere or the word air if you want relevant documents. However, this is much less necessary with "related documents".
We asked the same question ("what is new in the field of...") using the first option, "start a new search by asking a question", and in this option, both the English and French versions gave very relevant results in their respective languages.
We then posed the same question but using the "literature review" option, and, surprisingly, we found the same first eight references as we found with the "understand a topic" option, with the total number of 97 references (we didn’t check them one by one, but they are at least very close).
When we asked a question about industrial ovens, we were a bit surprised by the prominence of bread making and bakery references, as three of the first eight references mentioned baking bread, another mentioned a baker's oven, and a fifth one referenced cake baking. Check made, this was not illogical, as the term oven is commonly used for food cooking in a domestic or commercial setting. A search with "industrial furnace" gave very different results.
However, a search in French yielded completely irrelevant responses, all on the same topic—the British Industrial Revolution from 1760 to 1830!
Analysis of the "Literature Review" option
The results of the search on industrial ovens, unlike the results in the "understand a topic" issue, were not focused on bread making or bakery. However, another topic was very prominent. In fact, the word coke appeared 32 times among the 69 documents found.
If you search in French with the expression "four industriel," half of the results (in English) are relevant.
Returning to the search on CO2 capture from the atmosphere, the previously mentioned 97 results are once again old, with the most recent dating from 2018. Furthermore, in one-third of the documents (33), the word plant appears, indicating that CO2 capture is occurring at the end of a manufacturing process rather than from the atmosphere.
It is also worth noting that the obtained documents are reviews, which are indeed very useful for quickly getting an overview of a topic.
Our opinion
In general, users must remain active and vigilant when interacting with the tool, which is only partially a black box where no intervention is possible.
The idea of offering four ways to approach research corresponding to different issues is interesting, even though the differences in terms of results are not always spectacular.
Key points
- The ability, beyond the initial results, to easily access citing references, co-citations, or related documents, all of which are potentially relevant and up-to-date.
- The offering of a series of questions related to the initial question.
- The ability to select from lists of several hundred or thousands of potentially relevant documents in multiple ways.
Problematic aspects
- We were very bothered by the fact that a large portion of the initial-level responses were old, even very old. It is necessary to take a second step to obtain recent documents.
- The tool does not always precisely understand the question, and sometimes not at all, as shown by an example in French, and it occasionally provides answers focused on a specific aspect of the topic.
Bottom line
We conclude that the AI engine still needs some further training...!
François Libmann is the Director of FLA Consultants, based in France. A version of this article, in French, appeared in BASES, no. 428, September 2024.