"Wikipedia-grounded chatbot "outperforms all baselines" on factual accuracy"

Because no one else is doing it--not even the media.
Post Reply
User avatar
ericbarbour
Sucks Admin
Posts: 4663
Joined: Sat Feb 25, 2017 1:56 am
Location: The ass-tral plane
Has thanked: 1172 times
Been thanked: 1872 times

"Wikipedia-grounded chatbot "outperforms all baselines" on factual accuracy"

Post by ericbarbour » Mon Jul 17, 2023 6:38 pm

https://en.wikipedia.org/wiki/Wikipedia ... t_research
This article is a first draft of an analysis of the relationship between the availability of a scientific journal as open access and the fact that it is cited in the English Wikipedia (note: although it speaks of "Wikipedia", the article looks only at the English pages). It is a preprint and has not been peer-reviewed, so its results should be read with caution, especially since I am not sure about the robustness of the model and the results derived from it (see below). It is of course a very important issue, as access to scientific sources is key to the diffusion of scientific knowledge, but also, as the authors mention, because Wikipedia is seen as central to the diffusion of scientific facts (and is sometimes used by scientists to push their ideas).
It stinks of WMF propaganda. Using WP to "push their ideas" does happen, but is also a standing joke in many scientific circles. Smart people tend to realize that Wikipedia content has problems and is not fully reliable. But don't ask Nicholas Jullien, a recent en-WP import from French WP; he gives off an odor of Wiki-Fannery.

Much more informative:
→ The regressions seem to indicate that the reputation of the journal is not important to be cited in Wikipedia.
→ Predatory journals are known to be more often open access than classical journals, which means that this result potentially indicates that the phenomenon of open access reduces the seriousness of Wikipedia sources.
Yeah, THIS I can believe easily. WP has used "fake academic journals" and predatory publishers as "reliable references", and gotten away with it. No one has any idea how commonplace the practice is.

For even more disgust, read the Andreas Kolbe item below it, about Japanese WP becoming biased by right-wingers trying to rewrite Japan's history during World War II. THAT is truly disgusting.

User avatar
rubricatedseedpod
Sucks
Posts: 58
Joined: Fri Jun 23, 2023 6:56 pm
Location: The Jungle of Views
Has thanked: 18 times
Been thanked: 31 times

Re: "Wikipedia-grounded chatbot "outperforms all baselines" on factual accuracy"

Post by rubricatedseedpod » Mon Jul 17, 2023 9:26 pm

Just need Alexandra Elbakyan to make us a Sci-Hub grounded chat bot: that would really be useful!

Now, why don't we read the paper to determine how they define factual accuracy?
We define factual accuracy (FA) of a dialogue set as the percentage of claims that are supported by Wikipedia. The factual accuracy of a chatbot is the factual accuracy of the dialogues we obtain when pairing it with our user simulator on the 60 topics in our benchmark.
oh... :?
To be clear, they had a chatbot make up stuff using Wikipedia and then had humans check it against Wikipedia to obtain an accuracy score. If this indicates anything about Wikipedia, it would perhaps just be that it is written in an AI-readable style. What it does not indicate is that either the chatbot or Wikipedia is anywhere near truthful. And indeed, there is 0 mention of this circularity in the Signpost review. Thanks, Tilman Bayer!
Editing Wikipedia is not a substitute for being a person.

User avatar
ericbarbour
Sucks Admin
Posts: 4663
Joined: Sat Feb 25, 2017 1:56 am
Location: The ass-tral plane
Has thanked: 1172 times
Been thanked: 1872 times

Re: "Wikipedia-grounded chatbot "outperforms all baselines" on factual accuracy"

Post by ericbarbour » Tue Jul 18, 2023 1:09 am

rubricatedseedpod wrote:
Mon Jul 17, 2023 9:26 pm
Thanks, Tilman Bayer!
As you can see from his en-WP userpage, he was a bigshot on de-WP, until 2011 when he became a WMF employee. I suspect he was one of the hardened JImbosuckers pushed out in 2019 by Katherine Maher. Nothing will EVER be admitted.

He's not the worst insider but he is one of the most uncritical supporters of the WikiWay. He spends a LOT of time looking for research papers that mention WP or the WMF, only to drop them on the Signpost. And posting copies of the figures from those papers on Commons (presumably those papers are CC or free licensed?). Apart from patrolling "vandalism" he performs very little content work. In his early days he mostly copied en-WP articles to de-WP. Twenty years of his life, thrown away.

Post Reply