Evaluating generative AI tools for purchase: a checklist

Evaluating generative AI tools for purchase: a checklist

At academic libraries, we’re starting to get pitches from developers of various apps and software based on generative AI – pitches to purchase this software for our campus.

Because of that, I realized that it can be challenging to evaluate these tools. Especially since it may require having some knowledge of how the underlying technology works.

So I decided to draft a checklist for this purpose. You can download it here:
Evaluating Generative AI Tools: A Checklist. (PDF)

This is version 1.2, updated on Dec. 23, 2023. (first version was created in Oct. 2023)

It’s a starting point for thinking about what you might want to ask developers when considering a purchase. Your comments and feedback are welcome!


To learn more of the basics about generative AI, see our interactive tutorials on ChatGPT and Generative AI, from the University of Arizona Libraries.

And to stay current with the latest news about generative AI, follow me on any of these platforms. I post daily.

What’s next with ChatGPT?

What’s next with ChatGPT?

I recently heard someone ask, “What’s next with ChatGPT?” This was a person who doesn’t have time to follow the news about generative AI on a regular basis.

So for those who haven’t had time to keep up, here are some interesting developments.

I’m not aiming to predict the future (well maybe a little bit), only to summarize some news that might be of interest if you haven’t had time to keep up with this topic.

1. The free version of ChatGPT will connect to the internet, making hallucination a bit less of a problem. Currently a web connection is only available in the Plus version ($20/month)

2. Language models from Open AI, Microsoft, Google, and others will be incorporated in many tools that people already use (Microsoft Copilot for Windows 11, Google docs, etc). See also The AI revolution is about to take over your web browser.

3. Educators will find more ways to use it creatively with students in the classroom. And some are moving beyond worrying about students writing essays with ChatGPT. See AI detectors: Why I won’t use them. See also Why Is My Attitude Towards Generative AI Different From Previous AI in Education? And also 6 Tenets of Postplagiarism: Writing in the Age of Artificial Intelligence.

4. The amount of text you can work with at one time will keep increasing, allowing you to work with the text of an entire book (Anthropic’s Claude already does this). You can summarize, outline, ask questions of the book, and so on.

5. Many more software tools and apps will continue to be created (some useful, some not) that use this technology (large language models). Some useful ones: Duolingo, Be My Eyes, Elicit: The AI Research Assistant, Explainpaper (for reading research papers), for just a few. See this TED Talk from Sal Khan, How AI could save (not destroy) education.

6. Plugins will be incorporated into Bing Chat and Google’s Bard. (The Plus version of ChatGPT already has plugins). However, they don’t always seem to work well yet. Listen to this podcast episode: Are ChatGPT Plugins Overhyped?

7. There will be some progress with making AI less biased and more multicultural. Some of the open source models  will do this (see BLOOM). Another model (not open source) that shows promise is Claude from Anthropic, with their method known as “constitutional AI.

8. Copyright issues will not be solved for a while, as it takes time for the courts to decide how to interpret the law. See this video for a summary: Generative AI Meets Copyright, by Pamela Samuelson, Professor of Law, UC Berkeley. See also this interesting development: Japan Goes All In: Copyright Doesn’t Apply To AI Training. Another article: Creative Commons makes the case that training AI models is fair use, see Fair Use: Training Generative AI.

9. Large companies will connect language models to their own knowledgebases (known as grounding), with very little or no hallucination. See Bloomberg Uses Its Vast Data To Create New Finance AI, and Introducing Healthcare-Specific Large Language Models from John Snow Labs, and Ethics of large language models in medicine and medical research.

10. Have you heard about autonomous agents? These are AIs that can work towards a goal that you set, and assign different tasks to other AIs. See Auto-GPT, BabyAGI, and AgentGPT: How to use AI agents and How to Use AgentGPT to Deploy AI Agents From Your Browser, and What is Auto-GPT and why does it matter?

11. And farther out in the future, there will be multi-sensory AI that can use multiple senses. See ImageBind from Meta. This open source model combines text, audio, visual, movement, thermal, and depth data. It’s only a research project for now, but you can imagine how this might work in future models.

12. If you’ve heard all the news of AI experts signing statements about risks of AI causing human extinction, you might wonder… is that another thing we need to worry about in addition to climate change and nuclear war? Here is the best response to that that I’ve read: Let’s talk about extinction by Azeem Ahzar. A good point he makes is that “just because people are experts in the core research of neural networks does not make them great forecasters, especially when it comes to societal questions or questions of the economy, or questions of geopolitics.”

13. And finally, people will continue to make serious mistakes in their use of ChatGPT, (until more people develop some degree of AI literacy). See Lawyer cites fake cases invented by ChatGPT, judge is not amused, and Professor Flunks All His Students After ChatGPT Falsely Claims It Wrote Their Papers.

This is why we all need AI Literacy. If you want to develop your own AI literacy (and teach others to do the same), take a look at my May 18 webinar recording, AI Literacy: Using ChatGPT and AI Tools in Instruction. See also the handout for many more sources.

I’ll be doing a similar webinar for AMICAL on June 21, and I’m also talking with ALA about a series of webinars for later this year.

Follow me on Twitter or Mastodon for daily links to AI news.

AI Literacy: May 17 webinar

AI Literacy: May 17 webinar

ChatGPT has been making headlines, with both positive and negative stories, including concerns about plagiarism and false information. You may not have had time to keep up with all the news, and you might be wondering how to separate the hype from the reality.

On May 17th I’m offering a webinar to address these topics. No matter where you fall on the spectrum between critic and enthusiast, it’s important for all of us these days to have basic AI literacy.

AI Literacy: Using ChatGPT and Artificial Intelligence Tools in Instruction
May 17 at 1:30 pm CDT via the American Library Association

In this webinar, you’ll learn why understanding AI tools (like ChatGPT & Bing Chat) is an important part of information literacy and something library staff should be learning about now.

After participating in this event, you will:

  • Understand the basics of ChatGPT and how it works
  • Know about the company behind it and the data that went into the product’s training
  • Know several examples of how to use ChatGPT and similar tools effectively
  • Understand some common criticisms of the technology and problems with it in its current form
  • Know some other tools and apps are available that use this technology

You’ll come away with:

  • A basic understanding of this technology (AI tools based on large language models).
  • Knowledge of why AI literacy is an important part of information literacy.
  • Inspiration & tips for offering workshops, guides, or handouts for your users on this topic.
  • A bibliography of best sources for learning more.

Bring your questions and comments! We’ll include time for discussion.

 

ALA Member Price: $71.10
Non Member Price: $79.00
Credit Type: Certificate Available upon Completion
Chat GPT FAQ

Chat GPT FAQ

In January, I put together a document to share with my department at the Univ. of Arizona Libraries (Student Learning and Engagement): Chat GPT FAQ.

It’s a work in progress – and I welcome your comments.

There are so many interesting questions about what these new tools means for education!  So I’m learning everything I can about this topic and planning a webinar for librarians.

Feel free to share the FAQ with others!

Webinar coming soon!

On May 17th I’ll be offering a webinar via the American Library Association: AI Literacy: Using ChatGPT and Artificial Intelligence Tools in Instruction.

It will focus on the importance of AI literacy for librarians – so we can help our users develop that same literacy.

You’ll learn why understanding AI tools (like ChatGPT) is a necessary part of information literacy and something library staff should be learning about now.

I’ll post more about the webinar soon – after ALA has announced it on their website.