Text Mining

Library Collections and Text Mining

Below is a nonexclusive list of Library supported or subscribed resources that allow some method of text mining. Preferred methods must be followed, or access to the resources could be shut down for the entire campus.

Books

HathiTrust Research Center (HTRC) Analytics This link opens in a new window
Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access.

Project Gutenberg
Hosts over 50k ebooks, most of which are older books in the public domain.

Scholarly Sources

CORE: Open Access Research Papers
CORE provides a central API to access full content from tens of thousands of openly available scientific publications from thousands of OA repositories. They also provide full datasets by request.
Elsevier (ScienceDirect)
Researchers can text mine UCSC-subscribed journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes, and make sure to query the API from UC Santa Cruz IPs to ensure full access.
PLOS (Public Library of Science)
Python tool for downloading/updating/maintaining a repository of all PLOS XML article files. Use this program to download all PLOS XML article files instead of doing web scraping.
arXiv Bulk Data Access

Newspapers & Magazines

New York Times APIs (1851-present)
The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.

Chronicling America: Historical American Newspapers This link opens in a new window
The Chronicling America API provides access to information about historic U.S. newspapers and millions of digitized newspaper pages and their OCR data is available for bulk download.

Citations & Metadata

Library of Congress: 25 million bibliographic metadata records
The LOC release of 25 million MARC records for free bulk download. MARC (Machine Readable Cataloging Records) is a international metadata standard for the representation and communication of bibliographic and related information.
NY Times APIs
The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.
Open Academic Graph
Downloadable datasets for citations drawn from two large academic graphs: Microsoft Academic Graph (MAG) and AMiner.
PubMed and NLM: Data Guide
A guide to using this API, called E-Utilities, to access citation data for medical journal literature in PubMed and other NCBI databases, including the National Library of Medicine Catalog, MeSH, Gene, and PMC (PubMed Central).
CrossRef
The CrossRef Metadata API for text and data mining provides a common solution which works across open access and subscription-based publishers and is free for anyone to use.

Government Documents

Congress and Government Information APIs (Sunlight Foundation)
Sunlight Foundation provides APIs to a variety government information sources including: Congress API v3: information on legislators, districts, committees, bills, votes, as well as real-time notice of hearings, floor activity and upcoming bills. Open States API: Information on the legislators and activities of all 50 state legislatures, Washington, D.C. and Puerto Rico. Real-Time Federal Campaign Finance API: A JSON and CSV API that delivers up-to-the-minute campaign finance information on federal candidates, committees, PACs and other groups that file electronically with the Federal Election Commission.
CourtListener APIs and Bulk Legal Data
Opinions, docket files, and more from over 400 courts.

Please scrape responsibly

Using Python, Selenium, or other programmatic tools to scrape database search results (even cleverly) can result in access being shut down for the entire campus. Email research@library.ucsc.edu to ensure that actions applied to potentially copyrighted texts complies with legal standards when we publish our results.

Help

Send questions about text and data mining access to research@library.ucsc.edu to inquire about subject, copyright, technical, and licensing expertise.

1156 High Street spacer bullet

Santa Cruz, CA spacer bullet

95064

Feedback

Creative Commons Attribution 3.0 License except where otherwise noted.

Patrons with Disabilities
Privacy Policy

Staff Portal
LibApps Login
Incident Form (staff only)

Land Acknowledgement

The land on which we gather is the unceded territory of the Awaswas-speaking Uypi Tribe. The Amah Mutsun Tribal Band, comprised of the descendants of indigenous people taken to missions Santa Cruz and San Juan Bautista during Spanish colonization of the Central Coast, is today working hard to restore traditional stewardship practices on these lands and heal from historical trauma.

The land acknowledgement used at UC Santa Cruz was developed in partnership with the Amah Mutsun Tribal Band Chairman and the Amah Mutsun Relearning Program at the UCSC Arboretum.

University Library