Post-Digitization Workflow at the University of Illinois at Urbana-Champaign:
Work to do After Your Books are Digitized by OCA

Timothy Cole, Michael Norman, Kurt Groetsch, Matthew Cordial

Users look for books using a variety of tools. OCA began digitizing books from the library collections of the University of Illinois at Urbana-Champaign (UIUC) late last year. Digitized copies of our books became visible in the Internet Archive within days of being digitized. However, for on-campus faculty and students searching our local online catalog, the fact that a particular title had been digitized was not immediately apparent. We needed to do additional work to link from our online catalog, and from other local discovery services, to OCA digitized content. Additionally, in the tradition of multiple copies in multiple locations helps to protect and preserve content, we wanted to maintain local access and archival copies of books from our collections that had been digitized by OCA. As we scale up to a higher pace of digitization generally, the need to automate these processes became clear. Similar issues loom on the horizon in regard to content digitized by other vendors. This motivated us to design a new post-digitization workflow for handling digital copies of books and other resources derived from our collections.

Using the OCA digitization as a model and pathfinder, we began by developing scripts to download copies of digital files created by OCA. Library staff members use the Web-based OCA Meta-Manager to identify sets of books that have been digitized from our collection (in batches of about 25 at a time). The ability to order in Meta-Manager by curate date makes this task easy and straightforward. Tab-delimited files saved from OCA Meta-Manager are used as inputs to launch a script that then downloads files on a batch basis (usually overnight) from the Internet Archive using the GNU-licensed WGet utility. The download process is logged in a Microsoft SQL Server 2005 database. 

A second script is then launched subsequently which automatically detects and processes each downloaded digitized volume. This script, written using Windows Scripting Host (as is the download script), uses the Microsoft XML Parser and other built in objects of the scripting environment to migrate sets of downloaded files for each book to appropriate locations on the Library's Web servers. A subset of files appropriate for general access is put in a publicly accessible directory of our Web server. At the same time, a standard, universal use (for OCA content) Active Server Page (ASP) file is also copied into each volume-level directory. This ASP file automatically generates a splash screen for the digitized volume, taking required information from XML metadata files that were downloaded from OCA with the digital copy of each book. The complete set of all files associated with each book is copied into an online archive folder. Also as part of this process, we register Handle URIs for each book, associate each digitized volume with one or more topic-oriented digital book sub-collections, and update the MARC XML record for each book with handle and sub-collection information. The MARC XML is then copied elsewhere for use by a local system that will provide OAI-PMH metadata and an RSS feed of newly digitized books. Finally an entry is made in another of our SQL databases, associating both our UIUC Handle URI and the OCA URL for the book with codes that identify the record in our online catalog and on OCLC that needs to be updated or cloned in order to make users of our online catalog and OCLC aware that a book from our collection is now also available freely online. 

A third and final script is run to facilitate digital preservation locally of each book. This script organizes folders to be copied to a data DVD (last-resort backup). It also unzips all files, runs selected files against Harvard's JHOVE utility (capturing MIX technical metadata for image files), and creates a set of METS files for each book and DVD containing MODS metadata (derived from the MARC XML), PREMIS (preservation) metadata, and struct-map information showing page order information (as needed, for instance, for a METS-based page turner). The end result is a local copy for access, a local copy for preservation, updates to our local online catalog and to OCLC, and input for our local OAI-PMH data provider and RSS service.