Link Search Menu Expand Document

Software

Table of contents

  1. Types of Software
    1. 7-zip
      1. Install 7-zip binary
    2. AWS
    3. Bagit.py
      1. Installation
    4. BitCurator
      1. Installation
    5. dBpoweramp
    6. dd
      1. Use
    7. Disk Utility
    8. DiskPart
      1. Use
    9. disktype
      1. Installation
      2. Use
    10. Filemaker Pro 17
      1. Installation
      2. Use
    11. Forensic Toolkit
      1. Installation
      2. Use
    12. gtar
      1. Installation
    13. Iromlab
    14. Java
      1. Use
    15. openssl
    16. Oracle VM VirtualBox
      1. Installation
    17. pip
      1. Use
    18. Python3
      1. Installation
    19. QuickTime Player
    20. rsync
      1. Installation
    21. Siegfried
      1. Installation
      2. Use
    22. Sha1deep
      1. Installation
      2. Use
    23. tree
      1. Installation
    24. VLC
      1. Installation
  2. Scripts
    1. ft.sh
    2. kryofluxmove.sh
    3. metadata.sh
    4. Brunnhilde with disktype
    5. makesips script
    6. moveimages script
    7. movemetadata script
    8. movephotograph script
    9. FACTools
    10. iterative scripts
    11. CMS metadata import
  3. Previous and Occasional Software
    1. Emailchemy
    2. ePADD
    3. DROID
    4. MediaInfo
    5. FTK Imager
    6. IsoBuster
    7. Bagger
  4. Legacy Format Appraisal
    1. WinUAE
    2. apple3rtr
    3. miniVmac
    4. DosBox
    5. QuickView Plus
    6. CiderPress
    7. HFS Explorer
    8. Sheepshaver

Types of Software

The following is a list of software used in born-digital workflows at NYPL.

7-zip

7-zip is used in the lab to compress evidence and cases generated in Forensic Toolkit. It is also occasionally used to unpack collection material that is not supported by FTK.

Install 7-zip binary

Download 7-zip

AWS

Amazon Web Services features S3 and Glacier are used to store copies of born-digital material both before and after the collection has been processed by archivists.

Using the AWS dashboard requires a login.

Bagit.py

Bagit.py is used to transfer born-digital material to storage. Bagit creates file manifests that help to ensure transfers are complete and free of errors.

Installation

Check python version
python --version
or
python3 --version
then install
pip install bagit
or
pip3 install bagit

BitCurator

BitCurator is a Linux distribution with a suite of open source digital forensics and data analysis tools used in the lab as both a virtual machine and as bootable OS alongside Windows. BitCurator is used for transfer and metadata extraction, especially for formats unrecognized in Windows and Mac environments.

Installation

Download software BitCurator VM or Live CD

Current release of VirtualBox

The VirtualBox Extension Pack must be installed on the host for shared folder support, USB 2.0/3.0 support, and support for some NVMe SSDs. Once you’ve installed VirtualBox, use the link below the main download on VirtualBox download site to install the Extension Pack on your host.

Unpack the BitCurator Virtual Machine.

Once you’ve installed VirtualBox and the VirtualBox extension pack, start up VirtualBox.

Add Virtual Machine
From the menu bar, select the menu item “Machine -> Add…”, and navigate to the folder containing .vbox file that you extracted. Choose that file, and the Virtual Machine should appear in the list within the manager.

dBpoweramp

dBpoweramp is used in conjunction with Iromlab and the Nimbie to transfer born-digital material from optical media to storage.

dd

The command line utility dd is used in the lab to create disk images of optical media. In most cases transfers are made of optical media instead of disk images.

Use

In the terminal type
dd if=/dev/sr0 of=/cygdrive/z/ingest/diskImages/CollID/MediaID/objects/MediaID.iso
Hit return
Once complete the terminal will display bytes copied and return to a blank prompt.

Disk Utility

On MacOS Disk Utility is used in the lab to confirm that removable media is properly mounted.

DiskPart

On Windows DiskPart command line utility is used in the lab to confirm that removable media is properly mounted.

Use

Type
list disk
or
list volume
to confirm media is mounted.

disktype

Disktype is used on MacOS in the lab to extract filesystem metadata such as sector format for floppy disks.

Installation

brew install disktype

Use

cd <diskDirectory>
diskype <disk.001>

Filemaker Pro 17

Filemaker is used by Preservation and Collection Services as a collection management system. Born-digital media is inventoried in Filemaker and PCP tracks projects through Filemaker.

Installation

Installing Filemaker requires license keys. Make sure you have the proper keys and permissions before installing.

Use

Forensic Toolkit

Forensic Toolkit is used in the lab to appraise, and arrange born-digital material.

Installation

Forensic Toolkit is licensed software and requires a dongle to open. The software can be downloaded from AccessData

Use

gtar

Gtar is used on a Mac in the lab to compress KryoFlux stream files. It is called through the command line with the bash alias ‘compress’.

Installation

brew install gnu-tar

Iromlab

Iromlab is used in conjunction with dBpoweramp and the Nimbie to transfer born-digital material from optical media to storage.

Java

Java is required for tools like DROID and Bagger.

Use

Try commands
which java
and
java -version
if you are encountering java errors. You may need to put the path of the correct version of java in your bash profile.
In Windows you can use the search from the start menu to check the java version. Type ‘about java’ into search. Then click the About Java icon result.

openssl

openssl is used in the lab to SSH between computers. SSH is used to run workflow scripts located on ARCHV Mac.

Oracle VM VirtualBox

VirtualBox is used in the lab to run the BitCurator environment as a virtual machine.

Installation

Current release of VirtualBox

The VirtualBox Extension Pack must be installed on the host for shared folder support, USB 2.0/3.0 support, and support for some NVMe SSDs. Once you’ve installed VirtualBox, use the link below the main download on VirtualBox download site to install the Extension Pack on your host.

pip

Pip is used in the lab to install and upgrade Python modules like bagit.py.

Use

Pip should be installed with python3. Call pip with
pip or pip3
depending on your installation.
Check python version
python --version
or
python3 --version

Python3

Python3 is used in the lab to run a a variety of scripts, including bagit.py.

Installation

Windows:
Check python version
python --version
Installers are available at python.org

Mac:
Check python version
python3 --version
Replace the number below with the version you wish to install.
brew install python@3.8
BitCurator:
Check python version
python3 --version
Replace the number below with the version you wish to install.
sudo apt-get install python3.8

QuickTime Player

QuickTime is used in the lab to appraise Macintosh formatted born-digital video.

rsync

Rsync is used in the lab to transfer born-digital material when bagit.py is unable to complete a transfer. Rsync is used when transferring from a computer backup or a drive with a large number of system files.

Installation

Mac:
brew install rsync

Siegfried

Siegfried is used in the lab to identify and validate file formats. Siegfried output also includes checksums, useful when rsync is used for transfer instead of bagit.py.

Installation

Windows:
Download binary
Mac:
brew install richardlehane/digipres/siegfried

Use

Current installations of siegfried use the config file to store preferred command options.
sf -setconf -csv -hash sha1 -z
sets the preferred command options.
Find a complete list of sf commands with
sf -help

Sha1deep

Sha1deep is used in the lab to identify duplicate material across digital media in a collection. It is usually used in bash scripts.

Installation

Mac:
brew install m5deep

Use

Here is an example of how to use sha1deep to transfer only unique files from media.
Write checksums of transfer volume to file
sha1deep -r [directory or volume] >> collnumber.sha1
Write path of unique files per directory or volume to file with mediaid
sha1deep -rx [path to checksums file] [directory or volume] >> [path to checksums for media]
Remove ds_store from unique file list
sed '/.DS_Store/d' [path to file list] > [file list]
change terminal path out of home directory
cd ../../
Navigate to the media being transferred to avoid /Volumes/HD/folders being copied within media-id directory.
Make sure spaces are not escaped in paths. “ “ not “\ “
Transfer unique files
rsync -aPF --files-from=[path to file list] . [path to media]

tree

Tree is used in bash scripts in the lab to identify contents of media including number of folders and files.

Installation

Mac:
brew install tree

VLC

VLC multimedia player is used in the lab to view and appraise born digital audio and video.

Installation

https://www.videolan.org/vlc/index.html

Scripts

The following is list of scripts used in born-digital workflows at NYPL. The scripts are located in the digarch_scripts Github repository. A link to each script is provided after the description.

ft.sh

This script is used in the lab to create bagit transfers and Siegfried csvs for born-digital media.
ft.sh

kryofluxmove.sh

This script is used in the lab to move KryoFlux disk images to folders with the MediaIDs of the disks.
kryofluxmove.sh

metadata.sh

This script is used in the lab to move metadata from disk images to folders with the MediaIDs of the disks.
metadata.sh

Brunnhilde with disktype

This script is used in BitCurator to identify disk image type and pass the type to Brunnhilde to generate the correct output.
Brunnhilde with disktype

makesips script

This script is used to create a consecutive number of submission information packages for material from digital media.
makesips script
Alternatively, mkdir command can be used to create SIPs. This works when SIPs aren’t consecutively numbered. 0001 to 0009 require a different line from 0010 on.
mkdir -p CollID/Media-000{1..9}/{metadata/submissionDocumentation,objects}
mkdir -p CollID/Media-00{10..99}/{metadata/submissionDocumentation,objects}
mkdir -p CollID/Media-000{1,5,7,9}/{metadata/submissionDocumentation,objects}

moveimages script

This script is used in the lab to move a disk image into the objects folder of a SIP.
moveimages script

movemetadata script

This script is used in the lab to move metadata into the metadata folder of SIP.
movemetadata script

movephotograph script

This script is used in the lab to copy JPEGS of media from the photographs directory on the staging drive to the submissionDocumentation directory in the MediaID directory of SIPs.
movephotograph script

FACTools

This script is used in the lab to repackage Finding Aid Component packages.
FACTools

iterative scripts

Scripts are created per collection using a directory listing as input in a while loop. This allows previous move scripts to run per collection rather than one MediaID.

CMS metadata import

This script is used in the lab to import metadata from transfers to the FileMaker Collection Management System. Currently, output from the Nimbie (Iromlab) is used.

Previous and Occasional Software

The following is a list of software used in previous workflows. These tools can be used when transfer fails with current tools.

Emailchemy

Emailchemy has been used in the lab to migrate email formats.

ePADD

ePADD has been used in the lab to appraise and arrange email.

DROID

DROID has been used in the lab to identify and validate file formats.

MediaInfo

MediaInfo has been used in the lab to extract metadata from born-digital video.

FTK Imager

FTK Imager has been used in the lab to image media and extract files from media.

IsoBuster

IsoBuster has been used in the lab to image optical media and extract files from media.

Bagger

Bagger has been used in the lab to transfer born-digital material.

Legacy Format Appraisal

The following is a list of software used in the lab to view legacy formats not supported by FTK.

WinUAE

WnUAE has been used in the lab to view Amiga files.

apple3rtr

Apple3rtr has been used in the lab to view Apple II files.

miniVmac

MiniVmac has been used in the lab to view legacy Mac files.

DosBox

DosBox has been used in the to view DOS executables and word processing files.

QuickView Plus

QuickView Plus has been used in the lab to view a range of legacy formats like word processing and photo formats.

CiderPress

CiderPress has been used in the lab to view Apple file formats.

HFS Explorer

HFS Explorer has been used in the lab to view files on HFS filesystems.

Sheepshaver

Sheepshaver has been used in the lab to view legacy Mac files.