What is Digital Encoding? Digital Imaging Compression & Sampling A very short introduc?on Howard Besser NYU Moving Image Archiving & Preserva?on hhp://besser.tsoa.nyu.edu/howard/talks/ hhp://www.nyu.edu/?sch/preserva?on/ 1 Why Digital is different than Analog Digital Audio as example Compression for Audio and S?ll Images Frames and Frame Rate, Bit Rates, Chroma Subsampling The process of digital photography Digital Compression coding Wrappers (not discussing copy photography) 2 Sampling Sound Wave over Time Frequency : every for seconds Digital Sample Points on Analog Wave 3 4 Digital Audio Sampling convert from con?nuous to discrete Digi?za?on, sampling every 4 seconds, 1 bit Resolution 5 6 Sample Voltage Signal (in Time base 2) 0 4.3 1 4 6 1 8 6.5 1 12 1.5 0 16 4.2 1 20 4 1 1
Audio Digi?za?on sebngs depend on several factors The quality on the recording, and the quality you need for your users When you need to capture sounds that are less complex (shorter and less complex sounds), you can lower your signal resolu?on and/or sampling rate (the human voice on an interview requires less signal resolu?on/fidelity than most music Digi?za?on sebngs for speech are rela?vely low unless you intend to capture background sounds as well (background music whispers, gunshots, ) Resolu'on Human speech can be effec?vely reproduced at a rate of 5.5 khz (kilo Hertz) 5500 samples/sec Most natural world sounds and medium fidelity music can be reproduced at 11kHz with acceptable losses of fidelity 11000 samples/sec. high fidelity music at CD audio quality the sampling rate must be absolute minimum of 44.1 khz, giving a playback rate of 22 khz, which is just above the limit of human hearing, but most professional and academic organiza?ons recommend about double that (96 khz). But for 44.1 khz: 44100 samples/sec. 22000 samples/sec is just above the limit of human hearing 7 8 Quan'za'on The number of dis?nct sound levels that can be represented is determined by the number of bytes used to store the quan?za?on value. the most common quan?za?on strategy uses 16 bits (2 bytes). Ex : 11111111 2 16 = 65 536 discrete levels Audio: Pulse Code Modula?on (PCM) Sampling rate (number of samples per second) 48 khz, 96kHz Bit depth (range of possible values for each sample) 16 bit, 24 bit 9 10 Standards/Best Prac?ces for Sound Digi?za?on Are managed by professional communi?es with common interests Interna?onal Oral History Associa?on hhp://www.iohanet.org/resources/technical.html Interna?onal Associa?on of Sound and Audio Visual Archives (IASAA) hhp://www.iasa web.org/audio preserva?on tc04 Sound Direc,ons: Best Prac,ces for Audio Preserva,on h7p://www.dlib.indiana.edu/projects/sounddirec,ons /paperspresent/index.shtml University Libraries, S?ll Image Encoding 11 12 2
Film vs CCD Array CCD for capture 13 from hhp://micro.magnet.fsu.edu/primer/digitalimaging/concepts/fullframe.html 14 Pixels Pixels 15 16 Resolu?on Example: 50 800 ppi dpi, ppi, spi (scanner vs. camera) 17 18 3
Bit depth Bit depth example 8 bit, 24 bit, 36 bit 19 20 Full color spectrum vs 8 bits RGB Color 21 22 Compression for capturing con?nuous tone scenes JPEG/JFIF Predic?ve coding DCT Compression for capturing con?nuous tone scenes JPEG/JFIF Predic?ve coding DCT 23 24 4
Chroma SubSampling/ DCT encoding Mostly, eye is more sensi?ve to luminence than to chroma Divide image into matrix (8 x 8 cell blocks; more blocks for H.264 [but also include sub blocks]) Separate luminence (Y) from Chroma (Cb, Cr) Encode luminence for each individual cell, but encode chroma for groups of cells (chroma subsampling) 25 Encode more data for luminence than chroma 4:4:4 is fully sampled 4:2:2 uses half the chroma samples on each chroma channel 4:2:0 uses ¼ the chroma samples on each chroma channel Image from Tony Gladvin George 26 DCT Coding ar?facts Ar?facts become less visible as we zoom out 27 28 Ar?facts become less visible as we zoom out File Formats TIFF JFIF (JPEG) GIF J2K Raw DNG (Adobe) 29 30 5
Different file formats Other Ar?facts: Aliasing 31 Aliasing due to high frequency cleared up with low pass filtering before digi?zing hhp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/sab/report.html 32 Camera Metadata EXIF metadata example XMP (Adobe s extensible Metadata Plaworm) IPTC (Interna?onal Press Telecommunica?ons Council) hhp://www.iptc.org/site/photo_metadata/embedded_metadata_manifesto_%282011%29 EXIF (Exchangable Image File Format) 33 34 Digital Video Compression: Codecs the scheme used for compressing and encoding the material. Usually requires decoding soyware or hardware to play it. Hardware decoding in digital video tape players Usually soyware decoding in file based digital video with web browsers: might be built in to browser (jpg) or require browser plug in (flv) Otherwise, requires player soyware or hardware card (J2K) Encoding is usually asymetrical Takes lots of?me & resources to encode but decoding can be done in (near) real?me on standard wkst Moving Image Digital Files Each frame (25 30 fps) is essen?ally stored similarly to a s?ll digital image frame (in terms of representa?on and compression) Compressed formats save on storage space by not saving every single frame May store only the small differences from the previous frame May store periodic full frames and predict /interpolate btwn the stored frames 35 36 6
Digital Video Compression Different compression levels (Carl Fleischhauer) Spa?al image compression (intraframe) Temporal mo?on compression (interframe) And most compression schemes give you a range of choices in how heavily to compress 37 38 Temporal mo?on compression (interframe) basics Interframe compression more details 39 I Frame (key frame) encoded as s?ll image P Frame/Delta Frame differences from previous frame B Frame a virtual frame that averages difference btwn frames before and ayer image from PeHeri Aimonen 40 Intra Frame Compression Interframe Compression I Frame (or key frame) is only frame that doesn t depend upon other frames to display it, but it takes up the largest amount of storage/bandwidth Need I Frame to recover from problems Intervals btwn I Frames (4, 6, 8,, 150) can be specified in H.264 or FLV formats 41 42 7
Encoding Ar?facts & Related Issues If even one bit goes bad, one can lose signal un?l next key frame (because, with inter frame compression, all the frames before the next key frame only contain par?al informa?on) Browsing will only find key frames Can see a blocking effect of chroma subsampling around the?me of luma interference 43 Coding Ar?facts "Jens Henneman, Performance Art piece by Fred Armisen) care of Peter Oleksik. 720 x 980 HD video, compressed using MPEG2 codec (23.98 fps @ 2 mbit/s), with one pass CBR 44 Coding Ar?facts follow color light flash Coding Ar?facts follow color light flash 45 46 Tamiment 35mm telecined to Digibeta, then transcoded uncompressed into computer. Then, file was compressed using MPEG2 codec (3.7 mbit/s) Aspect Ra?o issues Terry Gilliam s Brazil, digi?zed 4:3 VHS using a DV50 codec s?ll displayed with bad aspect ra?o 47 48 8
Video Digi?za?on & Transfer Frame size & rate Frame size pixel array plus ra?o (4:3, 16:9) Frame rate (29.97 fps, 24 fps, ) Some digital compression & storage formats require a specific frame size 49 50 Bit Rate File Formats Data transfer rate Amount of data per second (coming across a network, or from disk/tape to screen) Usually measured in Mb/sec (8 Mbps, 30 Mbps, 50 Mbps, 200 Mbps) oyen noted as 30 Mbps (uncompressed), or 8 Mbps (compressed) AVI Microsoft format MOV Quicktime format MPEG compressed standardized format RM Real Media compressed format ASF Microsoft streaming format DV25 25 Mbits/sec, fixed compression, 4:1:1 color encoding, 3.6 MB per sec, 60 min. mini-dv 13 GB of storage DV50 50 Mbits/sec, 4:2:2 or 3:3:1 color encoding DV100 100 Mbits/sec, to be used for HDTV production cassette has From Steve Puglia from Adobe s A Digital Video Primer and http:// www.cit.cornell.edu/atc/materials/dig/videoformats.shtml 51 52 Jim Wheeler s list of Digital Video Formats MPEG MPEG 2 currently most commonly used for distribu?on, but difficult to use for produc?on MPEG 4 used with QuickTime and Windows Media files MPEG 7 & 21 metadata standards, not used for compression 53 54 9
Encoding formats MPEG 1, op?mized for CD ROM at 1.5 Mb/sec, 1992, used for VCDs MPEG 2, designed for 10Mb/sec, 1994, used for DVDs and many broadcast tv formats MPEG 4, made for speeds as slow as 64Kb/sec, can separate out s?lls/audio/video, good for anima?on and streaming H.264 (MPEG 4 part 10), used for Blu Ray and some broadcast tv J2k (mo?on JPEG 2000) used in Digital Cinema (DCDM) WMV.mov Flash (SWF), Real, 55 Encoding format general proper?es Frame Integrity DPX2 and J2k encode individual frames as separate files or dis?nct en??es Most versa?le (variety of implementa?on choices) Quick?me and MPEG 4 56 Transcoding Can always be done with enough horsepower But some quality can be lost, depending upon compression, moving btwn colorspaces (RGB and Y CbCr), etc. Streaming Latency is primary issue (which some?mes necessitates caching) Formats include Flash (SWF 7), Real, MPEG 4 57 58 Wrappers Digital Cinema For wrapping both the essence and the metadata about it into the same container Wrappers can either have everything embedded within a single file, or they can have pointers to files outside the wrapper Wrappers include: MXF (broadcas?ng world, can wrap anything) AAF (video edi?ng world, can wrap anything) METS (library world, can wrap anything) Quick?me (can wrap.mov, DV, MPEG 2/4, J2K, Flash) AVI (can wrap DV, J2K, DivX, mp3) ASF (proprietary Microsoy) wrapper for WMV 24 fps 2K (2,000 X 1,500) per frame for theatrical distribu?on 4K (4,000 X 3,000) per frame for master file (36M/frame uncompressed) 59 60 10
How much storage do you need? 1 hour video 640 X 480 24 bits color 25 fps =83 Gbytes Administra?ng Digital Video over?me Preserva?on of digital video files should follow standard repository prac?ces for preserving all types of digital content OAIS compliant repository architecture Metadata wrappers (including preserva?on metadata such as PREMIS that documents each change or reformabng or reinstalla?on or transforma?on) Periodic monitoring of check sums Regular Refreshing Migra?on and/or Emula?on strategies 61 62 Important Guide (1/2) Important Guide (2/2) 63 64 Re/Sources Besser, Howard. Talks, hhp://besser.tsoa.nyu.edu/howard/talks/ Besser, Howard. Introduc?on to Imaging, hhp://www.gehy.edu/research/ publica?ons/electronic_publica?ons/introimages/index.html AIC Guide to Digital Photography and Conserva?on Documenta?on, 2nd Edi?on, hhp://www.conserva?on us.org/index.cfm? fuseac?on=store.viewproduct&product_id=19276&prod_cat_id=13 Fleischhauer, Carl and Caroline Arms, Sustainability of Digital Formats, Planning for Library of Congress Collec?ons: Content Categories Moving Images, hhp://www.digitalpreserva?on.gov/formats/content/s?ll.shtml IMAP/EAI Preserva?on Guide hhp://www.eai.org/resourceguide/ preserva?on.html MIAP Website hhp://www.nyu.edu/?sch/preserva?on/ 65 11