Curriculum Vitae of Zhen Wen

Transcription

1 Curriculum Vitae of Zhen Wen CONTACT INFORMATION RESEARCH INTERESTS EDUCATION 4359 Beckman Institute Voice: (217) (O), (217) (C) University of Illinois at Urbana-Champaign Fax: (217) North Mathews Avenue Urbana, IL 61801, USA zhenwen Multimodal, Intelligent Human Computer Interaction Computer Vision, Visual Motion Tracking/Recognition, Intelligent Video Surveillance Computer Graphics, Virtual Environment Multimedia System Machine Learning and Pattern Recognition, USA PhD in Computer Science, January, summer, 2004 (expected) Dissertation: Face Motion Modeling, Analysis and Synthesis Advisor: Prof. Thomas Huang GPA: 3.97/4.0, USA MS in Computer Science, August, January, 2000 Tsinghua University, Beijing, China BS in Computer Science, September, July, 1998 GPA: 92/100 RESEARCH EXPERIENCE Research Assistant May, Present Worked with Prof. Thomas Huang in the Image Formation and Processing (IFP) group on several research projects, Face Motion Modeling, Analysis and Synthesis: Dissertation research investigates face motion analysis and synthesis, which is important for multimodal Human Computer Interface (HCI) and intelligent video surveillance. There exist mostly two types of approaches: geometric-based methods and appearance-based methods. Geometric-based methods are robust to variations of lighting and subjects. But it is difficult to capture motion details such as wrinkles which are important visual cues for perception. In contrast, appearance-based methods can handle motion details but are not robust to changes of lighting and people. In this thesis, an novel framework is proposed to combine the two types of approaches. On one hand, appearance-based motion model augments geometricbased motion model to deal with motion details. On the other hand, based on the geometric model, we propose adaptation methods to customize appearance model for new conditions. The efficacy of this framework in face motion analysis is demonstrated in a face expression recognition task. Experiments are being done to evaluate the effectiveness of synthetic talking face animation as visual stimuli for enhancing speech understanding in noisy environments. The face synthesis results can be used in audio-visual intelligent hearing aid and psychiatry treatment. Partial results have been presented in ICCV 03, CVPR 03. Synthetic Talking Face for Hearing-aid: In this project, we tried to model and synthesis face motion details and illumination effects. The goal is to improve face animation in terms of lipreadability. It can be used to augment speech-only hearing-aid with visual information to better help hard-of-hearing people in noisy environments. The work is done with UIUC faculties in

2 Speech and Hearing Science. Partial results have been presented in ICME 02, ICIP 02 and CVPR 03. Work is being done to identify and evaluate hypotheses about animation factors, which are important to lip-reading. Synthetic Talking Face for Autism Treatment: The goal of the project is to investigate the use of synthetic visual stimuli to treat autistic children. Synthetic stimuli can be more easily created and manipulated than natural stimuli. The key issue is to make synthetic stimuli as effective as the natural counterpart. Based on perception comparison study, we plan to identify and improve animation parameters critical to psychological perception. The work is done with psychiatry researchers at Univ. of Illinois at Chicago. Synthetic Talking Face for Low Bit-rate Communication: In noisy, dynamic, low-bandwidth environments such battle fields, visual information helps human to better understand speech-only communication. In this project, with Pengyu Hong, we developed iface system to build 3D face avatar for any given people. Then we designed and implemented system for animate the avatar using text, offline speech stream and real-time speech stream. The system can be used augment communication with visual information in a variety of conditions, e.g. text only, one-way speech only, two-way speech only communication. Results were presented in ICME 01, ACM Multmeida 01, IEEE Trans. on Neural Networks and International Journal on Image and Graphics. The system was demonstrated at Army Research Lab FedLab symposium 00 and symposium 01. IBM T.J. Watson Research Center, Hawthorne, NY Research intern May, August, 2003 Worked with Dr. Michelle Kim and other team members at the Multimedia system department on the Knowledge-based Runtime Framework for Adaptive Rich Media Composition project. With more and more diverse environments for rich media provisioning, principled media adaptation engines are demanded. We proposed a formalized framework to design and deploy MPEG-4 based rich media adaptation engine. At runtime, we use IBM policy toolkit to build an efficient engine to map environmental conditions to desired conditions. For design and deploy the policies used by the engine, we use ontology to organize the domain knowledge for media adaptation. For a given media application, applicable knowledge can be retrieved from ontology to compose the policies used by the adaptation engine. The framework is developed under Eclipse 2.0. Results were presented in the IBM summer student poster session and to various IBM researchers in policy toolkit and ontology management. Microsoft Research, Redmond, WA Research intern May, August, 2002 Worked with Dr. Zicheng Liu and Dr. Michael Cohen at the Communication, Collaboration and Signal Processing Group on the Low Bit-rate Face Video Streaming for Face-to-face Teleconference project. Detailed face appearance changes are important visual cues for people to perceive of emotional and cognitive states, but they are usually missing in low bit-rate face-to-face teleconference. We proposed to code face video using spatially varying error criteria, which could ensure smaller error for perceptually more import facial area under low-bit rate. Furthermore, we developed an efficient face synthesis technique to reconstruct face appearance from a small set of face images. This technique helped to further reduce bitrate based on the face that facial motion is highly structured. We compared our system with state-of-the-art coding standard H.26L. Our system achieved similar PSNR, but with higher visual qualities and faster processing speed. A US patent has been filed by Microsoft for this technique. Microsoft Research Asia, Beijing, China Visiting student Nov, Dec, 2001 I worked on learning facial motion models from motion capture data for analysis and synthesis. To enable flexible local motion analysis and synthesis (such as mouth tracking and animation), a parts-based algorithm was used in learning parts-based facial motion model. This work was done

3 in collaboration with Dr. Harry Shum of Microsoft Research Asia. Results were presented in one of chapters in the book 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body, published by Idea Group Inc. Microsoft Research, Redmond, WA Research intern May, August, 2001 Worked with Dr. Zicheng Liu, Dr. Zhengyou Zhang in the Collaboration and Multimedia System Group on A Fast Avatar Building System. The system could construct a personalized 3D face model from a single video stream. The original system was extended to build a full head model instead of only frontal face. Furthermore, the texture variations were modeled based on ratio-image technique and low-dimensionality of Lambertian reflection to create (1) motion details for facial expressions; and (2) lighting change effects. An image editing system is also developed to interactively edit the lighting effects in a single input face image (or texture). Results were presented in CVPR 03. Rockwell Science Center (now Rockwell Scientific Company), Thousand Oaks, CA Research intern May, August, 2001 Worked with Dr. Michael Chan in the Human Computer Interaction Group on Lip Tracking and Animation for Audio-Visual Speech Understanding. I ported a real-time lip tracking and audiovisual speech recognition system from SGI IRIX to Windows NT4.0, using OPENGL and Visual C++. The system was extended to use more image features for recognition. To enhance speech only communication with visual information, I designed and implemented a dynamic programming algorithm to drive a synthetic talking face using lip tracking results. Results were presented in Picture Coding Symposium 01 and Army Research Lab FedLab Symposium 01. TEACHING EXPERIENCE Guest lecturer - ECE 371TSH: Multimedia Signal Processing Spring 2002, 2003 In this graduate/undergraduate course, I gave lectures on face modeling, tracking and animation, and MPEG-4 standard ; designed and graded machine problems. Guest lecturer - ECE 447: Image Processing Fall 2001, 2002, 2003 In this graduate course, I gave lectures on warping, face modeling and tracking, and MPEG-4 standard ; designed and graded machine problems. Teaching assistant - CS 337: Introduction to VLSI System Spring 1999 I setup and maintained course web page; graded home works and exams; held regular office hours. Teaching assistant - CS 362: Logic Design Fall 1998 I setup and maintained course web page; graded home works and exams; held regular office hours. PATENT Pending: A System and Method for Low Bandwidth Video Streaming for Face-To-Face Communication, Microsoft, 2003 PUBLICATIONS BOOK [B1] Zhen Wen and Thomas Huang, 3D Face Processing, in preparation for publication by Kluwer Academic Publishers, expected in REFEREED JOURNAL/BOOK CHAPTER ARTICLES

4 [J1] Zhen Wen, Pengyu Hong, Jilin Tu and Thomas S. Huang, Learning 3D Face Deformation Models: Methods and Applications, to appear in the book 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body, Idea Group Inc, [J2] Pengyu Hong, Zhen Wen, and Thomas S. Huang, Speech driven face animation, in the book MPEG-4 Facial Animation - The standard, implementations, and applications, John Wiley & Sons, [J3] Pengyu Hong, Zhen Wen, Yang Li, Thomas S. Huang, Avatars, in the book Computer Science Handbook of the Army Research Lab ADID FedLab, [J4] Pengyu Hong, Zhen Wen, and Thomas S. Huang, Real-time speech driven expressive synthetic talking faces using neural networks, in IEEE Transaction on Neural Network, April [J5] Pengyu Hong, Zhen Wen, and Thomas S. Huang, iface: A 3D SYNTHETIC TALKING FACE, International Journal of Image and Graphics, vol. 1, no. 1, pp. 1-8, REFEREED CONFERENCE PAPERS [C1] Zhen Wen and Thomas Huang, Capturing Subtle Facial Motions in 3D Face Tracking, in Proc. of International Conference on Computer Vision (ICCV), page , Nice, France, (acceptance rate: 200 out of 1000, 20%) [C2] Zhen Wen, Zicheng Liu and Thomas Huang, Face Relighting with Radiance Environment Maps, in Proc. of International Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, page , Madison, WI, USA, (acceptance rate: 24%) [C3] Jilin Tu, Zhen Wen, Tao Hai, and Thomas Huang, Coding Face at Very Low Bit Rate via Visual Face Tracking, in Proc. of Picture Coding Symposium 03, page , Saint Malo, France, [C4] Zhen Wen, Thomas Huang and Zicheng Liu, On Recovering Detailed Face Deformation under General Lighting Using Height from Shading, in Proc. of International Conference on Multimedia and Expo (ICME), vol 1, page , Lausanne, Switzerland, [C5] Zhen Wen, Thomas Huang and Zicheng Liu, Model-based Face Image Coding Using Spherical Harmonics, in Proc. of International Conference on Image Processing (ICIP), vol.1, page , Rochester, NY, USA, [C6] Pengyu Hong, Zhen Wen and Thomas Huang, An Integrated Framework for Face Modeling, Facial Motion Analysis and Synthesis, in Proc. of ACM Multimedia, Ottawa, Canada, [C7] Zhen Wen, Pengyu Hong and Thomas Huang, Real Time Speech Driven Facial Animation Using Formant Analysis, in Proc. of International Conference on Multimedia and Expo (ICME), Japan, [C8] Pengyu Hong, Zhen Wen and Thomas Huang, Real-time Speech Driven Avatar with Constant Short Time Delay, in Proc. of International Conference on Augmented, Virtual Environments and 3D Imaging, Greece, [C9] Zhen Wen, Michael Chan and Thomas Huang, Face Animation Driven by Contour-Based Visual Tracking, in Proc. of the 22nd Picture Coding Symposium, page , Korea, April [C10] Pengyu Hong, Zhen Wen, Thomas Huang and Michael T. Chan, Speech Driven Avatars, Army Research Lab Symposium, College Park, MD, USA, 2001.

5 HONORS AND AWARDS INVITED TALKS Travel grant to IEEE CVPR 2003 The Forth Place at the V. Dale Cozad Business Plan Competition, 2001 (with Pengyu Hong, Danfeng Li, and Zhaoyu Liu) Certificate of Appreciation for superior performance in support of the Army Research Laboratory in the 4th Annual FedLab Symposium, 2000 Samsung scholarship to best undergraduate student, 1998, Tsinghua University Outstanding undergraduate scholarships, , Tsinghua University [T1] Invited lectures on Neural Networks and Face Analysis and Synthesis (with Prof. Tom Huang), Tsinghua University, Beijing, China, November [T2] Combining Shape and Texture for Facial Motion Analysis and Synthesis, NEC Labs America, San Jose, CA, September [T3] 3D Face Modeling, Analysis and Synthesis, Microsoft Research Asia, Beijing, China, November [T4] 3D Face Modeling, Analysis and Synthesis, Tsinghua University, Beijing, China, November SOFTWARE Real-time speech driven avatar: A software to animate 3D face model in real-time based on input speech stream. It can be used to enhance interaction-centric applications, such as video games, with talking characters. This software is developed using OpengGL and Microsoft Visual C++. The source codes were released to Industrial Technology Research Institute of Taiwan. The software was demonstrated and evaluated by Microsoft Xbox team. SERVICE Reviewer, Proceedings of IEEE special issue on Multimodal Human Computer Interface, 2003 Reviewer, Computer Vision and Image Understanding, special issue on face recognition, 2003 Reviewer, EURASIP Journal on Applied Signal Processing Reviewer, Journal of Visual Language and Computing Reviewer, International Conference on Multimodal Interfaces 2003 (ICMI-PUI 2003) Reviewer, ACM SIGGRAPH Symposium on Computer Animation (SCA) 2002 Reviewer, International Conference on Computer Vision (ICCV) 2001 Reviewer, International Conference on Multimedia and Expo (ICME) 2000, 2001 Reviewer, International Conference on Image Processing (ICIP) 2000, 2001 PERSONAL INFORMATION Citizenship: P. R. China (F1-visa) REFERENCES Professor Thomas S. Huang (advisor) Dr. Zicheng Liu ECE Department Researcher University of Illinois at Urbana-Champaign Microsoft Research 405 N. Mathews Ave One Microsoft Way Urbana, IL Redmond, WA (217) (425) huang@ifp.uiuc.edu zliu@microsoft.com

6 Dr. Zhengyou Zhang Dr. Michelle Kim Senior Researcher Manager, Composite Media Group Microsoft Research IBM T.J Watson Research One Microsoft Way, 19 Skyline Dr. Redmond, WA Hawthorne, NY (425) (914) Dr. Michael T. Chan Researcher GE Global Research One Research Circle, KW-C220A Niskayuna, NY (518)