Timesys University. Learn Embedded Linux by Building a Video Surveillance Device

Transcription

1 Timesys University Learn Embedded Linux by Building a Video Surveillance Device 1

2 2 Timesys Staff Answering your questions today are: Maciej Halasz Product Management Al Feczko Technical Pre-Sales/Sales Theresa Kisha Marketing

3 3 We Are Building a Video Surveillance Device! More than one third of all shops are under video surveillance 3

4 4 Building a Video Surveillance Device with LinuxLink Agenda Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development environment with the matching SDK Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using Qt Embedded for Linux Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording under Linux Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot and quick deployment 4

5 5 Module 1 Your Development Environment Embedded Linux Primer Where do I start my Linux design Product Requirements Assemble a custom BSP using the LinuxLink FREE Edition Web Interface Reflect product requirements Produce complete Linux design Learn how to find information Application/System Development environment setup with an SDK Deploy the images on the Phytec Vybrid system 5

6 6 Your Development Environment: Session Hardware/Software Requirements 6

7 7 What We Need to Build Our Product A Host Machine Linux is the best any recent version is ok Windows is fine, but you ll need a VirtualBox with a Linux installation Cross-development environment Linux source code for the product Bootloader Linux kernel APIs Various Linux utilities e.g. mkimage IDE with WYSIWYG for faster development A Development Kit 7

8 8 Your Workstation Setup Ethernet/Serial USB Development Environment Cross tools IDE Exercises 8

9 9 Having Fun Yet? Question bucket We will ask some review questions at the end of each Module. In return for the first correct answer for each question asked, we will award a giveaway item. 9

10 10 Embedded Linux Primer 10

11 11 Complete Software Solution for Vybrid Qt Application PEG Application Repository Docs Support Demos App on Linux App on Linux App on MQX JTAG DS5 Development & Debug Tools Multi OS Configuration & Build Tool Linux Kernel Device MQX Drivers Message passing Multi OS Infrastructure Bootloader Tower Modules Device Drivers Support Cortex A5 Cortex M4 11

12 12 Embedded Linux Reference Model Security Surveillance User Interface, Buttons, Stream sel, etc Auto-launch Camera Network control interface Sensor Control 3 Middleware openssh shell Qt Embedded setup scripts setup script networking MCC API LCD calibration sysfs Wireless tools 2 Linux kernel Driver Driver Splash Screen Driver Driver Driver Shared Memory 1 U-Boot bootloader Driver Driver Vybrid LCD Touch Screen Button Ethernet Memory Serial XIP USB SDIO 12

13 13 MQX Reference Model Application (s) Network control interface Sensor Control 2 Functional Blocks setup scripts setup script networking MCC API File system MQX kernel Driver Driver Services Driver Driver Driver Shared Memory 1 Bootloader Driver Driver Vybrid LCD Touch Screen Button Ethernet Memory Serial XIP USB SDIO 13

14 14 Linux and MQX strengths Linux MQX Core footprint 1.5M <100k Scheduler Kernel Configurability Priority, preemptive, latency not guaranteed Modular, fully customizable Priority, preemptive, latency guaranteed Modular, fully customizable APIs 40,000+ Limited System protection Separate address spaces Kernel/Application separated Single address space; no protection Kernel/Application 14

15 15 Vybrid Solution Approach: LinuxLink Tower MQX Linuxlink cloud development allows customers to rapidly deploy pre-built or custom BSP/SDKs for Tower or other hardware platforms. 15

16 16 Embedded Linux Challenges mainline 1000s of Linux packages Toolchain GCC4.1? uclibc or glibc? Linux kernel APIs/libs Debug Additional drivers? optimizations Ver.3.10? GStreamer Ver. 0.22? Private plugins Custom software gstreamer.c om Embedded? Version? Need HW acceleration? Which modules? Open Source plugins Third party Development Ready When? For 2010, 57% of all projects finished late or were cancelled Source: EE Times Embedded Market Study,

17 17 Use Timesys If You Have an aggressive schedule. Have an understaffed engineering team. Need proven middleware, configured for your product. Need help from a real person when you re stuck. Highly customizable embedded distribution GCC4.1? uclibc or glibc? APIs/libs Dependencies 1000s of packages Linux kernel Toolchain Optimizations drivers versions Debug Open Source plugins Third party Expert Support Your custom embedded Linux platform Professional Services (optional) Development Ready 17

18 18 RTOS Challenges Future Proofing new/future protocols (Stacks) Bluetooth Light Peripheral Support across product families Prototyping and Evaluation support Creating drivers to use with a scheduler Would like to spend your time developing application code and not creating or maintaining a scheduling system Multi-thread support with synchronization Portability of application code to other CPUs Resource handling Adding new features without affecting higher priority functions Support for upper layer protocols such as: TCP/IP, USB, flash systems, web servers CAN protocols, embedded GUI, SSL, SNMP Determinism 18

19 19 MQX Solutions Established community of users and code examples MQX is a 15-year-old RTOS Solid RTOS with extensive code examples and community support Standardized Driver creation Support across all Freescale MCU Portfolio Extensive Stack Library Extensive third-party community development support End User empowered to create custom BSP Non Royalty RTOS Ported Tower Systems for easy system lever evaluation 19

20 20 Product Requirements Where Do I Start My Linux Development? 20

21 21 Requirements First Glance Video Surveillance Features: GUI with the following features: Live feed screen Recorded feed management screen Control panel screen Detect movement via light sensor Use Multi-core communication (MCC) layer Trigger video recording Capture video feed from USB connected cameras Store on a local file system Manage recordings Utilize a video overlay See video feed and controls on a single screen Display text on a video feed 21

22 22 Security Surveillance Device (Blueprint) Security Surveillance Device User Interface, Buttons, Stream sel, etc Auto-launch Video playback Camera Capture Network control interface Light Sensor Middleware setup script openssh shell Qt Embedded LCD calibration setup scripts GStreamer Codecs networking Wireless tools MCC API Linux kernel Driver Driver Splash Screen Driver Driver Driver Shared Memory U-Boot bootloader Driver Driver Vybrid LCD Touchscr een VPU IPU Ethernet Memory Serial NAND USB SDIO 22

23 23 Exercise 1: Assemble a Custom BSP Using the LinuxLink FREE Edition Web Interface 23

24 24 LinuxLink Architecture 24

25 25 Exercise Details Identify the initial set of requirements: All needed software to simply boot a reference board Build a BSP for development Build a custom BSP and a matching SDK Use LinuxLink FREE Edition Web Interface Start with a small Linux design, and build it up per your requirements Provides you with a small embedded design with only the features you need 25

26 26 Requirements Helper Feature What I need Where to select Linux kernel Latest, 3.x kernel Linux kernel Cross-Toolchain GCC, Binutils, GDB Toolchains C Library glibc Toolchains GStreamer gst-plugins-good Packages Font freetype Packages Qt qt-embedded-linux Packages Networking openssh Packages System initialization busybox Packages 26

27 27 Account Login info We have accounts setup specifically for Timesys University Login with the following credentials: U: labuserx where X is 1-20 P: lab123 You can register for a free personal account 27

28 28 Application/System Development Environment Setup with an SDK 28

29 29 Embedded Linux Reference Model Application System Hardware Software Application (e.g. Base Station Control) Application 1 Application 2 Middleware and Application Services Networking Security UI Reliability LCD Networking Linux kernel Driver Driver Driver Driver more Board SOC Virtualization Target Reliability HW accel. App Features Third-Party Application Hundreds of APIs from Open Source and Proprietary SD USB more Host Development Tools Cross-Development Environment (SDK) IDE Application Debugging Build System (e.g. Factory) RFS, Kernel, App Collaborative Development Profiling Tracing System Management KGDB/ printk Debugging JTAG SMP support, RT enhancements Target Images Binaries Deployment Target Management 29

30 30 Embedded Linux Reference Model Target Host Application System Hardware Software Application (e.g. Base Station Control) App Features Development Tools Application 1 Application 2 Middleware and Application Services Networking Security UI Reliability LCD Networking Linux kernel Driver Driver Driver Driver more Board SOC Reliability HW accel. Cross-Development Third-Party Environment (SDK) IDE Application Application Debugging Build System (e.g. Factory) RFS, Kernel, App Collaborative Hundreds Development of ProfilingAPIs from Open Source Tracing and System Proprietary Management KGDB/ printk Debugging JTAG SMP support, RT enhancements Target SD Images USB more Binaries Deployment Target Management Virtualization 30

31 31 Development Environment Setup BSP/SDK Summary Software Development Kit (SDK) Install on your host PC Can find instructions on how to setup the board Desktop Factory tools Install on your host PC 31

32 32 Exercise 2: Setup Your Application Development Environment 32

33 33 Exercise Details Setup your own development environment Cross-toolchain IDE Libraries and header files The goal Easily develop your value-add software 33

34 33 Exercise Details Setup your own development environment Cross-toolchain IDE Libraries and header files The goal Easily develop your value-add software Answer: 33

35 33 Exercise Details Setup your own development environment Cross-toolchain IDE Libraries and header files The goal Easily develop your value-add software Answer: + 33

36 33 Exercise Details Setup your own development environment Cross-toolchain IDE Libraries and header files The goal Easily develop your value-add software Answer: + TimeStorm IDE 33

37 34 Exercise Details Install an SDK from a build output Installation command (optional): sh <SDK name.sh> For your convenience the BSP/SDK has been downloaded for you and preinstalled under: /home/tsu-lab/lab-5/bsp Look at a directory structure Verify toolchain with command: <fully qualified path to toolchain s gcc> -### 34

38 35 Deploying Linux Using NFS 35

39 36 Is NFS useful? Run the custom BSP on the target U-Boot bootloader Linux kernel Network mounted RFS The goal Easily modify all aspects of your RFS Easily deploy and test your application 36

40 37 Network File System setup Host Ethernet Serial Target NFS server DHCP server (local subnet) TFTP server (/tftpboot) Console U-Boot Console Host RFS Target RFS Target RFS 37

41 38 NFS Setup Details Services already preset for you NFS server DHCP server (local subnet) TFTP server (/tftpboot) Timesys offers getting started guides to walk you through the process Setup U-Boot environment to load/run a Linux kernel > setenv bootfile uimage //image to load > setenv ipaddr //your local IP > setenv serverip //server IP > saveenv //store variables in NAND > Setenv rootpath <rfs full path> //specify path to nfs exported rfs > run nfs boot //boot the image using pre-set command Full directions can be found in the Getting Started Guide for the Phytec Vybrid board. Access it through a LinuxLink dashboard. 38

42 39 Module 1 Summary Learned about the Vybrid LinuxLink needed for all exercises Reflected initial product requirements in Linux BSP and SDK Built a custom BSP with LinuxLink FREE Edition Experiment with a pre-built starting point Setup a development environment System level development and optimizations Development of a value-add software (applications) Discussed NFS deployment Image transfer Bootloader configurations 39

43 40 Module 1 Quiz 1. What do you need to know before you start your initial design? 2. Why would you be interested in Linux kernel updates? 3. What are the benefits of using LinuxLink? (name at least 1) 4. How soon can you start your application development? 40

44 41 Building a Video Surveillance Device with LinuxLink Agenda Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development environment with the matching SDK Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using Qt Embedded for Linux Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording under Linux Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot and quick deployment 41

45 42 Module 2 Modify the underlying Linux image to support Qt Qt development tools installation/setup Coffee Vending Machine GUI development Create a QMainWindow application Use a number of Qt widgets including QPushButton QLabel Layouts and more Add a custom widget for the clock Create a resource file Test your application locally Cross-compile for the Vybrid Phytec target Debug on the target 42

46 43 Security Surveillance Device (Blueprint) Security Surveillance Device User Interface, Buttons, Stream sel, etc Auto-launch Video playback Camera Capture Network control interface Light Sensor Middleware setup script openssh shell Qt Embedded LCD calibration setup scripts GStreamer Freescale Codecs networking Wireless tools MCC API Linux kernel Driver Driver Splash Screen Driver Driver Driver Shared Memory U-Boot bootloader Driver Driver Vybrid LCD Touchscr een VPU IPU Ethernet Memory Serial NAND USB SDIO 43

47 44 Adding APIs/Libraries Using the LinuxLink PRO Desktop Interface 44

48 45 LinuxLink Value Proposition Advanced Customization (all aspects of your Linux platform) Factory Only easy to use, innovative Build System with: Advice engine recommends software based on your hardware and application requirements Update engine automatic updates for your custom configuration Build up vs. strip down to small distribution Affordable costs ~40% less Build Repeatability Easy Extensibility 45

49 Desktop Factory 46

50 47 Factory Architecture The Desktop Factory is the Timesys build engine with the menuconfig UI. The UI creates/edits the Factory work-order in a.config file. The Factory fetches source code for components required by the work order, and follows recipes to cross build those components. Each component has its own recipe, embody in a pair of files which drive the UI and the make build Timesys Corp. 47

51 48 Exercise 3: Basic UI Basic navigation, Searching, Help, String entry Source code locations Running parallel make jobs Package options Templates A peek at the.config file 2013 Timesys Corp. 48

52 49 Factory: High-Level Flow At a high level, the Factory flows through these five stages. However, due to dependencies that require components to be built out of order, the Factory operates in a recursive manner. 1. Fetch Sources, Patches, Additional files 2. Build Host Tools and Libraries (toolchain) 3. Build Target Binaries (Root File System) 4. Package binaries 5. Create Installers 1. All sources, patches, and other files (e.g., startup scripts) are fetched from the designated source (typically Timesys Repository). 2. The host tools are built including ccache, and the gcc cross compiler suite (glibc or uclibc, binutils, etc.). Any additional host utilities are built as well. Simply stated, everything that will execute on the x86 host are built here. 3. All target software is built. The kernel image and RFS image are created here as well. 4. As requested (e.g., RPM, deb), binaries are packaged. 5. Installers (e.g. SDK) are created Timesys Corp. 49

53 50 Factory: Directory Tree factory bin doc include src target toolchain build Makefile Config.in.config./bin/ - contains scripts useful for extending Factory../doc/- contains license and version information../include/ - contains common recipes along with the KConfig library../src / contains the downloaded sources../target/ - contains sub-directories containing recipes for the kernel, boot-loaders, and RFS../toolchain/ - contains the recipes for each toolchain component as well as the host utilities that are built with the toolchain../build/ the ouput directory tree. Makefile includes all other make rule files within the Factory tree Config.in - includes all other Config.in files and sets global and default options..config - the generated work order used by the make rules Timesys Corp. 50

54 51 Factory: Build Output Tree Build-{ } toolchaininitial{ }tgz images packages toolchain rfs <pkg name> toolchainfinal-{ }tgz { }.sh NOTE: { } in names imply architecture or other build specific attribute. toolchain-initial-{ }.tgz - This is a bare toolchain (cross compiler, etc.) toolchain-final-{ }.tgz - Bare toolchain plus all selected package libraries. { }.sh - SDK installer./images/ - contains kernel, bootloader, and RFS images./packages/ contains binary packages in RPM, DEB, IPKG, or TGZ../<package name>/ - each package has its own working directory../toolchain/ toolchain working directory./rfs/ - RFS working directory. This can be modified by hand and then the RFS images can be recreated with the 'rfs-images' target. NOTE: this directory cannot be directly NFS exported as it does not have valid device files (/dev). The device files are populated during the RFS image creation process Timesys Corp. 51

55 52 make Targets Make Targets checksystem, checktool-oprofile, checktool-ltp, checktool-mpatrol, checktool-gdb advice, showpackages, showoutput checkupdates, update, updatenoprompt, oldconfig Clean, download-clean, distclean, rfsdistclean, packages-distclean, software-distclean, reset-toolchain -clean, -distclean, -fetch, -unpack, - patch, -configure, -build, -package, - rfs-install, -host-install, -show menuconfig, config Functionality System requirements and compatibility checks. Information commands Updating Clean commands Per-application targets (all preceded by application name without version e.g. busybox-clean) workorder (.config) editors 2013 Timesys Corp. 52

56 53 Exercise 4: Enhance Your Design Locally by Adding Required Packages 53

57 54 Exercise Details Find Desktop Factory under: /home/tsu-lab/lab-5/factory/factory-<string> Look at the help screen by running make help Bring up a configuration screen make menuconfig Locate and select missing packages 54

58 55 Requirements Helper Feature What I need Where to select Linux kernel Latest, 3.x kernel Linux kernel Cross-Toolchain GCC, Binutils, GDB Toolchains C Library glibc Toolchains GStreamer gstreamer Packages gst-plugins-good Packages MCC mcc-kmod, libmcc Packages Video 4 Linux v4l-utils Packages Font freefont Packages Text display pango Packages Qt qt-embedded-linux Packages Networking openssh Packages System initialization busybox Packages 55

59 56 Qt Development Tools Installation/Setup 56

60 57 Qt Development Tools Options DS5 with TimeStorm & Qt plugins Qt Creator Installation DS5 can be downloaded from ARM Download TimeStorm and Qt plugins from LinuxLink Preinstalled in your UBUNTU Run Desktop icon for ARM DS5 57

61 58 IDE and Cross-Environment Host PC/Linux IDE Recognizes Automatically LinuxLink Desktop Interface 58

62 59 Security Surveillance Device GUI Development 59

63 60 The Qt Story The Qt toolkit is a multi-platform C++ GUI toolkit (class library) Developed by Troll Tech AS, later acquired by Nokia Qt is licensed under dual license: LGPL Commercial (mostly if you plan to extend Qt and sell it) Around 1997, Qt was chosen as the code basis for the KDE Linux desktop environment Latest version Qt 5.0 is available for Windows, Unix, Linux, Embedded Linux, Mac OS X libraries and Symbian OS Recent additions include: Qt Quick (programming with QML) Qt Mobility Target form factor simulator (cell phones) 60

64 61 Qt/Embedded (cont d) Modular approach Each additional functionality can be enabled by specifying module to be added Core classes provide the baseline GUI building tools Can run with or without X Can be easily ported from other OSs 61

65 62 Qt Flexible and Powerful UI API Used in many applications in various markets WYSIWYG approach Can take advantage of Freescale hardware: OpenGL ES API Multiple frambuffers 62

66 63 Widgets Base class for all UI widgets QPushButton, QLineEdit, QTabView, Properties width, height, backgroundcolor, font, mousetracking, backgroundpixmap, etc. Slots repaint, show, hide, move, setgeometry, setmainwidget, etc. Signals mousemoveevent, keypressevent, resizeevent, paintevent, enterevent, leaveevent, etc. 63

67 64 Event Handling Qt approach to IPC: signals and slots A widget sends out various signals Object methods can be declared as slots Compatible signals and slots can be connected or plugged together (parameter types must match!) Clear UI/Logic separation This separation between UI components and program elements lends itself to component-based programming 64

68 65 Signals and Slots 65

69 66 Signals and Slots (cont.) #include <QObject> class Counter : public QObject { Q_OBJECT public: Counter() { m_value = 0; } int value() const { return m_value; } public slots: void setvalue(int value); signals: void valuechanged(int newvalue); private: int m_value; }; 66

70 67 Events Triggered by Signals Signals: emit events declare as signals You don't implement them, you send them with the keyword emit e.g. emit(counteratzero(100)) Slots: receive and handle events Normal member functions declared as slots Connect: must connect signals to slots QObject::connect( Object 1, SIGNAL(activated(int)), Object 2, SLOT(handlingMethod(int)) ); 67

71 68 qmake The qmake utility is typically invoked with the following commands: qmake project qmake IDE tools invoke qmake automatically You can invoke it as well (Project->qmake) Rules: Be sure to place code in its own directory. qmake scans all subdirectories for dependencies. Looks at the.pro file and generates Makefile structure 68

72 69 Qt Application Structure.pro file Holds information on all files that will be used to build a predefined executable(s) Used by qmake to generate Makefiles for your project.ui file User Interface file Can open in a designer view Has canvas on which you can drag and drop widgets Its XML format is translated into C file (moc_) at compile time.qrc file Holds information on all resources (non C++ code) used Referenced throughout the project e.g. QPushButton icons.qss file Used to define a style sheet for a custom look and feel Loaded at runtime by your application 69

73 70 Exercise 5: Create a GUI application with DS5 &TimeStorm/Qt plugins 70

74 71 Video Surveillance Application Start DS5 Desktop icon Open a Workspace /home/tsu-lab/ds5-workspace-clean Create a new Qt GUI Project UI Type QMainWindow Don t need to select any additional Qt frameworks (initially) Create the following components Remove menu bar and status bar Use layouts Use a QTabView Example can be found in Exercise 5 folder in downloads. Preset the size of all UIs to selected LCD size (will help you see actual design) Design UIs on all three screens 71

75 72 Exercise 6: Develop Logic Behind the GUI 72

76 73 Exercise Details: Video Surveillance Qt Application Define/Connect signals (Hide Button) Add code for Hide Button Capture the signal in a service class Implement the service routine Set alpha blending to 0 Switch the building layout graphic Utilize the border-image property of widgets style sheet Compile and run your application on your host Cross compile and debug your application on the target 73

77 74 Module 2 Summary Used LinuxLink PRO Edition Desktop Factory interface Added needed packages and rebuilt the runtime BSP and SDK images Used an IDE to design/develop Qt based UI Developed a GUI with the following: Layers, several Widgets and Labels Implemented signals and handlers Cross-compiled for Phytec s Vybrid target 74

78 75 Module 2 Quiz 1. Name at least one graphics package (other than QT)? 2. What tools can you use to develop with Qt? 3. How do you trigger cooperation between objects in Qt? 4. What is the purpose behind using Layers? 75

79 76 Building a Video Surveillance Device with LinuxLink Agenda Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development environment with the matching SDK Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using Qt Embedded for Linux Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording under Linux Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot and quick deployment 76

80 77 Module 3 GStreamer Primer MQX overview Pingpong example Multi-Core Communication (MCC) Layer Purpose Setup Add MCC communication in Security Surveillance Application Add appropriate code to the UI application 77

81 78 Security Surveillance Device (Blueprint) Security Surveillance Device User Interface, Buttons, Stream sel, etc Auto-launch Video playback Camera Capture Network control interface Light Sensor Middleware setup script openssh shell Qt Embedded LCD calibration setup scripts GStreamer Codecs networking Wireless tools MCC API Linux kernel Driver Driver Splash Screen Driver Driver Driver Shared Memory U-Boot bootloader Driver Driver Vybrid LCD Touchscr een Framebuffer IPU Ethernet Memory Serial NAND USB SDIO 78

82 79 GStreamer Primer 79

83 80 GStreamer: Element Is the most important class of objects in GStreamer. You will usually create a chain of elements linked together and let data flow through this chain of elements. It has one specific function, which can be the reading of data from a file, decoding of this data or outputting this data to your sound card (or anything else). By chaining together several elements, you create a pipeline that can do a specific task, for example media playback or capture. GStreamer ships with a large collection of elements by default, making the development of a large variety of media applications possible. If needed, you can also write new elements!!! Examples: v4l2src filesrc oggdemux alsasink v4lsink 80

84 81 GStreamer: Pads Pads are element's input and output, where you can connect other elements. They are used to negotiate links and data flow between elements in GStreamer. A pad can be viewed as a "plug" or "port" on an element where links may be made with other elements, and through which data can flow to or from those elements. Pads have specific data handling capabilities: a pad can restrict the type of data that flows through it. Links are only allowed between two pads when the allowed data types of the two pads are compatible. 81

85 82 GStreamer: Pads Data types are negotiated between pads using a process called caps negotiation. For the most part, all data in GStreamer flows one way through a link between elements. Data flows out of one element through one or more source pads, and elements accept incoming data through one or more sink pads. Source and sink elements have only source and sink pads, respectively. Dynamic vs. on Request pads On Request pad examples: multiplexers, aggregators, tee elements. Aggregators (Mix) are elements that merge the content of several input streams together into one output stream. Tee elements are the reverse: they are elements that have one input stream and copy this stream to each of their output pads, which are created on request. Whenever an application needs another copy of the stream, it can simply request a new output pad from the tee element. 82

86 83 GStreamer: Pipelines A pipeline is a special subtype of a bin that allows execution of all of its contained child elements. As you set a pipeline to PAUSED or PLAYING state, data flow will start and media processing will take place. Once started, pipelines will run in a separate thread until you stop them or the end of the data stream is reached. 83

87 84 GStreamer: Communication buffers objects for passing streaming data between elements in the pipeline. Buffers always travel from sources to sinks (downstream). events objects sent between elements or from the application to elements. Events can travel upstream and downstream. Downstream events can be synchronized to the data flow. messages objects posted by elements on the pipeline's message bus, where they will be held for collection by the application. Messages can be intercepted synchronously from the streaming thread context of the element posting the message, but are usually handled asynchronously by the application from the application's main thread. Messages are used to transmit information such as errors, tags, state changes, buffering state, redirects etc. from elements to the application in a thread-safe way. Queries allow applications to request information such as duration or current playback position from the pipeline. Queries are always answered synchronously. Elements can also use queries to request information from their peer elements (such as the file size or duration). They can be used both ways within a pipeline, but upstream queries are more common. 84

88 85 GStreamer: Architecture Divided into packages Baseline Plugins Third-party Third-party plugins can take advantage of hardware- specific acceleration Developed by Freescale Integrated and available in LinuxLink 85

89 86 GStreamer Command Line Tools Embedded player gplay Creates its own pipeline based on file extension Execute a pipeline gst-launch Learn more about an element capability gst-inspect Provides information on all available gstreamer elements Helps you find out specific element capabilities # gst-inspect grep enc # gst-inspect ffdec_mpeg4 86

90 87 GStreamer Playground Video playback # gst-launch videotestsrc! fbdevsink device=/dev/fb2 Webcam live stream # gst-launch v4l2src! fbdevsink device=/dev/fb2 # gst-launch v4l2src device=/dev/video0! fbdevsink device=/dev/fb2 87

91 88 GStreamer Playground Webcam capture # gst-launch v4l2src! fbdevsink device=/dev/fb2 # gst-launch v4l2src device=/dev/video0! ffenc_mpeg4! matroskamux! filesink location=test.file Text overlay # gst-launch v4l2src device=/dev/video0! textoverlay text= Hello! fbdevsink device=/dev/fb2 Clock overlay # gst-launch v4l2src device=/dev/video0! clockoverlay halign=right valign=bottom shaded-background=true time-format="%y.%m.%d"! fbdevsink device=/dev/fb2 Playing and recording at the same time # gst-launch v4l2src device=/dev/video0! clockoverlay halign=left valign=top time-format="%y/%m/%d %H:%M:%S"! tee name="splitter"! queue! fbdevsink device=/dev/fb2 sync=false splitter.! queue! ffenc_mpeg4! matroskamux! filesink location=recording.file 88

92 89 Exercise: GStreamer Pipeline 89

93 90 Exercise Details: Capture Camera Feed with GStreamer Verify available codecs gst-inspect gst-inspect grep mux Play camera feed on HDMI output Components to be used gst-launch v4l2src (camera on /dev/video0) fbdevsink device=/dev/fb2 90

94 91 GStreamer Language Bindings Language Python Perl.NET C++ Qt Android Guile Haskell Java Ruby Vala Status Released. See the gst-python module. Released as part of the gtk2-perl project. The bindings are still under development. See the gstreamersharp module. Released. See the gstreamermm GNOME module. Released. See the qt-gstreamer module. Under development. See the gstreamer android module. Released under the aegis of the guile-gnome project. See Haskell.org. Released as part of the gstreamer-java project. The Ruby bindings are released as part of Ruby-GNOME2. Released as part of Vala. 91

95 92 MQX & MCC Overview 92

96 93 MQX Reference Model Application (s) Network control interface Sensor Control 2 Functional Blocks setup scripts setup script networking MCC API File system MQX kernel Driver Driver Services Driver Driver Driver Shared Memory 1 Bootloader Driver Driver Vybrid LCD Touch Screen Button Ethernet Memory Serial XIP USB SDIO 93

97 94 OS Comparison (for the purpose of training) MQX O/S components are libraries linked with your application to create a single bare-metal execution image. Source distributed as DS-5 (Eclipse) projects in a single source tree. Built with ARM bare-metal compiler. (advanced) A single address space no memory protection, mapping, or transformations between hardware I/O, O/S components, application. Programs are tasks each in their own thread. Linux Kernel is a bare-metal executable, but isn t enough. Your application (except possibly device drivers) is built separately and runs (launched) from a file system. Therefore a root file system (RFS) is a required component of Linux. Timesys Factory (on web or desktop) builds the RFS. It will build the Kernel and your Application, but they can be also built outside as well. Built with (Timesys) gcc compiler. (advanced) Linux utilizes the memory management unit (MMU) of the A5 core. All processes are hardware protected from stepping on each other s memory. Programs run as a process which can have many threads Timesys Corp. 94

98 95 Overview of MCC Multi-Core Communication Communication layer between MQX and Linux Lightweight, fast API calls are simple send / receive between endpoints Endpoints are [core, node, port] triplets Sort of IP host, address, port but not really. Core is fixed (A5=0, M4=1] Linux can have multiple nodes, MQX only 1 Each can have multiple endpoints, arbitrary numbering Uses shared SRAM and interrupts Received data can be passed by pointer or copied Variable timeouts Configurable: Buffer sizes # of buffers Max # of endpoints 2013 Timesys Corp. 95

99 96 MCC Architecture A5 - Linux M4 - MQX User App process(es) API shared library Kernel Module Driver Interrupts Hardware Semaphore User App (tasks) API library Endpoint 0,0,1 Endpoint 0,2,3 Endpoint 1,0,1 Endpoint 1,0,2 Free buffer pool Shared SRAM 2013 Timesys Corp. 96

100 97 Simple Example A5 - Linux 1. I will receive on endpoint [0,0,1] 2. mcc_initialize(0) 3. mcc_create_endpoint([0,0,1], 1) 4. mcc_send([1,0,2], hello, 5, 5000) // <= 5 secs for buffer 5. mcc_recv_copy([0,0,1], &buf, sizeof(buf), length, 0xffffffff) M4- MQX 1. I will receive on endpoint [1,0,2] 2. mcc_initialize(0) 3. mcc_create_endpoint([1,0,2], 2) 4. mcc_recv_nocopy([1,0,2], &buf_p, length, 0xffffffff) 5. mcc_send([0,0,1], hello, 5, 5000) // <= 5 secs for buffer 2013 Timesys Corp. 97

101 98 Resource Sharing M4 MQX pingpong/ Photosensor example MCC A5 Linux boa webserver Sensor demo Security Surveillance App theater-vybrid Vybrid SOC I2C Bus X Z Y Photosensor Phytec-LCD Touchscreen 2013 Timesys Corp. 98

102 99 Protecting I2C Sequences MQX Application 1. Open the I2C driver a) Disable and clear I2C b) Clear out all I2C events c) Set the station address d) Set the frequency e) Enable I2C 2. Set master mode 3. Set the device address 4. Do read/write sequence 5. Send out stop 6. Close the driver. Must be protected in application. Hardware semaphore protection. Linux Driver 1. Enable I2C controller 2. Set master mode, Interrupts 3. Set the device address 4. Wait for interrupt 5. Do read/write sequence including wait for interrupt 6. Send out stop 7. Disable I2C controller Protected in driver. Transparent to application Timesys Corp. 99

103 100 A5/M4 Boot Process TIME U-Boot Reset, copy U-Boot to SDRAM and jump to start address Basic System Init. (IMPORTANT) Copy the Linux kernel Image to SDRAM from SD Decompress the kernel if needed Jump to upload address and start the kernel Kernel Run Kernel Init code Init kernel subsystems (device drivers) Init SD card Mount SD card partition with RFS Execute init script MQX Load to specific address in memory with mqxboot utility Application Depends on your specific requirements Boot Process Sequence Power On U-boot Hardware Init U-boot Copies kernel to memory/ uncompress Kernel Boot Hardware/Subsystems initialization RFS Fetch/Mount/Boot into MQX Load to addr & run User Application 100

104 101 Exercise 8: MQX & MCC 101

105 102 Exercise Details: MQX & MCC Components Open DS5-Workspace-clean-preset 1. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/bsp_twrvf65gs10_m4 2. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/psp_twrvf65gs10_m4 3. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/build/ds5/mcc_twrvf65gs10_m4 4. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/examples/pingpong Import the Linux side of the light sensor application Sensor demo or MCCDemo (already in the project) Start pingpong/mcc/mqx component on M4 core run startmqx script on the target location: /home/ Note: Serial Console will behave strange as it is now shared between M4 and A5 102

106 103 Integrate Application Components Added a QThread based class Can handle independently MCC traffic Leverage MCCDemo application Create a custom Signal-Slot pair to trigger camera feed when desired light threshold is reached Leverage start_camera() method and onstopbutton() slot Compile and deploy on Vybrid Place a breakpoint at start_camera method and debug on target 103

107 104 Module 3 Quiz 1. How does GStreamer know which codec to use? 2. How can you construct a pipeline? 3. Name one major difference between Linux and an RTOS 4. What are endpoints? 104

108 105 Building a Video Surveillance Device with LinuxLink Agenda Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development environment with the matching SDK Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using Qt Embedded for Linux Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording under Linux Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot and quick deployment 105

109 106 Module 4 How to measure boot time in our dev environment Fast boot optimizations Boot process overview Bootloader optimizations Kernel level optimizations Filesystem optimizations Other options 106

110 107 Boot time Measurement 107

111 108 Measuring the boot time Each embedded design is different Instrumentation is available Helps measure boot time at all stages Most are open source based Can be easily removed when you are done The trick behind boot time optimization is to know where your system is spending time. You may end up losing a lot of time with little results! 108

112 109 Available Instrumentation Bootloader Logic analyzer (using GPIO pins) Kernel Measurement Printk Compile kernel with: CONFIG_PRINTK_TIMES=y Switch on dynamically: echo Y >/sys/module/printk/parameters/time initcall_debug On boot command line, use: initcall_debug=1 User space measurement Bootchart ( Strace Run strace ct 2>/tmp/strace.log bluetoothd Linux Trace Toolkit System uptime Add (echo -n "uptime:" ; cat /proc/uptime) to an RC script 109

113 110 Fast boot Optimizations 110

114 111 Bootloader Optimizations Low hanging fruit Set the bootdelay variable to 0 (time savings 4s) Preset the bootcmd; do not use setenv (time savings 0.5s) Disable console (time savings 2s) CFG_CONSOLE_INFO_QUIET CONFIG_SILENT_CONSOLE In our case- silent=yes Disable other tests (time savings 2-6s) <1s 75s Boot Time Additional modification/enhancements If possible, use uncompressed Linux kernel Optimize the NAND read operation, to shorten image copy time Rewrite/disable CRC32 checksum code Load the image directly to Entry point Set CONFIG_LOADADDR If NOR Flash is used, leverage XIP For large kernel image, use different compression algorithms 111

115 112 Linux Kernel Optimizations <1s 75s Boot Time Low hanging fruit (time savings: 2-15+s) Use uncompressed kernel Uncompressing takes time Remove unused kernel options Not used networking i.e. IPV6, multiple file systems Debug features and symbols (for final deployment) Build not initially used device drivers as Loadable Kernel Modules Keep the features needed at boot time built into the kernel Remaining drivers built as LKMs will make kernel smaller Consider various approaches for your RFS deployment JFFS2 with appended journal summary (skip flash scan) CRAMFS, UBIFS Suppress the console output Use quiet with your kernel command line 112

116 113 Making the Linux Kernel Small and Fast Linux kernel options we will look at today Kernel Option CONFIG_EMBEDDED CONFIG_IKCONFIG CONFIG_KALLSYMS CONFIG_BUG CONFIG_HOTPLUG CONFIG_DNOTIFY CONFIG_EXT2 CONFIG_PRINTK CONFIG_PRINTK_TIME CONFIG_CC_OPTIMIZE_FOR_SIZE Comment Disables or tweaks a number of kernel options and settings. Think uclinux Saves complete kernel configuration in the kernel Prints our symbolic crash information and backtraces Disables BUG and WARN functions Can be disabled if no external devices will be attached and if you use static device files File change notification to user space Disable if using jffs2 file system Makes kernel silent when disabled A way to track where time is spent at boot time Will select Os instead of O2 resulting in a smaller kernel 113

117 114 Userspace Optimizations Filesystem modifications Optimize the RC Scripts (remove the ones you don t use) If possible replace RC scripts with a single init Pre-link Link applications statically Avoid udev create needed device nodes ahead of time Use static IP address Filesystem modifications Staged booting Code level modifications Add a splash screen <1s 75s Boot Time 114

118 115 Fast Boot Offering from Timesys If your requirements include fast booting, Timesys can help you save time Implement added levels of optimizations with open source techniques Some of the deep-level optimizations techniques, described earlier Use non open source technologies available to Timesys to further speed up the boot-time Typically needed when many services have to be up and running asap This engineering experience is available as a service 115

119 116 Module 4 Quiz 1. Why is fast boot useful? 2. What knowledge is required to get under 10s? 3. Can one get to sub 2s boot time? 116

120 117 How to Reach Us Maciej Halasz for questions related to this workshop Al Feczko for questions related to LinuxLink subscriptions (o) If you have a LinuxLink userid and password, you can work with our technical support engineers by submitting support requests directly 117

121 118 Q&A Thanks for attending! 118