SUSE Advanced Troubleshooting: The Boot Process Lecture

Transcription

1 SUSE Advanced Troubleshooting: The Boot Process Lecture SUS21 Novell Training Services ATT LIVE 2012 LAS VEGAS

2 Proprietary Statement Copyright 2011 Novell, Inc. All rights reserved. Novell, Inc., has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed on the Novell Legal Patents Web page ( and one or more additional patents or pending patent applications in the U.S. and in other countries. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher. Novell, Inc. 404 Wyman Street, Suite 500 Waltham, MA U.S.A. Novell Trademarks For Novell trademarks, see the Novell Trademark and Service Mark list ( html). Third-Party Materials All third-party trademarks are the property of their respective owners. Software Piracy Throughout the world, unauthorized duplication of software is subject to both criminal and civil penalties. If you know of illegal copying of software, contact your local Software Antipiracy Hotline. For the Hotline number for your area, access Novell s World Wide Web page ( and look for the piracy page under Programs. Or, contact Novell s anti-piracy headquarters in the U.S. at 800-PIRATES ( ) or Disclaimer Novell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes. Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page ( for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals. This Novell Training Manual is published solely to instruct students in the use of Novell networking software. Although third-party application software packages are used in Novell training courses, this is for demonstration purposes only and shall not constitute an endorsement of any of these software applications. Further, Novell, Inc. does not represent itself as having any particular expertise in these application software packages and any use by students of the same shall be done at the student s own risk.

3 Contents SECTION 1: The Boot Process 4 SECTION 2: The Boot Process Overview 6 SECTION 3: Troubleshooting Big Picture 8 SECTION 4: Boot Process Details 10 SECTION 5: Troubleshooting Techniques 39 SECTION 6: Boot Process Labs 49

4 Troubleshooting Section 01: The Boot Process The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 4

5 Objectives Boot Process Overview Troubleshooting Big Picture Boot Process Details 5

6 The Boot Process Section 02: The Boot Process Overview The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 6

7 Graphical boot process overview BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm The boot process is linear by nature. Understanding the general process of how a server boots will help to narrow down boot related problems. This graphical overview of the boot process will be used throughout the presentation to indicate which topic will be covered and when. The topic to be covered will be highlighted. The troubleshooting table is based on this orderly boot process. An important troubleshooting technique is to be aware of what messages appear on the screen when each of these stages is reached. This will help you narrow down boot related problems. 7

8 The Boot Process Section 03: Troubleshooting Big Picture The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 8

9 Troubleshooting big picture Boot Installed System BIOS MBR Bootloader Chroot Installed System, init 1 PROMPT_FOR_CONFIRM boot boot.local Kernel Initial Ramdisk init init rc mingetty xdm You can apply the troubleshooting process to the sequential nature of the boot process itself. Boot Installed System (BIS) is a method where you use the boot loader, kernel and ram disk from the installation media to boot your installed system. BIS bypasses everything in the Boot Installed System bubble above. Consequently if BIS works, your boot problem is contained within that bubble. BIS lets you basically narrow your boot problem in half. If BIS still fails, your problem is most likely in the bubble on the right. There are three primary troubleshooting techniques used in this bubble to help narrow down problems, they are Chroot Installed System (CIS), administrative run levels like run level 1 (init 1), and using flow control from /etc/sysconfig/boot to slow down the loading of system daemons (PROMPT_FOR_CONFIRM). 9

10 The Boot Process Section 04: Boot Process Details The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 10

11 BIOS & MBR BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering the BIOS and the transition from BIOS to the first stage boot loader. 11

12 BIOS The BIOS determines the order and location of hard drives The BIOS loads the first 512 bytes on the hard disk with the the boot flag set into memory Control is transferred to the MBR What's on screen? POST Messages The BIOS boots from the master boot record (MBR). It basis the boot device based on boot order and the boot flag set on the disk. The BIOS banner and POST messages are clear on-screen indicators for your location in the boot process. 12

13 MBR (Master Boot Record) The first 512 bytes on the hard disk Divided into three parts Code Area: First 446 bytes, mostly used by boot loaders Partition Table: 64 bytes, four 16 byte entries Signature: 2 bytes, also known as the Magic Number. The BIOS executes the Code Area if the Magic Number 0x55AA is found. What's on screen? Usually just a black screen, but you might see No Operating System Found An MBR as described in this course assumes an msdos disk type, as opposed to gpt, sun or others. 13

14 Troubleshooting BIOS/MBR Update BIOS and hardware component firmware Mark a disk partition bootable If there are no partitions marked bootable, you will need to set the bootable flag on the partition containing /boot. This is usually the root (/) partition or the /boot partition if it is separate. To see if a partition is marked bootable, run fdisk -l, which would show a partition with the boot flag like this: Device Boot Start End Blocks Id System /dev/sda1 * Linux If sda1 was not marked bootable, set the boot flag by running: parted -s /dev/sda toggle 1 boot If you don't have a boot partition at all, you may have a corrupted MBR. You should restore it from backup, or restore it from the supportconfig's fs-diskio.txt file's sfdisk entry. This process will be documented in the lab manual. 14

15 Bootloader: GRUB BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering the GRUB second stage boot loader. 15

16 GRUB Loads the kernel and initrd into memory and boots Consists of three stages: stage1, stage1.5 and stage2 /boot/grub/stage1 is written to the MBR It's only purpose is to load stage2 stage2 location is hard coded into stage1 /boot/grub/*stage1_5 provide file system drivers /boot/grub/stage2 provides the main boot loader functions Reads /boot/grub/menu.lst Loads kernel and initrd via direct filesystem access Stage 1 is copied into the code section of the MBR. It's purpose is to launch the appropriate stage 1.5 for the filesystem, and then load stage 2. Stage 2 is the main boot loader code that reads the menu.lst configuration file and loads the kernel and ramdisk into memory. 16

17 GRUB Reads disk devices from the BIOS or other disk controllers Naming convention (hd0,0) hd0 is the first disk detected,,0 is the first partition on that disk (hd1,2) The second disk detected on the third partition GRUB does not distinguish among IDE, SCSI or SATA devices The devices are saved in: /boot/grub/device.map What's on screen? GRUB menu or grub> prompt You know you have reached the second stage boot loader if you see the GRUB menu or the grub> prompt. 17

18 Troubleshooting GRUB A grub RPM validation will not check /boot/grub/stage*. The /usr/lib/grub/* files are owned by the RPM and copied to /boot/grub with an RPM post install script Reinstall the grub RPM package yast -i grub Reinstall GRUB grub-install Recreate or check the /boot/grub/menu.lst YaST, System, Boot Loader, Other, Propose New grub> tab completion If you are having boot problems, reinstalling grub and/or the grub package can help resolve the issue. GRUB's tab completion is a quick trick to help find a file. The GRUB find command is used to identify the devices on which that file exists. It is not used to search for an unknown file. 18

19 vmlinuz & initrd BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering the kernel and the initial ramdisk. 19

20 Kernel The GRUB kernel command loads the kernel file directly from the /boot directory The kernel file is /boot/vmlinuz-3.* and is owned by the kernel base RPM package Additional /lib/modules kernel drivers are owned by the kernel RPM package /boot/grub/menu.lst provides the default location for the kernel and menu screen for booting The kernel uses root= to help identify the root partition What's on screen? kernel The kernel is the heart of the SUSE Linux operating system. The kernel /boot/vmlinuz- 3.* file is provided by the kernel RPM package and is loaded into memory by GRUB. GRUB will display kernel just prior to booting the kernel. Once the kernel is booting, you will see the familiar kernel time stamped messages with the format [ ]. 20

21 Troubleshooting the kernel Validate the kernel and kernel base packages Boot Installed System Run rpm -Vf /boot/vmlinuz-* to verify the kernel package If the kernel does not pass validation, reinstall the kernel or kernel base rpm Make sure root= in /boot/grub/menu.lst is pointing to the correct root device If you are at a grub> prompt try loading the kernel without any options Make sure the root device matches in /boot/grub/menu.lst root= parameter, the /etc/fstab, and the ramdisk. Once you have checked the root device in menu.lst and fstab, you can run mkinitrd to make sure that device is put correctly into the ramdisk. It is good to remember that kernel drivers are part of the kernel RPM package. Verifying the kernel RPM package will not only validate the kernel itself, but all the drivers associated with it. 21

22 Initial ramdisk The initial ramdisk default file is /boot/initrd-3.* It is owned by the kernel base package The initrd file is not included in an rpm validation Provides an initial root filesystem for the kernel Contains all kernel drivers, binaries or configuration files needed to find and mount the root filesystem The mkinitrd command creates the initial ramdisk What's on screen? initrd Once GRUB loads the kernel into memory, it then loads the ram disk into memory. The kernel needs a root filesystem to operate. When the kernel is first loaded, it is not aware of the root filesystem on the hard disk, so a temporary root filesystem is provided using the initial ram disk. The ram disk is a compressed cpio archive and contains all the drivers and binaries necessary to find and mount the root partition. Once the root partition is mounted, all additional drivers and filesystems can be loaded and mounted from the root filesystem on the hard disk. 22

23 Troubleshooting initial ramdisk Boot Installed System Edit /etc/sysconfig/kernel Make sure the INITRD_MODULES variables has the correct drivers included to find the root partition From the command prompt, run 'mkinintrd' Check /boot and make sure the symbolic links below are pointing to the correct kernel and initrd /boot/vmlinuz /boot/initrd Reboot to test The best troubleshooting option for a problematic ram disk is to rebuild it. Make sure /etc/sysconfig/kernel INITRD_MODULES has a correct module list, and then run mkinitrd to rebuild the ram disk. You can extract the contents of the ramdisk as follows: 1. mkdir -p /boot/tmp 2. cd /boot/tmp 3. zcat /boot/initrd-3.* cpio -id 4. ls -l /boot/tmp 23

24 init (In the RAMDISK) BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm There are two init commands used in the boot process. The first is a script created by the nkinitrd command that gets embedded into the root directory of the ramdisk (/boot/initrd). The other is a dynamically linked executable installed on the system in /sbin/init. Now considering the init in the ramdisk. 24

25 init (RAMDISK) Responsible for finding the root filesystem Mounts the root filesystem Establishes network connections if needed for root Loads modules associated the root device What's on screen? Loading <module_name> Mounting root partition The init script is located in the root directory of the ram disk. It loads all necessary modules for mounting the root filesystem. The init script also runs all other binaries required to find and mount the root filesystem. 25

26 Troubleshooting init (RAMDISK) Boot Installed System Edit /etc/sysconfig/kernel Make sure the INITRD_MODULES variables has the correct drivers included to find the root partition From the command prompt, run 'mkinintrd' Check /boot and make sure the symbolic links below are pointing to the correct kernel and initrd /boot/vmlinuz /boot/initrd Reboot to test The mkinitrd command creates the ram disk init script. The best troubleshooting technique to try is to check the /etc/sysconfig/kernel INITRD_MODULES variable and run mkinitrd. The init script will be recreated. 26

27 /sbin/init process BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering /sbin/init 27

28 /sbin/init functionality /sbin/init is the master process It is always PID 1 The kernel is programmed to launch /sbin/init by default All other processes are forked from init It reads and processes /etc/inittab What's on screen? INIT: version 2.86 booting Once the root filesystem is mounted, the kernel run /sbin/init. It is the master process and runs with a process ID of 1. It is responsible for starting system daemons. It's configuration file is /etc/inittab. 28

29 Troubleshooting /sbin/init Validate sysvinit, which owns /sbin/init Validate aaa_base, which owns /etc/inittab Review the man pages for inittab(5) Modify /etc/sysconfig/boot to: Enable PROMPT_FOR_CONFIRM Try RUN_PARALLEL= no Enable FLOW_CONTROL Bypass /sbin/init with Chroot Installed System (CIS), init=/bin/bash or init=/bin/sash Boot to single user mode to minimize running daemons Boot Options: ' 1' or run 'init 1' at the prompt Make sure the associated RPM packages pass validation. Confirm the /etc/inittab has been configured properly. The init= kernel parameter can be used to bypass /sbin/init. 29

30 init: boot & boot.local BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering /etc/init.d/boot 30

31 init: boot & boot.local Runs all of the /etc/init.d/boot.d scripts What's on screen when boot starts? System Boot Control: Running /etc/init.d/boot System Boot Control: The system has been setup What's on screen when boot.local starts? Done System Boot Control: Running /etc/init.d/boot.local init runs /etc/init.d/boot to run all boot level init scripts The boot.d scripts setup the basic framework of the server, like mounting all additional filesystems. They are run first, before run level daemons. Generally they are run once, and do not keep running as system daemons. Once the boot level scripts are complete, /etc/init.d/boot runs the /etc/init.d/boot.local script. You can put any commands in this file you want to run prior to starting the run level scripts. 31

32 Troubleshooting init: boot & boot.local If a boot service fails, modify /etc/sysconfig/boot to help determine which one PROMPT_FOR_CONFIRM FLOW_CONTROL If the server will not boot, use Chroot Installed System (CIS) Files needed and it's (RPM) /bin/bash (bash) /etc/init.d/boot (aaa_base) /etc/init.d/boot.d/* (various packages) The boot messages scroll pretty fast. You can slow down the messages using flow control or prompt for confirm. Flow control lets you press Ctrl-S to stop the screen long enough to read the error messages, and Ctrl-Q to resume the screen. Prompt for confirm tells the operating system to prompt you to load each individual service during the boot process. The settings are found in the /etc/sysconfig/boot file. 32

33 init: Resource Control (rc) BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering /etc/init.d/rc 33

34 Resource Control (rc) Starts all of the daemons in a specified run level init uses the /etc/inittab to determine the default runlevel # The default runlevel is defined here id:3:initdefault: What's on screen when rc starts? Done Master Resource Control: Master Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: init uses /etc/init.d/rc to run all runlevel scripts The resource control script controls the scripts in a specified run level. The run level is specified in the /etc/inittab file or on the kernel command line. 34

35 Troubleshooting Resource Control If a system daemon fails, modify /etc/sysconfig/boot to help determine which one PROMPT_FOR_CONFIRM FLOW_CONTROL If the server will not boot, use Run Level S or 1 Chroot Installed System (CIS) Files needed and it's (RPM) /bin/bash (bash) /etc/init.d/rc (aaa_base) /etc/init.d/rc?.d/* (various packages) You can slow down the messages using flow control or prompt for confirm. Flow control lets you press Ctrl-S to stop the screen long enough to read the error messages, and Ctrl-Q to resume the screen. Prompt for confirm tells the operating system to prompt you to load each individual service during the boot process. The settings are found in the /etc/sysconfig/boot file. 35

36 init: User Login BIOS /boot/grub/stage1mbr boot /etc/init.d/boot.d/s* /etc/init.d/boot.local /boot/grub/stage2 Initial Ramdisk Kernel init rc mingetty /etc/init.d/rc[prev_run_lvl].d/k* /etc/init.d/rc[cur_run_lvl].d/s* init xdm Considering the user login 36

37 User Login mingetty is used for tty1 tty6 xdm is used for tty7, the graphical login screen init uses /etc/inittab to determine how the login screens will behave By default init respawns the mingetty and xdm processes everytime they are terminated This allows you to login again after logging out What's on screen? Login prompt The /sbin/init process reads /etc/inittab to determine which application to run on which terminal. It runs mingetty for virtual consoles 1 through 6 and an X display manager on terminal 7. The /etc/sysconfig/displaymanager is used for the display manager configuration. 37

38 Troubleshooting User Login Check /etc/inittab to make sure /sbin/init is managing logins correctly Files affecting logins with it's (RPM) /etc/nologin (n/a) /sbin/mingetty (mingetty) /etc/pam.d/login (login) /etc/sysconfig/displaymanager (n/a) /etc/sysconfig/windowmanager (n/a) If you usually login from the X console, try logging into the virtual consoles or vise versa Try logging into a newly created test user If you have a graphical display problem, you can change to run level 3 (init 3), run sax2 -a, and then change back to run level 5 (init 5). This allows sax2 to automatically probe the X configuration. 38

39 The Boot Process Section 05: Troubleshooting Techniques The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 39

40 Objectives Troubleshooting Procedure Boot Installed System (BIS) Administrative Run Levels Chroot Installed System (CIS) Flow Control Troubleshooting Table 40

41 Troubleshooting Procedure Let the server boot until it fails Write down verbatim what's on the screen Match on-screen landmarks to the Troubleshooting Table Use Boot Installed System (BIS) to bypass GRUB, kernel and ram disk issues Use an administrative run level for daemon failure Use Chroot Installed System (CIS) if all else fails Address the issues and files associated with the location of the boot failure (see Troubleshooting Table) Notes: 41

42 Boot Installed System (BIS) Provided on the installation media Bypasses issues relating to: BIOS, boot loader, kernel and ram disk Allows the server to be up while troubleshooting Boot Installed System (BIS) Procedure 1.Used mostly in lines 1-7 of the Troubleshooting Table. 2.Boot from DVD1 3.Select Installation 4.Accept the License Agreement 5.Click Next, and Next to skip media checks 6.Select Repair Installed System and Next 7.Select Expert Tools 8.Select Boot Installed System NOTE: Select Repair Installed System directly from the DVD boot menu does not probe as thoroughly as the Repair Installed System from the Installation option. 42

43 Administrative Run Levels Run level S and 1 use the installed system Minimal system daemons are started Useful when attempting to isolate a misbehaving system daemon Run level S and 1 are very similar to chroot installed system (CIS), as far as run levels go. However, run levels S and 1 use the installed system's boot loader, kernel and ram disk to boot. It just doesn't start all the system processes like run level 3 or 5 do. So, run level S and 1 are preferred over CIS. There are a couple of ways to change to run level S or 1. You could just type init 1. However, if you are troubleshooting system processes that fail at boot time or cause the server to misbehave as a result; you will want to reboot the server, bypass the default run level and boot into the run level 1. To boot to run level 1, do the following: 1) Boot the server normally 2) Select the kernel you usually boot to 3) Tab or click in the Boot Options field 4) Append 1 (a space followed by the number 1) to the boot options line 5) Type root's password If you need network access, run /etc/init.d/network start 43

44 Chroot Installed System (CIS) CIS uses rescue mode to access the installed system Boot Installed System (BIS) and Run Level 1 are preferred When is CIS used? BIS and run level 1 fail When the root user account does not work properly, use CIS Notes: 44

45 Chroot Installed System (CIS) Procedure 1)Used mostly in lines 8 and above of the troubleshooting table. 2)Boot from DVD1 3)Select Rescue System, Rescue login: root 4)Your first goal is to find and mount the root / partition, so we can see /etc/fstab 1) Run cat /proc/partitions to find the disk devices the OS sees 2) For each device, display the partition table ls boot:~ # parted s /dev/sda print Disk geometry for /dev/sda: 0kB 2147MB Disk label type: msdos Number Start End Size Type File system Flags 1 32kB 214MB 214MB primary ext2 boot, type= MB 535MB 321MB primary linux swap type= MB 2147MB 1612MB extended lba, type=0f 5 535MB 1012MB 477MB logical reiserfs type= MB 1596MB 584MB logical reiserfs type= MB 2147MB 551MB logical reiserfs type=83 3) You can ignore type 82 swap and type 0f extended partitions 4) To find the root partition, you may need to just guess. For example, 1) mount /dev/sda1 /mnt 2) ls -l /mnt 3) If the /mnt directory listing shows /etc and /root, then its the root partition 4) Repeat these steps for each device until you find root. In this case, the root device is /dev/sda6 5) mount /dev/sda6 /mnt 5)Mount all additional file systems relative to /mnt 1) Once you have mounted the root filesystem, run cat /mnt/etc/fstab to see all the other filesystem mount points. 2) Mount all file systems manually as shown in /mnt/etc/fstab. mount /dev/sda1 /mnt/boot mount /dev/sda5 /mnt/var mount /dev/sda7 /mnt/usr 3) Rebind the /proc, /sys and /dev filesystems. mount rbind /proc /mnt/proc mount rbind /sys /mnt/sys mount rbind /dev /mnt/dev 6)Chroot to the installed system: chroot /mnt 7)To return to the rescue system, type exit. 45

46 Flow Control File: /etc/sysconfig/boot FLOW_CONTROL Use Ctrl-S to stop and Ctrl-Q to resume PROMPT_FOR_CONFIRM Prompts to load each service at boot time SHIFT-PGUP Scrolls up a couple of pages on the console boot screen The normal boot messages display on the screen very fast. There are ways to slow it down and test each service as it loads. The boot messages are controlled by variables set in the /etc/sysconfig/boot file. FLOW_CONTROL= yes Allows you to stop the boot process messages using Ctrl-S and resume them with Ctrl-Q. PROMPT_FOR_CONFIRM= yes CONFIRM_PROMPT_TIMEOUT= 5 This will display the prompt: Enter interactive startup mode? y/[n](5s) You will need to select y to enter interactive startup mode within the CONFORM_PROMPT_TIMEOUT period, otherwise the server will boot normally without prompting to load system daemons. After you enter interactive startup mode, you will be prompted to load each service with the following: Start service <service_name>, (Y)es/(N)o/(C)ontinue? [y] The CONFIRM_PROMPT_TIMEOUT value also applies to each service start prompt. This was not true with earlier versions of SLES. Once the server has booted up, you can use Shift-PgUp to scroll up about two screens worth of boot messages, regardless of the /etc/sysconfig/boot settings. However, if you switch to other consoles (ie tty2), you will not be able to use this keystroke. 46

47 Troubleshooting Table Troubleshooting Table's Purpose Narrow down boot related issues faster Identifies location in the boot process Associated files at each step On-screen landmarks to help narrow down where you are in the boot process Troubleshooting suggestions at each step Troubleshooting Table # Location Files On-Screen Landmarks Troubleshooting/ Potential Fixes 1 BIOS N/A BIOS messages Update firmware Make sure a disk device is marked bootable 2 MBR /boot/grub/stage1 GRUB loading stage2... BIS grub-install /dev/<disk> 3 GRUB /boot/grub/stage2 /boot/grub/menu.lst 4 GRUB /boot/vmlinuz /boot/initrd GRUB menu or grub> prompt root (hd?,?) kernel /<path_to_vmlinuz> initrd /<path_to_initrd> BIS grub-install /dev/<disk> Check /boot/grub/menu.lst Reinstall kernel RPM mkinitrd 5 kernel /boot/vmlinuz Kernel driver info scrolling RAMDISK driver initialized: BIS Reinstall the kernel RPM 6 initrd /boot/initrd /etc/sysconfig/kernel 7 initrd:init /init in /boot/initrd /etc/sysconfig/kernel RAMDISK: <relevant message> Starting udevd Creating devices Load <module_name> There will be loading for each module defined in /etc/sysconfig/kernel INITRD_MODULES and their dependent modules. BIS Extract and view the /boot/initrd BIS Mkinitrd creates the initrd:ini file 47

48 Troubleshooting Table (cont) SUSE Advanced Troubleshooting: The Boot Process # Location Files On-Screen Landmarks Troubleshooting/ Potential Fixes 8 sbin:init /sbin/init /etc/inittab 9 sbin:init:boot /bin/bash /etc/init.d/boot /etc/init.d/boot.d/* /etc/sysconfig/boot INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Each service shows: done, failed or skipped System Boot Control: The system has been setup init 1, then CIS Use boot options init=/bin/bash or init=/bin/sash to bypass running /sbin/init init 1 starts the minimum services CIS starts no services PROMPT_FOR_CONFIRM= y es RUN_PARALLEL= no FLOW_CONTROL= yes (Ctrl-S stops, Ctrl-Q resumes) 10 sbin:init:boot /etc/init.d/boot.local System Boot Control: Running /etc/init.d/boot.local init 1 or CIS 11 sbin:init /sbin/init /etc/inittab 12 sbin:init:rc /bin/bash /etc/init.d/rc /etc/init.d/rc?.d/* /etc/sysconfig/boot 13 sbin:init /sbin/init /etc/inittab 14 sbin:init:mingetty /etc/issue /etc/motd /etc/nologin /sbin/mingetty /etc/pam.d/login 15 sbin:init:x /etc/sysconfig/ displaymanger /etc/sysconfig/ windowmanager INIT: Entering runlevel: 3 Master Resource Control: previous runlevel:n, switching to runlevel: 3 Master Resource Control: Running /etc/init.d/before.local Each service shows: done, failed or skipped Master Resource Control: Running /etc/init.d/after.local Master Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: N/A <contents of /etc/issue> login: Graphical login screen init 1 or CIS Init 1 or CIS PROMPT_FOR_CONFIRM= y es RUN_PARALLEL= no FLOW_CONTROL= yes init 1 or CIS init refers to it's inittab file to know how to run the login programs init 1 bypasses mingetty CIS init 1 bypasses X login CIS 48

49 The Boot Process Section 06: Labs The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab. 49

50 LABS 1-10: Boot Process Labs Summary: In these exercises, you will be simulating several boot failures and troubleshooting them using the troubleshooting techniques described above. Special Instructions Complete labs 1 through 10. If you have time, continue working on the additional labs. Duration: The remainder of class machine You are not expected to complete all the labs. Labs may be broken in several ways, making the issue more complex. The purpose of many labs with varying degrees of difficulty is to meet the needs of a wide range of system administrators. 50

51 51

52 UnpublishedWorkofSUSE.AlRightsReserved.Thisworkisanunpublishedworkandcontainsconfidential,proprietary,andtradesecretinformationofSUSE.AcestothisworkisrestrictedtoSUSEemployeswhohaveanedtoknowtoperformtaskswithinthescopeoftheirasignments.Nopartofthisworkmaybepracticed,performed,copied,distributed,revised,modified,translated,abridged,condensed,expanded,colected,oradaptedwithouthepriorwritenconsentofSUSE.Anyuseorexploitationofthisworkwithoutauthorizationcouldsubjectheperpetratortocriminal andciviliability.generaldisclaimerthisdocumentisnotobeconstruedasapromisebyanyparticipatingcompanytodevelop,deliver,ormarketaproduct.itisnotacomitmentodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.susemakesnorepresentationsorwarantieswithrespectothecontentsofthisdocument,andspecificalydisclaimsanyexpresorimpliedwarantiesofmerchantabilityorfitnesforanyparticularpurpose.thedevelopment,release,andtimingofeaturesorfunctionalitydescribedforsuseproductsremainsathesolediscretionofsuse.further,susereservestherightorevisethisdocumentandtomakechangestoitscontent,atanytime,withoutobligationtonotifyanypersonorentityofsuchrevisionsorchanges.al SUSEmarksreferencedinthispresentationaretrademarksoregisteredtrademarksofNovel,Inc.intheUnitedStatesandothercountries.Al third-partytrademarksarethepropertyoftheirespectiveowners. 52