Projects on the Intel Single-chip Cloud Computer (SCC)

Size: px

Start display at page:

Download "Projects on the Intel Single-chip Cloud Computer (SCC)"

Oscar Allen
5 years ago
Views:

Andreas Polze Operating Systems and Middleware Group Hasso Plattner Institute for Software Systems Engineering

1 Projects on the Intel Single-chip Cloud Computer (SCC) Jan-Arne Sobania Dr. Peter Tröger Prof. Dr. Andreas Polze Operating Systems and Middleware Group Hasso Plattner Institute for Software Systems Engineering University of Potsdam Outline 2 Intel SCC Overview Our Contributions Debugging Support: Virtual Serial Ports Understanding the SCC: New SCC Linux More Debugging: Dynamic Voltage and Frequency Scaling SCC Virtualization: The RockyVisor Summary 1

Intel Single-Chip Cloud Computer 3 48 x86 cores organized in

bandwidth 4 integrated DDR3 memory controllers Hardware

hardware cache-coherency Research Processor, not a product

2 Intel Single-Chip Cloud Computer 3 48 x86 cores organized in 24 tiles A 24-router mesh network with 256GB bisection bandwidth 4 integrated DDR3 memory controllers Hardware support for message passing Power management: 24 FIs, 6 VIs No hardware cache-coherency Research Processor, not a product Third take Intel DX58SO 4 Intel SCC Management Console PC (MCPC) Intel Core i7 2

3 5 Intel SCC RockyLake inside here Management Console PC (MCPC) Intel Core i7 South Bridge via FPGA On-Board I/O 6 Host/MCPC Connection PCIe x4, BMC-LAN BMC SCC (under the heat spreader) 3

P54C (original Pentium@~90MHz) 32-bit Physical Addresses All components accessed via 46-bit System Address Route to tile (x,y

4 SCC: Cluster-on-a-chip 7 Heterogeneous cores Memory and I/O bandwidth/latency depend on position on the die 32-bit cores: not all system memory visible all the time Again: no hardware cache coherency SCC Address Translation 8 GaussLake -Cores derive from P54C (original Pentium@~90MHz) 32-bit Physical Addresses All components accessed via 46-bit System Address Route to tile (x,y coordinates) Destination ID (directly connected to router) 34-bit address per destination Mapping via per-core Look-Up Table (LUT) 4

5 SCC Physical -> System Address Map. 9 Traditional P54C: Physical Address would be seen on FSB On SCC: Physical Address mapped to System Address via LUT Look-Up Table Memory access via mesh msg. 8-bit LUT index translates to: 1-bit bypass 8-bit route 3-bit destination 10-bit address extension Up to 34 bit (16GB) per destination Outline 10 Intel SCC Overview Our Contributions Debugging Support: Virtual Serial Ports Understanding the SCC: New SCC Linux More Debugging: Dynamic Voltage and Frequency Scaling SCC Virtualization: The RockyVisor Summary 5

6 SCC Debugging: Virtual Serial Ports 11 How to debug operating systems on SCC cores? Virtual Ethernet requires full network driver stack (oops ) Regular x86 Linux: 16550A UART ( serial port ) SCC does not have a chipset, no UART implemented Our contribution: Virtual Serial Ports 4x 16550A-compatible UARTs per SCC core (192 total) Fully transparent for SCC Linux; use stock driver But without interrupts Implemented in kernel driver on MCPC Connected to device on MCPC via virtual Null-Modem Cable minicom D /dev/crbif0rb0c9ttys0 (Core 9 s /dev/ttys0) Understanding the SCC: New SCC Linux 12 Intel-provided SCC Linux: Based on (released 2006) Outdated, no virtualization Various patches for SCC scattered throughout the kernel Hard to understand Hard to port to newer versions Our contribution: New patches based on Linux subarchitectures SCC BIOS: minimal BIOS emulation, no more patches Re-engineered other patches into subarchitecture callbacks New drivers for hardware features (global TSC, SCC /proc, ) Merged with new Intel patches in September (on-site) To be published with next SCC firmware in December Maintained by our group at HPI 6

Outline 13 Intel SCC Overview Our Contributions

Understanding the SCC: New SCC Linux More

SCC Virtualization: The RockyVisor Summary More

Controller (VRC) and Clock Generators 6 Voltage

(100-800MHz) Reconfigurable at runtime (aside

7 Outline 13 Intel SCC Overview Our Contributions Debugging Support: Virtual Serial Ports Understanding the SCC: New SCC Linux More Debugging: Dynamic Voltage and Frequency Scaling SCC Virtualization: The RockyVisor Summary More Debugging: Dynamic Voltage and Frequency Scaling (DVFS) 14 SCC has programmable Voltage Regulator Controller (VRC) and Clock Generators 6 Voltage Islands: 2x2 tile 24 Frequency Islands: each tile ( MHz) Reconfigurable at runtime (aside from mesh and routers) 7

8 Fixing undocumented JTAG Sequences 15 Implementation Bug: Tile frequencies cannot be changed Depending on startup configuration (initial tile, mesh and memory clock), reprogramming tile clock generators fails silently Register settings change, but frequency does not Time measurement on cores unreliable Accidental under-volting possible Reason: Fixed Frequency Bits/ Override Settings set on startup Undocumented bits in per-tile register control which settings can be changed by software Our contribution: Fixed undocumented JTAG sequences Approved by Intel, next SCC firmware will include our patches Outline 17 Intel SCC Overview Our Contributions Debugging Support: Virtual Serial Ports Understanding the SCC: New SCC Linux More Debugging: Dynamic Voltage and Frequency Scaling SCC Virtualization: The RockyVisor Summary 8

9 Software-Managed Coherency 18 Shared Memory Driver: Lessons learned 19 Shared Memory Driver simulates coherency for applications Still requires changes to management layer (malloc, mmap, ) Not sufficient for SMP simulation for legacy software Simulating an SMP (really: Single System Image/SSI) system User mode/custom libc: simulate system calls (many ) Kernel mode (upper level): simulate SMP-style API (many ) Kernel mode (lower level): synchronize data structures Breaks OS layering (+ brittle) Hypervisor mode: simulate SMP platform Unmodified Applications and OS Hypervisor implements SMP interfaces the real HW lacks That s what the RockyVisor does on SCC 9

10 SCC Virtualization: The RockyVisor 20 A Hypervisor for the Intel SCC Platform ( RockyLake ) Hosted Virtual Machine, Hypervisor runs as process in host OS Virtual Machine Monitor and Hypervisor used as synonyms Multiple independent nodes, each having local memory Host OS: One instance per node ( Level 1 OS ) Default SCC operating mode SCC Virtualization: The RockyVisor 21 Separate hypervisor process per LV1 instance Hypervisors communicate via on-die message passing Responsibilities Virtual devices (console, network, ) belong to single HV Virtual (cache-coherent) Shared Memory VM-global state in shared memory (virtual interrupt controller) Virtual CPU per HV (registers, TLB, shadow page tables) 10

11 SCC Virtualization: The RockyVisor 22 Cooperating Hypervisors provide illusion of SMP hardware Cache coherency simulated via DSM-like protocol SCC already supports shared memory, so no replication Explicit cache flushes if memory needed by other node Use x86 Page Protection to prevent harmful interference SCC Virtualization: The RockyVisor 23 Cooperating Hypervisors provide illusion of SMP hardware Each hypervisor adds one Virtual CPU per physical CPU VCPUs communicate just like in real SMP Inter-Processor-Interrupts (IPI) via Inter-HV messages 11

12 SCC Virtualization: The RockyVisor 24 Cooperating Hypervisors provide illusion of SMP hardware Allow standard SMP OS ( Level 2 OS ) in VM LV2 kernel needs to support virtual peripherals Summary 25 Intel SCC A prototype for future many-core architectures Our contributions Debugging Support for cores: Virtual Serial Ports New Linux patches for easy porting to new kernel versions Fixed JTAG sequences to enable Frequency Scaling Will both become official part of SCC software distribution Distributed Hypervisor to support symmetric multiprocessing operating systems (SMP OS): Our RockyVisor 12

13 26 13

Tile Processor (TILEPro64)

Tile Processor Case Study of Contemporary Multicore Fall 2010 Agarwal 6.173 1 Tile Processor (TILEPro64) Performance # of cores On-chip cache (MB) Cache coherency Operations (16/32-bit BOPS) On chip bandwidth