Developing and Sustaining your Enterprise GIS Design

Transcription

1 Developing and Sustaining your Enterprise GIS Design Dave Peters ESRI Systems Integration 1of 51

2 Overview System Architecture Design Process Enterprise GIS Evolution GIS Software and Migration Roadmap Managing Technology Change Network Communications Data Administration Hardware Performance Update Sustaining Enterprise Operations 2of 51

3 System Architecture Design What Is System Architecture Design? User Needs Hardware Specifications Data Resources Network Design People Hardware Applications 3of 51

4 System Architecture Design Why Is System Design Important? Support System Performance Requirements Reduce Overall System Cost Balanced System Hardware and Network Design A Solid Foundation for High-Performance Solutions 4of 51

5 System Design Process GIS Needs Assessment Users Applications Data Implementation Strategy GIS Professional Consultant System Architecture Design Technology Overview Existing Environment GIS User Needs Overview System Configuration Alternatives System Design Components System Loads Analysis Hardware Sizing/Selection Implementation Strategy GIS System Design Architect 5of 51

6 Enterprise GIS Evolution Department File Servers GIS User LAN GIS User GIS User GIS User 6of 51

7 Organizational GIS Department File Servers Utilities Parks Centralized Data Warehouse IT Utilities ArcSDE Assessor WAN Arc Desktops Assessor WAN Parks Distributed Client/Server Departmental GIS Assessor WAN Arc Desktops Centralized Database IT Data Warehouse Utilities Departmental GIS Operations ArcSDE ArcIMS/WTS Centralized Data Sharing (server consolidation) Parks ArcTerminals/Browsers Centralized Database Enterprise GIS Operations Centralized Data Administration 7of 51

8 Community GIS Arc Desktops Assessor Centralized Database IT Utilities ArcSDE ArcIMS/WTS WAN WWW Parks Company GIS GIS Data Marts ArcSDE ArcIMS Geography Network County GIS ArcTerminals/Browsers Enterprise GIS Operations Department and Central Database Data Integration Application Service Providers ArcSDE ArcIMS/ Terminal Servers Arc Desktops Regional Metadata WWW Arc Terminals/Browsers City GIS Internet GIS Operations Shared Database Environments Data Integration 8of 51

9 Files -Coverages -Shapes -Images System Architecture Options Data Resources ArcSDE -Georelational -GeoDatabase Attributes -Customers -Facilities -Documentation G.NET Architecture File/Image Server Storage Area Network WTS ArcIMS Regional Metadata Web Geography Network ArcSDE Server DBMS Server 100 Mbps Ethernet WAN WWW Remote Sites Arc Desktops -Analysis -Maintenance -Operations -Studies Arc Browsers -Map Products -Analysis -Operations -Studies ArcGIS Architecture Arc Terminals -Analysis -Maintenance -Operations -Studies Arc 8 Laptops -Data Updates -Analysis Arc 8 Mobile Users -Data Updates -Analysis Web Browsers -Data Download -Maps on Demand -Data Shipping 9of 51

10 Hardware Life-Cycles Definitions Current Useful Obsolete Non-functional Length of time between major hardware technology releases Current release software is still supported on this equipment New releases of software are not supported with this equipment After this point the infrastructure is no longer functional. Cost of maintenance is greater than residual value of equipment 10 of 51

11 2002 Hardware Life-Cycles (Months) Useful Server Technology Technology Current Useful Obsolete Non-functional Network Infrastructure Local area networks Wide area networks* Computer Hardware Data Servers Application Servers Desktop Workstations Laptop Workstations Terminal clients * Internet bandwidth increasing at 300% per year 11 of 51

12 Sustaining Enterprise Operations Managing Technology Change Strategic Plan Strategic Plan 2002 Deployment Authorization 2003 Deployment Authorization System System Architecture Architecture Design Design Strategic Strategic Plan Plan System System Architecture Architecture Design Design Strategic Strategic Plan Plan Maintenance Maintenance 12 of 51

13 Systems Integration Support Efforts System Design Strategies White Paper ( System Design Strategies Workbook System Design Support Hardware Sales Support (Regional Office/Systems Integration Staff) Platform Configuration Tools Technical White Papers System Integration Test Reports Installation Guidelines System Architecture Design Training (ESRI Training Class) (Technology Exchange Workshops) System Architecture Design Consulting (Professional Consulting for Enterprise GIS) Enterprise Systems Lab (Internet Demonstration Site ( Performance Validation Testing ArcIMS Architecture Design Consulting (Professional Consulting for ArcIMS Deployment) 13 of 51

14 GIS Software Evolution Overview Single-User Professional Several GIS Users Projects/Map Products Desktop Viewers Query and Analysis Operations Remote Users Enterprise GIS Operations Data Development Data Conversion GIS Projects Map Production GIS Data Maintenance Several GIS Editors Data Administration Project Research Ad Hoc Mapping General Operations Specific Operations Work Management Delivery Routing Emergency Response Embedded Applications Internet Map Products Data Publishing Data Integration Desktop Interface Enterprise GIS Personal GIS Query and Analysis ArcInfo ArcView GIS MapObjects ArcInfo ODE X-Emulation Windows Terminal Clients ArcView IMS MapObjects IMS ArcIMS ArcGIS Desktop - ArcInfo - ArcEditor - ArcView ArcInfo Coverages ArcInfo LIBRARIAN GIS File Servers Local Area Networks Shapefiles ArcStorm Spatial Database Engine Application Servers Terminal Servers Wide Area Networks Map Servers Web Servers Intelligent Data ArcSDE 14 of 51

15 ArcGIS Migration Roadmap Legacy Systems Coverages Librarian Shapefiles Image Files ArcStorm File Server AML ArcInfo Layers ==> Feature Class (Tables) Attributes ==> Object Class (Tables) ArcGIS Architecture Custom ArcGIS ArcSDE Admin Commands ArcCatalog/ArcToolbox) Business Process Data Model SDE Layers (Simple Layers) Register with GeoDatabase SDE Layers (simple layers) Object Relational (Data Model) Edit Versions (Reconcile/post) Default Version Integrated Architecture Non-versioned Unregistered with GeoDatabase GeoDatabase Non-Versioned/ Registered Relationships/ Domains Versioned Multi-user Editing DBMS Tables Columns (attributes) Rows (features) Avenue ArcView 3 ArcGIS ArcIMS Browser *ArcMap Service includes access to version data layers ArcView 3 User Application Needs Assessment AML=> ArcObjects Avenue => ArcObjects/ArcIMS Personal GeoDatabase File Server Chart Symbols Migration Options Data Access 15 of 51

16 System Design Strategies Network Communications 16 of 51

17 Client/Server Protocols File Servers SDE Server Query Processing on Server Data Cache Typical Data Transfer 1 MB Spatial Data NFS CIFS TCP/IP Application Processing on Server UNIX Server WTS Server Web/IMS Server Applications Cache Query Processing on Client UNIX or Windows Clients X.11 RDP ICA HTTP Typical Data Transfer 100 KB Display Data KB displays 1-2 KB displays KB Images Display/Control Cache Remote Terminal Clients X-Windows Terminals Windows Terminals Web Browsers 17 of 51

18 Client/Server Performance Client/Server Communications Network Traffic Transport Time 56 Kbps 1.54 Mbps 10 Mbps 100 Mbps File Server to Workstation Client (NFS) 1 MB => 10 Mb + 40 Mb = 50 Mb 893 Sec. 32 Sec. 5 Sec. 0.5 Sec. SDE Server to Workstation Client (SDE API) 1 MB => 10 Mb >> 5 Mb 89 Sec. 3.2 Sec. 0.5 Sec Sec. UNIX Application Server to X-Terminal Client 100 KB => 1 Mb 18 Sec. 0.6 Sec. 0.1 Sec Sec. Best Practices Windows Terminal Server to Terminal Client (ICA) 100 KB => 1 Mb >> 20 Kb 0.3 Sec Sec Sec Sec. Web Server to Browser Client (HTTP) 50 KB => 500 Kb 9 Sec. 0.3 Sec Sec Sec. 18 of 51

19 Shared Network Performance Client A Client B Client C Ethernet Shared LAN Segment RANDOM TRANSMISSIONS NO COLLISIONS ALLOWED Broadcast Environment Acceptable Collision Avoidance 25 35% Bandwidth Capacity Network transmission capacity is reached when the statistical probability of any two random transmissions occur on the same network segment during the same period of time. 19 of 51

20 Network Configuration Guidelines Network Design Standards Current User Environment Needs Addresses Statistical Application Use Provides Basis for Initial System Design Network Management Ongoing Traffic Management Task Hardware, Application, and User Dependent Strongly Affected by Work Environment and Changes in Computer Technology 20 of 51

21 Network Design Guidelines 21 of 51

22 Data Administration Ways we can store spatial data Ways we can connect to spatial data Ways we can protect spatial data Ways we can backup spatial data Ways we can move spatial data New ways we can access spatial data 22 of 51

23 Ways we can store spatial data File Server ArcSDE DBMS Coverages LIBRARIAN Shapefiles Personal GeoDatabase Image Files MrSID Simple SDE Layers (Unregistered) Registered GeoDatabase Layers Simple Layers (view only) Versioned Layers (edit sessions) Object Relational (domains/relationships) Image Layers (pyramid) 23 of 51

24 Ways we can connect to spatial data File Server Coverages Librarian Shapefiles Image Files Mr SID ArcSDE Simple Layers (view only) Versioned Layers (edit sessions) Object Relational (domains/relationships) Image Layers Internet Data Sources Terminal Server ArcIMS ArcIMS Desktop Local Files Personal Geodatabase Laptop User workspace Cell PDA Laptop Local File Server Desktop Communication Technology Lease Line Internet Metro WAN VPN Wireless Satellite Cable DSL Modem Terminal Local Files Personal GeoDatabase 24 of 51

25 Storage Configuration ArcSDE Storage Strategy Oracle Hot Spare SQL2000 Hot Spare GB Drives Data Tables Index and Log Files RAID GB Usable RAID 1,0 144-GB Usable Data Tables and Index Tables Log Files RAID GB Usable RAID 1 72-GB Usable 25 of 51

26 A B C D E F G H SELECTED ON-LINE Ways we can backup spatial data Host Tape Backup (Tape backup / disk copy) Network Tape Backup DBMS Log DB Online Backup Server Batch Process DB Snapshot DB Online Backup Client Batch Process DB Snapshot DBMS Log Disk Tape Backup Disk Tape Backup Desktop Storage Area Network Tape Backup DBMS Log Desktop Tape Backup Disk Disk Disk Backup Disk Online Backup Client Batch Process Volume Snapshot 26 of 51

27 Ways we can move spatial data Tape Backup DBMS Disk Log (Traditional tape backup/disk copy) DB Online Backup Batch Process DB Snapshot Tape Backup Tape DB Restore Batch Process Slow Tape Backup Tape Restore DBMS Disk Log Disk Backup Disk or CD Disk Restore DBMS Log DB Online Backup Batch Process DB Snapshot DB Restore Batch Process DBMS Log Disk Disk Copy Disk Disk Copy 27 of 51

28 Ways we can move spatial data (Database migration) GeoDatabase Transition DBMS Edit loads ArcSDE Admin Commands Batch Processes ArcCatalog/ArcToolbox DBMS Log Disk Disk Desktop Editor Primary Backup (read only) Desktop Viewers ArcSDE ArcSDE Admin Admin Commands Commands Extract Extract to to file, file, transport, transport, import import to to ArcSDE ArcSDE ArcCatalog/ArcToolbox ArcCatalog/ArcToolbox Migration Migration Tools Tools ArcSDE ArcSDE to to ArcSDE ArcSDE ArcSDE ArcSDE to to Personal Personal Geodatabase Geodatabase Personal Personal Geodatabase Geodatabase to to ArcSDE ArcSDE 28 of 51

29 Ways we can move spatial data (Database replication) Desktop Editor Data model modifications DBMS Edit loads Disk Primary Log Database Replication Synchronous -Commit Logic -High Bandwidth -Performance? Edit loads Asynchronous -Time Delay -Scheduled Transfer -Low Bandwidth OK -Performance OK Edit loads on backup server DBMS Disk ArcSDE data model modifications may be required to support DBMS replication ArcSDE data model modifications may be required to support DBMS replication Edit load applied to backup server to support replication Edit load applied to backup server to support replication Synchronous replication supported over high bandwidth (LAN/Fiber Connects) Synchronous replication supported over high bandwidth (LAN/Fiber Connects) Asynchronous replication supported over low bandwidth (WAN connections) Asynchronous replication supported over low bandwidth (WAN connections) - near real time - near real time - transfer during off peak hours - transfer during off peak hours Log Backup (read only) Desktop Viewers 29 of 51

30 A B C D E F G H SELECTED ON-LINE A B C D E F G H SELECTED ON-LINE Ways we can move spatial data (Disk level replication) DBMS Edit loads Asynchronous -Time Delay -Scheduled Transfer -Low Bandwidth OK -Performance OK DBMS No Server Load Desktop Editor Viewers Disk Disk Mirror Mirror/SnapCopy Incremental changes after initial copy Disk Disk Copy Mirror/SnapCopy Desktop Viewers Disk storage volume must be configured to support data replication strategy Disk storage volume must be configured to support data replication strategy Asynchronous replication supported over low bandwidth (WAN connections) Asynchronous replication supported over low bandwidth (WAN connections) Backup server database restart required to access replicated volume Backup server database restart required to access replicated volume 30 of 51

31 Current Disconnected Editing (ArcGIS 8.2 with ArcFM/ArcFM Designer) ArcFM/Designer Disconnected Editing (multiple edit transactions) DBMS Edit loads Disk V1 Upload Version edits Download Reconciled edits Checkout Personal GeoDatabase Desktop Editor Viewers Primary PGD V Laptop Editor Field Editing PGD V1 Laptop Editor 31 of 51

32 Future Disconnected Editing (Disconnected Editing ArcGIS 8.3) ArcGIS 8.3 Disconnected Editing (single edit transactions) DBMS Edit loads Disk V1 Insert Version updates Checkout Personal GeoDatabase Desktop Editor Viewers Primary PGD V Laptop Editor Field Editing PGD V1 Laptop Editor 32 of 51

33 Future ArcSDE Replication ( ArcGIS 9.1) Loosely Connected Distributed Database Replication Reconcile and post Corporate updates Download Corporate Version updates Corporate CV-1 CV-2 Reconcile and post Corporate updates Download Corporate Version updates Upload Regional Version updates ArcSDE Upload Regional Version updates Region 1 Reconcile and post Regional updates Corporate Editor Reconcile and post Regional updates Region 2 V1 ArcSDE V2 V1 ArcSDE V2 Regional Editor Disconnected Editing Regional Editor Regional Editor Disconnected Editing Regional Editor PGD V1 Regional Editor PGD V1 Regional Editor 33 of 51

34 System Design Strategies System Sizing Tools Performance Updates 34 of 51

35 Pentium 200 Pentium Pro 200 Pentium II 233 Pentium II 266 Pentium II 300 Pentium II 350 Pentium II 400 Pentium III 500 Pentium III 550 Pentium III 650 Pentium III 700 Pentium III 733 Pentium III 800 Pentium III 866 Pentium III 900 Pentium III 933 Pentium III 1000 Pentium III 1200 Intel Xeon 1400 Intel Xeon 1500 Intel Xeon 1700 Intel Xeon 2000 Intel Xeon 2200 Intel Xeon PC Intel Platform Performance Arc02 = 2002 Single ArcInfo User Workstation Performance Sint2000 Sfp2000 Arc01 = 2001 Single ArcInfo User Workstation Performance Arc00 = 2000 Single ArcInfo User Workstation Performance Arc02 Arc99 = 1999 Single ArcInfo User Workstation Performance Arc98 = 1998 Single ArcInfo User Workstation Performance Arc00 Arc01 Arc99 Arc97 Arc98 35 of 51 Relative Performance

36 How do we handle change? Theory of Relative Performance The relative performance of two servers is directly proportional to their compute capacity Performance of Server A Performance of Server B = Clients of Server A Clients of Server B 36 of 51

37 How do we measure relative performance? Standard Performance Evaluation Corporation (SPEC Mission) To develop technically credible and objective benchmarks so that both computer designers and purchasers can make decisions on the basis of realistic workloads 37 of 51

38 Platform Performance Chart Overview Platform SPEC Benchmark Performance ArcSDE Server Model 16 GB 18 GB 20 GB Arc SPECrate_int Performance Level 2000 Performance Level 2001 Performance Level 2002 Performance Level 4 GB 6 GB 8 GB 10 GB 12 GB 14 GB SunFire MHz Arc Arc Arc GB Peak Concurrent Clients Concurrent Clients ArcSDE Memory = *users 38 of 51

39 SPEC Benchmark Results SPEC2000 Benchmarks ( Company System # CPU Processors (CPU) Result Sun Microsystems Sun Enterprise 3500/ MHz UltraSPARC-II 2.46 Sun Microsystems Sun Enterprise 3500/ MHz UltraSPARC-II 4.86 Sun Microsystems Sun Enterprise 3500/ MHz UltraSPARC-II 9.69 Sun Microsystems Sun Enterprise 3500/ MHz UltraSPARC-II 19.5 Sun Microsystems Sun Enterprise MHz UltraSPARC-II of 51

40 ArcSDE Server Sizing Example 18 GB 20 GB Arc SPECrate_int GB 4 GB 6 GB 8 GB 10 GB 16 GB 14 GB 12 GB (2.46) Sun E MHz Arc Arc Arc Peak Concurrent Clients 40 of 51

41 Platform Sizing Models Workstations (ArcInfo, GIS Viewers, Terminal Clients) Application Compute Servers IMS Map Servers GIS File Servers ArcSDE Servers 41 of 51

42 Workstation Platforms Intel Workstations Intel Xeon 2400 Intel Xeon 2200 Intel Xeon 2000 Intel Xeon 1700 Intel Xeon 1500 Arc 2002 Performance Intel Xeon 1400 Pentium III 1200 Pentium III 1000 Pentium III 933 Arc 2001 Performance Pentium III 900 Pentium III 866 Pentium III 800 Pentium III 733 Arc 2000 Performance Pentium III 700 Pentium III 650 Pentium III 550 Arc 1999 Performance Pentium III 500 Pentium II 400 Pentium II 350 Pentium II 300 Arc 1998 Performance Pentium II 266 Pentium II 233 Pentium Pro 200 Pentium 200 Arc 1997 Performance Pentium 150 Pentium 120 Pentium 100 ArcGIS Desktop ArcView Map Objects Windows Terminal Clients ArcIMS Browsers ArcGIS Memory = 512 MB Platform Performance (SPECfp2000) 42 of 51

43 Terminal Server Performance Arc02 Terminal Server Performance Model File Server ArcSDE Server Terminal Server (4 ArcInfo Users/CPU) Terminal Server (8 ArcInfo Users/CPU) Terminal Clients Terminal Clients 43 of 51

44 Windows Terminal Server File Server Data Source Server Processing Equivalents Use for configurations with File Data source 6 GB Memory Arc SPECrate_fp GB Memory 4 GB Memory Intel Xeon MHz PIII Xeon MHz Intel Xeon MHz Arc Arc Arc ArcGIS WTS Memory = *users Total Concurrent Users 44 of 51

45 Windows Terminal Server ArcSDE Server Data Source Server Processing Equivalents Use for configurations with ArcSDE data source 8 GB Memory SPECrate_fp GB Memory Intel Xeon MHz PIII Xeon MHz 6 GB Memory Intel Xeon MHz Arc Arc Arc GB Memory Arc ArcGIS WTS Memory = *users Total Concurrent Users 45 of 51

46 ArcIMS Server Performance File Server ArcSDE Server Map Server (2-sec Map Service Time) Map Server (1-sec Map Service Time) 2001 Baseline PIII Baseline PIII 900 WAN WAN Browser Clients 1800 Transactions per hour/cpu Browser Clients 3600 Transactions per hour/cpu 46 of 51

47 Internet Map Server Performance ArcIMS Map Servers 25 File Data Source 2 GB ArcSDE Data Source 20 SPECrate_fp SunFire280R MHz SunFire280R MHz ,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 15,000 16,000 17,000 18,000 19,000 20,000 21,000 22,000 23,000 24,000 25,000 1 GB Intel Xeon MHz Intel Xeon MHz IBM p MHz MB 512 MB Intel Xeon MHz PIII MHz PIII MHz Baseline Peak Transactions per Hour 47 of 51

48 Data Server Performance Arc02 Server Loading Model File Server ArcSDE Servers One server batch process = 20 GIS clients One server batch process = 20 ArcSDE clients File Server (20 GIS Clients/CPU) ArcSDE Server (20 ArcSDE Clients/CPU) Workstations Workstations OR 48 of 51

49 Workgroup GIS Data Server (ArcSDE and File Server) SPECrate_int ArcSDE Server Processing Equivalents Include IMS peak map server loads IMS file server load = 5 times peak map server load Batch server process = 20 GIS clients SunFire MHz Unisys ES GB HP rp MHz 4 GB IBM p MHz 6 GB Intel Xeon MP MHz 8 GB Unisys ES GB Arc Arc Arc Arc Intel Xeon MHz ArcSDE Memory = *users Peak Concurrent Clients 49 of 51

50 Enterprise GIS Data Server (ArcSDE and File Server) ArcSDE Server Processing Equivalents Include IMS peak map server loads IMS file server load = 5 times peak map server load Batch server process = 20 GIS clients IBM p GB IBM p GB 32 GB SPECrate_int SunFire MHz 16 GB Unisys ES MHz 12 GB 10 GB 8 GB 6 GB 4 GB 2 GB 20 GB 18 GB IBM p HP Superdome MHz SunFire MHz Unisys ES MHz Unisys ES MHz Intel Xeon MHz ArcSDE Memory = *users Peak Concurrent Clients 50 of 51

51 Sustaining Enterprise Operations Managing Technology Change Strategic Plan Strategic Plan 2002 Deployment Authorization 2003 Deployment Authorization 51 of 51

52 Thank You 52 of 51