WebGL. Creating Interactive Content with WebGL. Media #WWDC14. Session 509 Dean Jackson and Brady Eidson WebKit Engineers

Transcription

1 Media #WWDC14 WebGL Creating Interactive Content with WebGL Session 509 Dean Jackson and Brady Eidson WebKit Engineers 2014 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

2

3

4

5

6 JavaScript

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21 AngryBots by Unity

22 AngryBots by Unity

23 Swooop by PlayCanvas

24 Swooop by PlayCanvas

25

26 What You Will Learn

27 What You Will Learn Setting up WebGL in your page

28 What You Will Learn Setting up WebGL in your page How to do basic drawing

29 What You Will Learn Setting up WebGL in your page How to do basic drawing Advanced rendering and animation

30 What You Will Learn Setting up WebGL in your page How to do basic drawing Advanced rendering and animation Relationship to other HTML features

31

32

33

34

35

36

37

38

39

40 three.js MontageJS Copperlicht Babylon JS Turbulenz Cesium Goo Engine

41 Motivation

42 Motivation Powerful graphics in Web content

43 Motivation Powerful graphics in Web content Interoperability through Open Standard

44

45 BEGIN { print "Hello, world!" } display dialog "Hello, world!" (princ "Hello, world!") #include <stdio.h> int main() { printf("hello, world!\n"); return 0; } program HelloWorld; begin WriteLn('Hello, world!'); end. print "Hello, world!" package main import "fmt" func main() { fmt.println("hello, world!") } #import <Foundation/Foundation.h> int main(void) { NSLog(@"Hello, world!\n"); return 0; } PRINT "Hello, world!" DISPLAY 'Hello, world!'. STOP RUN. HAI CAN HAS STDIO? VISIBLE "HAI WORLD!" KTHXBYE public class HelloWorld { public static void main(string[] args) { System.out.println("Hello, world!"); } } <?php echo 'Hello, world!';?> puts "Hello, world!" %!PS /Courier 72 selectfont moveto (Hello World!) show showpage #include <iostream> int main() { std::cout << "Hello, world!" << std::endl; return 0; }

46

47

48

49

50

51

52

53

54 Creating, Configuring, and Drawing

55

56

57

58

59

60

61 Somewhere to Draw Canvas

62 Somewhere to Draw Canvas <canvas> element Or create one via JavaScript: var canvas = document.createelement("canvas");

63 Somewhere to Draw Canvas var canvas = document.queryselector("canvas");

64 Somewhere to Draw Canvas var canvas = document.queryselector("canvas");! canvas.width = 600 * window.devicepixelratio; canvas.height = 400 * window.devicepixelratio;

65

66

67 Something to Draw With Creating the WebGLRenderingContext

68 Something to Draw With Creating the WebGLRenderingContext var gl = canvas.getcontext("webgl");

69

70

71 Configuring the System Creating and setting up the resources

72 Configuring the System Creating and setting up the resources Before we can do the actual draw operation:

73 Configuring the System Creating and setting up the resources Before we can do the actual draw operation: Create some buffers

74 Configuring the System Creating and setting up the resources Before we can do the actual draw operation: Create some buffers Create a program to use while drawing

75

76

77 (0.55, 0.75) (-0.8, -0.3) (0.7, -0.8)

78 (0.55, 0.75) (-0.8, -0.3) (0.7, -0.8)

79 (0.55, 0.75) (-0.8, -0.3) (0.7, -0.8)

80 (0.55, 0.75) (-0.8, -0.3) (0.7, -0.8)

81 (0.55, 0.75) (-0.8, -0.3) (0.7, -0.8)

82 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);

83 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);

84 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);

85 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);! var trianglebuffer = gl.createbuffer();

86 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);! var trianglebuffer = gl.createbuffer(); gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.bufferdata(gl.array_buffer, vertices, gl.static_draw);

87

88 JavaScript

89 JavaScript

90 JavaScript

91 WebGL Rendering Pipeline JavaScript

92 WebGL Rendering Pipeline

93 WebGL Color Primitive Vertex Processing Rasterizer Fragment Depth / Dither Buffer Assembly Shader Stencil Blend Rendering Pipeline

94 Primitive Processing Vertex Shader Primitive Assembly Rasterizer Fragment Shader Depth/ Stencil Color Buffer Blend Dither

95 Primitive Processing Vertex Shader Primitive Assembly Rasterizer Fragment Shader Depth/ Stencil Color Buffer Blend Dither

96 Vertex Shader Fragment Shader JavaScript

97 Vertex Shader Fragment Shader JavaScript

98 Vertex Shader Fragment Shader JavaScript Program

99 Vertex Shader

100 Vertex Shader Vertex Shader Vertex Shader

101 Vertex Shader Vertex Shader Vertex Shader

102 Vertex Shader Vertex Shader Vertex Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader

103 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);

104 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);

105 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);

106 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);! var fragmentshader = gl.createshader(gl.fragment_shader); gl.shadersource(fragmentshader, "... source code..."); gl.compileshader(fragmentshader);

107 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);! var fragmentshader = gl.createshader(gl.fragment_shader); gl.shadersource(fragmentshader, "... source code..."); gl.compileshader(fragmentshader);! var program = gl.createprogram();

108 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);! var fragmentshader = gl.createshader(gl.fragment_shader); gl.shadersource(fragmentshader, "... source code..."); gl.compileshader(fragmentshader);! var program = gl.createprogram(); gl.attachshader(program, vertexshader); gl.attachshader(program, fragmentshader);

109 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);! var fragmentshader = gl.createshader(gl.fragment_shader); gl.shadersource(fragmentshader, "... source code..."); gl.compileshader(fragmentshader);! var program = gl.createprogram(); gl.attachshader(program, vertexshader); gl.attachshader(program, fragmentshader); gl.linkprogram(program);

110 var vertexshader = gl.createshader(gl.vertex_shader); gl.shadersource(vertexshader, "... source code..."); gl.compileshader(vertexshader);! var fragmentshader = gl.createshader(gl.fragment_shader); gl.shadersource(fragmentshader, "... source code..."); gl.compileshader(fragmentshader);! var program = gl.createprogram(); gl.attachshader(program, vertexshader); gl.attachshader(program, fragmentshader); gl.linkprogram(program);! gl.useprogram(program);

111

112

113

114 var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute);

115 var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute);! gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.vertexattribpointer(positionattribute, 2, gl.float, false, 0, 0);

116 var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute);! gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.vertexattribpointer(positionattribute, 2, gl.float, false, 0, 0);

117 var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute);! gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.vertexattribpointer(positionattribute, 2, gl.float, false, 0, 0);! gl.drawarrays(gl.triangles, 0, 3);

118

119 <!DOCTYPE html> <head> <title>webgl Triangle</title> <style> canvas { width: 600px; height: 400px; } </style> </head> <script id="vertexshadersource" type="text/glsl"> // Simple vertex shader. Simply returns the position it was provided. attribute vec4 position; void main() { gl_position = position; } </script> <script id="fragmentshadersource" type="text/glsl"> // Extremely simple fragment shader. Draws every pixel red. #ifdef GL_ES precision mediump float; #endif! void main() {

120 gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.bufferdata(gl.array_buffer, vertices, gl.static_draw);!! // ---- DRAWING ---- // Clear to black. gl.clearcolor(0, 0, 0, 1); gl.clear(gl.color_buffer_bit);! // Bind the vertex attributes for the draw operation. We first make // sure we're talking to the correct buffer, then we're going to // associate that buffer with the vertex attribute from the program. // Our buffer is an array of (x,y) points, so 2 floating point values // per vertex. gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.vertexattribpointer(positionattribute, 2, gl.float, false, 0, 0);! } // The actual draw call. We know there are 3 points in the buffer // (a triangle). gl.drawarrays(gl.triangles, 0, 3);! window.addeventlistener("load", drawtriangle, false); </script> <body>

121 Shaders

122 GLSL

123 GLSL C-like language designed for parallel graphics

124 GLSL C-like language designed for parallel graphics Primitives: Vectors, matrices

125 GLSL C-like language designed for parallel graphics Primitives: Vectors, matrices Built-in functions: Trigonometry, vector math, smoothing, and clamping

126 Vertex Shader Vertex Shader Vertex Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader Fragment Shader

127 Vertex Shader Fragment Shader

128 Vertex Shader Fragment Shader

129 Vertex Shader Global Constants (uniforms) Fragment Shader

130 Vertex Shader gl_position Global Constants (uniforms) Fragment Shader

131 Vertex Shader gl_position Global Constants (uniforms) Fragment Shader gl_fragcolor

132 Vertex Shader Source Code attribute vec4 aposition;! void main() { gl_position = aposition; }

133 Vertex Shader Source Code attribute vec4 aposition;! void main() { gl_position = aposition; }

134 Vertex Shader Source Code attribute vec4 aposition;! void main() { gl_position = aposition; }

135 Fragment Shader Source Code precision mediump float;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

136 Fragment Shader Source Code precision mediump float;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

137 Fragment Shader Source Code precision mediump float;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

138 Fragment Shader Source Code precision mediump float;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); } red green blue alpha

139 Demo Live Shader Editor

140 Shaders C-like programs that run on the GPU Control over vertex positions and pixel colors Extremely powerful

141 Advanced Rendering Merging WebGL with the rest of HTML Brady Eidson WebKit Engineer

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163 var vertices = new Float32Array([ -0.8, -0.3, 0.7, -0.8, 0.55, 0.75 ]);! var trianglebuffer = gl.createbuffer(); gl.bindbuffer(gl.array_buffer, trianglebuffer); gl.bufferdata(gl.array_buffer, vertices, gl.static_draw);

164 var vertices = new Float32Array([... ]);! var discbuffer = gl.createbuffer(); gl.bindbuffer(gl.array_buffer, discbuffer); gl.bufferdata(gl.array_buffer, vertices, gl.static_draw);

165 var vertices = new Float32Array([ , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

166 , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

167

168 Any Data You Want

169 Any Data You Want for Any Point You Want

170

171

172

173

174

175

176 var vertices = new Float32Array([ , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]);! var geometrybuffer = gl.createbuffer(); gl.bindbuffer(gl.array_buffer, geometrybuffer); gl.bufferdata(gl.array_buffer, vertices, gl.static_draw);

177 var texturecoords = new Float32Array([ , , , , , , , , , , ]);! var texturebuffer = gl.createbuffer(); gl.bindbuffer(gl.array_buffer, texturebuffer); gl.bufferdata(gl.array_buffer, texturecoords, gl.static_draw);

178 Configuring Our Shaders var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute);! gl.bindbuffer(gl.array_buffer, positionbuffer); gl.vertexattribpointer(positionattribute, 3, gl.float, false, 0, 0);

179 Configuring Our Shaders var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute); gl.bindbuffer(gl.array_buffer, positionbuffer); gl.vertexattribpointer(positionattribute, 3, gl.float, false, 0, 0);! var textureattribute = gl.getattriblocation(program, "atexturecoord"); gl.enablevertexattribarray(textureattribute);!

180 Configuring Our Shaders var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute); gl.bindbuffer(gl.array_buffer, positionbuffer); gl.vertexattribpointer(positionattribute, 3, gl.float, false, 0, 0);! var textureattribute = gl.getattriblocation(program, "atexturecoord"); gl.enablevertexattribarray(textureattribute); gl.bindbuffer(gl.array_buffer, texturebuffer); gl.vertexattribpointer(textureattribute, 2, gl.float, false, 0, 0);

181 Vertex Shader Source Code attribute vec4 aposition;! void main() { gl_position = aposition; }

182 Vertex Shader Source Code attribute vec4 aposition; attribute vec2 atexturecoord;! void main() { gl_position = aposition; }

183 Vertex Shader Source Code attribute vec4 aposition; attribute vec2 atexturecoord;! varying vec2 vtexturecoord;! void main() { gl_position = aposition; }

184 Vertex Shader Source Code attribute vec4 aposition; attribute vec2 atexturecoord;! varying vec2 vtexturecoord;! void main() { gl_position = aposition; vtexturecoord = atexturecoord; }

185 Fragment Shader Source Code precision mediump float;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

186 Fragment Shader Source Code precision mediump float;! varying vec2 vtexturecoord;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

187 Configuring Our Shaders var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute); gl.bindbuffer(gl.array_buffer, positionbuffer); gl.vertexattribpointer(positionattribute, 3, gl.float, false, 0, 0);! var textureattribute = gl.getattriblocation(program, "atexturecoord"); gl.enablevertexattribarray(textureattribute); gl.bindbuffer(gl.array_buffer, texturebuffer); gl.vertexattribpointer(textureattribute, 2, gl.float, false, 0, 0);

188 Configuring Our Shaders var positionattribute = gl.getattriblocation(program, "aposition"); gl.enablevertexattribarray(positionattribute); gl.bindbuffer(gl.array_buffer, positionbuffer); gl.vertexattribpointer(positionattribute, 3, gl.float, false, 0, 0);! var textureattribute = gl.getattriblocation(program, "atexturecoord"); gl.enablevertexattribarray(textureattribute); gl.bindbuffer(gl.array_buffer, texturebuffer); gl.vertexattribpointer(textureattribute, 2, gl.float, false, 0, 0);! var sampleruniform = gl.getuniformlocation(program, "sampler");

189 Fragment Shader Source Code precision mediump float;! varying vec2 vtexturecoord;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

190 Fragment Shader Source Code precision mediump float;! varying vec2 vtexturecoord;! uniform sampler2d sampler;! void main() { gl_fragcolor = vec4(1.0, 0.0, 0.0, 1.0); }

191 Fragment Shader Source Code precision mediump float;! varying vec2 vtexturecoord;! uniform sampler2d sampler;! void main() { gl_fragcolor = texture2d(sampler, vtexturecoord); }

192 Texture Sources

193 Texture Sources <img> element

194 Texture Sources <img> element XMLHttpRequest

195 Texture Sources <img> element XMLHttpRequest <video> element

196 Texture Sources <img> element XMLHttpRequest <video> element <canvas> element

197 <img id= TextureImage src= FlatCompass.png >

198 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); </script>

199 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); gl.activetexture(gl.texture0); gl.bindtexture(gl.texture_2d, texture); </script>

200 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); gl.activetexture(gl.texture0); gl.bindtexture(gl.texture_2d, texture);! var image = document.getelementbyid( TextureImage ); </script>

201 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); gl.activetexture(gl.texture0); gl.bindtexture(gl.texture_2d, texture);! var image = document.getelementbyid( TextureImage );! gl.teximage2d(gl.texture_2d, 0, gl.rgba, gl.rgba, gl.unsigned_byte, image); </script>

202 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); gl.activetexture(gl.texture0); gl.bindtexture(gl.texture_2d, texture);! var image = document.getelementbyid( TextureImage );! gl.teximage2d(gl.texture_2d, 0, gl.rgba, gl.rgba, gl.unsigned_byte, image); </script>

203 <img id= TextureImage src= FlatCompass.png >! <script type= text/javascript > var texture = gl.createtexture(); gl.activetexture(gl.texture0); gl.bindtexture(gl.texture_2d, texture);! var image = document.getelementbyid( TextureImage );! gl.teximage2d(gl.texture_2d, 0, gl.rgba, gl.rgba, gl.unsigned_byte, image); gl.uniform1i(sampleruniform, 0); </script>

204

205

206 Demo Building a compass

207

208

209 The Web Platform

210 The Web Platform

211 The Web Platform

212 The Web Platform

213 The Web Platform

214

215

216

217

218

219

220

221 When to Draw

222 FIXME: The contents of will likely be different later on June 2nd, should update the video

223 FIXME: The contents of will likely be different later on June 2nd, should update the video

224 Do Not Use Timers!

225 requestanimationframe()

226 requestanimationframe() 1 Call requestanimationframe() with a callback

227 requestanimationframe() 1 Call requestanimationframe() with a callback var drawingcallback = function() { }

228 requestanimationframe() 1 Call requestanimationframe() with a callback var drawingcallback = function() { }! requestanimationframe(drawingcallback);

229 requestanimationframe() 2 Draw your scene var drawingcallback = function() { }! requestanimationframe(drawingcallback);

230 requestanimationframe() 2 Draw your scene var drawingcallback = function() { updatephysics(); drawcompass(); drawterrain(); }! requestanimationframe(drawingcallback);

231 requestanimationframe() 3 Request the next callback var drawingcallback = function() { updatephysics(); drawcompass(); drawterrain(); }! requestanimationframe(drawingcallback);

232 requestanimationframe() 3 Request the next callback var drawingcallback = function() { updatephysics(); drawcompass(); drawterrain(); requestanimationframe(drawingcallback); }! requestanimationframe(drawingcallback);

233 Demo Grand finale

234 Wrapping Up

235 Wrapping Up Rich, powerful, fast graphics

236 Wrapping Up Rich, powerful, fast graphics WebGL available in Safari on OS X Yosemite and ios 8

237 Wrapping Up Rich, powerful, fast graphics WebGL available in Safari on OS X Yosemite and ios 8 Also available in WKWebView

238 More Information Evangelism Safari for Developers WebKit Developer Technical Support Apple Developer Forums

239 Related Sessions Introducing the Modern WebKit API Nob Hill Tuesday 2:00PM Web Inspector and Modern JavaScript Russian Hill Thursday 10:15AM

240 Labs Safari and WebKit Lab Media Lab B Wednesday 4:30PM Safari and WebKit Lab Media Lab B Thursday 2:00PM

241